Diving into Kronecker Adapters: Component Design Matters

Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget–aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at \url{https://github.com/rainstonee/CDKA}.

Large AI models are powerful, but fully retraining them for every new task is expensive and often impractical. A common solution is to fine-tune only a small set of added parameters, called adapters, while keeping the original model mostly unchanged. Kronecker adapters are one such method, and they can make surprisingly expressive updates with relatively few trainable parameters. However, these adapters depend heavily on how their internal components are designed. Existing methods often choose these component shapes and numbers by simple rules or trial and error. In this work, we study how these choices affect the adapter’s ability to imitate full fine-tuning. We find that the dimensions and number of Kronecker components play an important role in performance. Based on this understanding, we propose Component Designed Kronecker Adapters, or CDKA, which gives practical guidelines for choosing better component configurations under a fixed parameter budget. Experiments on different model architectures and data types show that CDKA improves fine-tuning effectiveness.