Diving into Kronecker Adapters: Component Design Matters

Danchen Yu; Feng Zhou; Jiayu Bai; Robert C. Qiu; TianQi Hou; Zenan ling; Zhenyu Liao

REVIEW 4 major objections 6 minor 4 references

For Kronecker adapters, the component shape (r1, r2, r) — not attainable rank — decides how well the update aligns with full fine-tuning, and the paper distills this into the rule: small r1, large r2, moderate r.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Kronecker adapter performance depends on the dimensions and number of its Kronecker components, and component design with small r1, large r2, and moderate r improves alignment with full fine-tuning.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Kronecker component design is a real and promising idea, but the theory doesn't cover the practical setup and the budget-controlled experiments aren't fully budget-controlled — worth refereeing, not yet worth trusting. the 4 major comments →

arxiv 2602.01267 v2 pith:GYROO6BV submitted 2026-02-01 cs.LG

Diving into Kronecker Adapters: Component Design Matters

Jiayu Bai , Danchen Yu , Zhenyu Liao , TianQi Hou , Feng Zhou , Robert C. Qiu , Zenan Ling This is my paper

classification cs.LG

keywords Kronecker adaptersparameter-efficient fine-tuningcomponent designsubspace alignmentKronecker product SVDLoRAlarge language models

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Parameter-efficient fine-tuning should not treat a Kronecker adapter as a black box. The paper shows that the same attainable rank can give very different accuracy depending on whether it is realized as different component dimensions or a different number of components, and that the differences are systematic. Using the Kronecker-product singular value decomposition, the authors prove that the alignment between the adapter's learned subspace and the full fine-tuning gradient is controlled by r1, r2, and r. From the proof they extract three principles: increasing r1 degrades performance, increasing r2 improves it, and adding components r only helps up to the rank of the target update. A reader should care because the resulting method, CDKA, turns the previously underperforming Kronecker adapter into a competitive PEFT method at a small fraction of the parameter cost of strong baselines.

Core claim

The central claim is that for the general Kronecker adapter ∆W = Σ B^(i) ⊗ A^(i), the expressive capacity is not captured by the maximum attainable rank r·r1·r2. Using the Kronecker SVD, the authors show that after gradient descent on a linear regression with isotropic sub-Gaussian inputs, the overlap between the adapter's subspace and the full fine-tuning gradient depends on the three design parameters in a specific way. The allowed initialization scale α grows like √r2 and shrinks with √(r r1), so large r2 widens the regime in which alignment holds, while large r1 and r narrow it. The paper concludes with the design principles: increasing r1 tends to hurt, increasing r2 consistently helps,

What carries the argument

The load-bearing object is the Kronecker product singular value decomposition (KPSVD): a matrix K is written as Σ σ_i B^(i) ⊗ A^(i) by applying an ordinary SVD to the reshaped matrix Kreshape(K). The analysis tracks how the component matrices A^(i), viewed as columns of eA_t, rotate under gradient descent toward the top singular subspace of the reshaped full-fine-tuning gradient eG_0. Theorem 3.2 bounds the alignment error, and its proof shows that both the admissible initialization variance α and the number of steps needed scale favorably in r2 but unfavorably in r1 and r — this asymmetry is what carries the entire argument. The second piece is Theorem 3.4, which derives the scaling factor

Load-bearing premise

The load-bearing assumption is that the alignment theorem proven for near-zero Gaussian initialization on a linear, isotropic-input loss still describes real fine-tuning with Kaiming initialization and AdamW — an assumption that is not established because the practical initialization variance Θ(r2) lies outside the theorem's proven range O(√r2).

What would settle it

A concrete falsifier: measure the alignment quantity ∥U_{r*,⊥}(eG_0) U_{r*}(eA_t)∥2 during actual AdamW fine-tuning of a transformer, sweeping r1, r2, r while holding the parameter budget constant. The theory predicts monotone improvement with r2 and monotone decline with r1; if the measured alignment or downstream accuracy instead saturates or inverts at large r2 (beyond the regime where the α bound holds), or if the performance ranking across configurations is unchanged when the λ scaling from Theorem 3.4 is removed, the claimed mechanism is not what drives the empirical results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Component design choices like r1 = r2 (KronA) or chasing full-rank updates are shown to be suboptimal; the high-rank promise of Kronecker adapters is only realized through asymmetric components.
Under a fixed parameter budget, the guidelines say to first add components until r reaches the effective rank of the target update (empirically r* ≈ 2–8), then spend remaining budget on r2 rather than r or r1.
Kronecker adapters become competitive with the strongest PEFT baselines: with 0.41M trainable parameters (12.5% of LoRA-One's budget), CDKA reaches 87.30 GLUE average near full fine-tuning's 87.91, and 56.71 on GSM8k versus 55.40 for LoRA-One on LLaMA-2-7B.
The scaling rule λ ∝ 1/√(r r2) gives a config-independent operating point, so one learning-rate schedule can serve a wide range of r1, r2, r choices without retuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A natural next test is whether the same asymmetry appears in other structured adapters (Hadamard product, block-diagonal, tensor-train): the proof mechanism — singular alignment of reshaped gradients — suggests any product-structured update will inherit an r- vs r2-style trade-off.
The theory implies that 'rank' is the wrong currency for comparing PEFT methods: two adapters with equal rank can have opposite training dynamics, so benchmarks should report component geometry as well as rank and parameter count.
One testable extension the paper leaves open: the bound in Eq. (12) suggests r2 can be pushed toward din, at which point each Kronecker factor A becomes nearly square and the adapter resembles a low-rank-plus-identity update; whether performance then saturates or collapses is an empirical question.
The transfer from the linear-isotropic setting to real transformers rests on the variance bound; a direct measurement of the alignment quantity during real AdamW fine-tuning would confirm whether the theoretical mechanism is what drives the empirical gains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Kronecker component design is a real and promising idea, but the theory doesn't cover the practical setup and the budget-controlled experiments aren't fully budget-controlled — worth refereeing, not yet worth trusting.

read the letter

Dear colleague,

Quick take: CDKA is worth engaging with, but not yet the principled story it claims. The empirical headline—Kronecker adapters hitting 56.71 on GSM8k and 87.30 average GLUE with far fewer parameters than LoRA-One—is real enough to take seriously, and the paper is the first to turn component choice (r1, r2, r) from heuristic into explicit design rules. But the theoretical derivation of those rules does not cover the actual setup, and the “same parameter budget” experiments are not all same-budget. Both need fixing before the rules get cited.

What’s good. The KSVD-based alignment analysis is a sensible extension of LoRA-One’s argument to mixtures of Kronecker products, and the appendix is detailed. Lemma B.2 and Theorem B.3 follow Zhang et al. with attribution; I found no circular dependence. Theorem 3.4’s λ = 1/√(r·r2) stabilization is simple, useful, and supported by the gradient-norm figure. The GLUE and reasoning results are competitive and clearly improve over KronA. That is a real within-subfield contribution.

Where it’s soft. First, Theorem 3.2’s initialization regime is not where the method runs. Eq. (12) bounds α at about 1e-15·√r2 for LLaMA-2-7B, while Appendix D says Kaiming initialization gives A0 variance Θ(r2)—so the practical operating point is orders of magnitude outside the proven range. Remark 3.3 hand-waves “sufficiently large” without checking. The design principles are therefore extrapolations for the actual setup, not derivations. Second, the empirical sweeps are not controlled: Table 7 is labeled same parameter budget, but under the paper’s own Eq. (5), (2,2,8) costs 16 budget units while (2,16,8) costs 65. Tables 2–5 also vary total parameter count with the hyperparameter being tested, which confounds “r2 helps” with “more parameters help.” Third, the final configurations on GSM8k and HumanEval appear selected from sweeps on those same evaluation sets, so the headline numbers carry selection bias. Code is promised but no artifact is linked, which matters for a paper whose value is partly empirical.

Net: the central idea—component design, not attainable rank, drives Kronecker-adapter behavior—is plausible and probably partly right, but the evidence as presented does not establish the specific principles. I would send this to review with a request for controlled budget experiments and either a proof covering the practical initialization or an honest reframing of the theory as illustrative. I wouldn’t cite the design rules yet; I would cite it as evidence that Kronecker adapters deserve systematic component tuning.

Referee Report

4 major / 6 minor

Summary. The paper studies the component design of Kronecker adapters, i.e., the choices of r1, r2, and r in the update sum Σ_i B^{(i)} ⊗ A^{(i)}. It argues that the subspace alignment between Kronecker adapters and full fine-tuning is governed by this component design, derives three principles (increasing r1 degrades, increasing r2 improves, increasing r saturates), and packages the resulting configuration choices plus a λ-scaling stabilization rule as CDKA. Empirical support is provided through sweeps on LLaMA-2-7B/GMS8k and through comparisons on GLUE with T5-Base, GSM8k/HumanEval with LLaMA-2-7B, and GSM8k with LLaMA-3.1-8B. The appendix contains a detailed gradient-descent alignment analysis adapted from Zhang et al. (2025), with explicit attribution.

Significance. If the design principles and the transfer of the theoretical analysis to practice are sound, CDKA is a practically valuable PEFT recipe: it reaches 56.71 on GSM8k with LLaMA-2-7B (above LoRA-One's 55.40) and an 87.30 GLUE average with 0.41M trainable parameters. The paper ships code, its main theorem is stated with attribution to prior work, and the appendix proof is detailed enough to be checked. However, the paper's novelty claim rests specifically on the claim that the r1/r2/r principles are derived, not merely observed; the current manuscript does not yet make that derivation load-bearing for the experiments in a convincing way.

major comments (4)

[Section 3.2 / Appendix B, Eq. (12) and Appendix D] The theory-to-practice transfer in Remark 3.3 is not supported. Eq. (12) is a sufficient condition on the Gaussian initialization variance α of the A-components. For a typical LLaMA-2-7B linear layer (din≈4096) with r1=2, r=8, θ=0.1, κ≈1, and σ1(Ḡ0), ξ of order 1, the right-hand side of Eq. (12) is of order 10^{-13} r2^{5/4}. Appendix D states that Kaiming initialization gives Var(A0)=Θ(r2), so the actual A-entry scale is Θ(√r2) — several orders of magnitude above the theorem's bound. Remark 3.3 merely asserts that the bound should be 'sufficiently large' and never checks this condition. Moreover, Eq. (12) is a sufficient condition for alignment at a fixed α; it does not by itself imply monotonic improvement of alignment as r2 grows at a fixed α. Thus the claimed principled derivation of the three design principles does not cover the experimental operating point.
[Section 3.3, Table 7] Table 7 is labeled 'same parameter budget' but the rows do not have the same budget under Eq. (5). In the paper's own param units: (2,2,8) gives 16, (2,16,2) gives 16.25, (2,2,32) gives 64, and (2,16,8) gives 65. So the 57.95 for the last row is obtained with roughly 4× the budget of the first row. The stated guideline — that after r exceeds r*, increasing r2 is more beneficial than increasing r — is therefore confounded with a large parameter-count increase. This needs to be redone with matched budgets, as the section title promises.
[Section 3.2, Tables 3–5] The individual validation sweeps do not control total parameter count. For example, in Table 4 with r1=2 and r=1, increasing r2 from 2 to 16 changes the Eq. (5) budget from 2 to 8.125 units; in Table 5, increasing r from 1 to 64 increases the budget linearly by a factor of 64. The monotone improvement in Table 4 and the plateau in Table 5 may therefore reflect parameter-count effects rather than a structural r1/r2/r effect. Table 6 provides a controlled comparison for simultaneous r1=r2 scaling, but the individual r1, r2, and r validations that introduce the principles are confounded.
[Section 3.2–3.3 and Section 4.2] Component configurations and α values are selected using the same GSM8k benchmark on which the final numbers are reported. For instance, the MetaMathQA configuration r1=2, r2=2, r=8 is the best row in Table 7, and the r*∈[2,8] boundary is inferred from GSM8k sweeps; Table 13 then sets task-specific α for the same tasks. No held-out validation set or nested selection protocol is described. This makes the headline numbers vulnerable to selection bias. A separate validation split, or reporting all swept configurations on the final test sets, is needed to support the state-of-the-art claims.

minor comments (6)

[Abstract] The abstract says 'various architectures and modalities', but the experiments cover only language models on NLP tasks (T5-Base, LLaMA-2-7B, LLaMA-3.1-8B). Please align the abstract with the actual experimental scope.
[Section 4.2, first paragraph] The text says 'As shown in Table 9' when discussing NLG results; the NLG results are in Table 10. This cross-reference should be corrected.
[Eq. (7)] The definition of Y appears to contain a typo: it is written as Y=[x1,...,xN] but should presumably be Y=[y1,...,yN] (a dout×N matrix of ground-truth outputs).
[Section 3.2, validation for r2] The sentence 'indicating the use of a larger 2' is missing the subscript: it should read 'larger r2'.
[Theorem 3.2 / experimental setup] Theorem 3.2 assumes Gaussian initialization of Ā0 with variance α², while the experiments use Kaiming-initialized A with the λ scaling of Eq. (15). The relation between the theoretical α and the experimental λ, α hyperparameters should be stated explicitly; otherwise the empirical verification is not directly connected to the theorem.
[Eq. (5)] Eq. (5) writes the parameter count as proportional to r(r1/r2 + r2/r1), which assumes din≈dout. The proportionality is fine, but all budget comparisons in Tables 6–7 implicitly assume this. It would help to state the exact expression with din and dout, or to note that the same approximate relation holds layer-wise in the tested transformer linear layers.

Circularity Check

0 steps flagged

No circularity found; the design principles are extrapolated from stated bounds, not restated inputs.

full rationale

The paper's derivation chain is self-contained. Theorem 3.2 is an adaptation of Zhang et al. (2025) with explicit attribution, and the alignment analysis is a genuine mathematical statement about a linearized Kronecker-adapter dynamics, not a restatement of the empirical sweeps. Remark 3.3 follows by inspecting the monotonicity of the bounds in Eq. (12) and Eq. (13): larger r2 increases the allowable α and decreases the required t*, while larger r1 and r have the opposite effect. These are extrapolations from stated sufficient conditions, not definitions or fitted parameters renamed as predictions. The practical concern that the experiments operate outside Theorem 3.2's initialization regime (Kaiming variance Θ(r2) vs. the tiny α bound) is a validity/transfer gap, not an input-output equivalence. No load-bearing self-citation exists: the cited LoRA-One work is by different authors and is used with attribution. The hyperparameter selection on validation sets is a selection-bias/soundness issue, not circularity. The paper therefore receives a non-finding on circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or mathematical entities are postulated; CDKA is a configuration-plus-scaling method. The epistemic burden is carried by the three free hyperparameters, the small-alpha transfer assumption, and the empirical sweeps used to choose configurations.

free parameters (3)

r1, r2, r (component configuration) = GLUE: 3,3,1; LLaMA-2 math: 2,2,8; LLaMA-2 code: 2,8,4
Per-task settings selected via in-paper sweeps on the target datasets (Tables 3-5 and 7); not uniquely determined by the theoretical analysis.
alpha (scaling hyperparameter in Eq. 16) = 16 (T5), 64 (LLaMA-2/3.1)
Tunable constant in the scaling factor; no sensitivity analysis is reported.
r* empirical boundary = 2-8
Empirical rule for whether to increase r or r2 (Section 3.3), inferred from GSM8k sweeps; not predicted by Theorem 3.2.

axioms (5)

standard math Kronecker-product SVD and the Kreshape equivalence (Definition 2.1 / E.1)
Foundation for writing any matrix as a sum of Kronecker components and for the alignment analysis.
domain assumption Inputs X are i.i.d. isotropic zero-mean sub-Gaussian and the loss is squared error
Defines the setting of Theorem 3.2 and the alignment result; does not hold for real language-model inputs.
domain assumption Divisibility constraints (r1 | dout) and (r2 | din)
Stated in Section 1.2 and required for the Kronecker component shapes to have integer dimensions.
domain assumption Each Kronecker component attains full rank r1 and r2 in the rank bound
Used in Eq. (6) for rank(DeltaW) = r * r1 * r2; components need not be full rank during training.
ad hoc to paper The small-alpha theory regime transfers to practical Kaiming initialization
Remark 3.3 asserts the alpha upper bound in Eq. (12) is large enough for practice, but Kaiming init has variance Theta(r2) while the bound grows like sqrt(r2); the paper does not establish that the practical alpha lies in the proven range.

reviewed 2026-08-03 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Diving into Kronecker Adapters: Component Design Matters." pith.science (2026). https://pith.science/paper/GYROO6BV

@misc{pith2026260201267,
  author       = {Pith},
  title        = {Pith review of: Diving into Kronecker Adapters: Component Design Matters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GYROO6BV}},
  note         = {Machine review of arXiv:2602.01267}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget-aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith

[4]

eAt eBt # , then we can rewrite the above iteration to: Zt+1 =HZ t − ˆEt+1,(21) where H=

URL https://openreview.net/forum? id=KwIlvmLDLm. Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y ., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement.arXiv preprint arXiv:2402.14658, 2024. 11 Diving into Kronecker Adapters: Component Design Matters A. Experiments Details We fine-tune all linear...

Pith/arXiv arXiv 2024
[2023]

Zhang, Y ., Liu, F., and Chen, Y

URL https://openreview.net/forum? id=lq62uWRJjiY. Zhang, Y ., Liu, F., and Chen, Y . LoRA-one: One- step full gradient could suffice for fine-tuning large language models, provably and efficiently. InForty- second International Conference on Machine Learning,
[2025]

Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H

URL https://openreview.net/forum? id=TwJrTz9cRS. Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H. Fedpara: Low- rank hadamard product for communication-efficient fed- erated learning. InInternational Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=d71n4ftoCBy. Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning...

Pith/arXiv arXiv 2022
[2799]

PMLR, 2019. Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. InInternational Confer- ence on Learning Representations, 2022. URL https: //openreview.net/forum?id=nZeVKeeFYf9. Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y . HiRA: Parameter-efficient ha...

2019

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.

[1] [4]

eAt eBt # , then we can rewrite the above iteration to: Zt+1 =HZ t − ˆEt+1,(21) where H=

URL https://openreview.net/forum? id=KwIlvmLDLm. Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y ., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement.arXiv preprint arXiv:2402.14658, 2024. 11 Diving into Kronecker Adapters: Component Design Matters A. Experiments Details We fine-tune all linear...

Pith/arXiv arXiv 2024

[2] [2023]

Zhang, Y ., Liu, F., and Chen, Y

URL https://openreview.net/forum? id=lq62uWRJjiY. Zhang, Y ., Liu, F., and Chen, Y . LoRA-one: One- step full gradient could suffice for fine-tuning large language models, provably and efficiently. InForty- second International Conference on Machine Learning,

[3] [2025]

Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H

URL https://openreview.net/forum? id=TwJrTz9cRS. Hyeon-Woo, N., Ye-Bin, M., and Oh, T.-H. Fedpara: Low- rank hadamard product for communication-efficient fed- erated learning. InInternational Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=d71n4ftoCBy. Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning...

Pith/arXiv arXiv 2022

[4] [2799]

PMLR, 2019. Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. InInternational Confer- ence on Learning Representations, 2022. URL https: //openreview.net/forum?id=nZeVKeeFYf9. Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y . HiRA: Parameter-efficient ha...

2019

Diving into Kronecker Adapters: Component Design Matters

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it