{"id":"c2906ae6-d509-44bf-b553-57dcf8cf964a","arxiv_id":"2508.00230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"KRAdapter, a Khatri-Rao product adapter, produces full-rank high-effective-rank weight updates for parameter-efficient fine-tuning and reports improved out-of-distribution performance over LoRA and other full-rank PEFT methods on vision-language and language benchmarks.","lead":"KRAdapter builds the fine-tuning update for a large model from the Khatri-Rao product of two small matrices, producing full-rank updates that low-rank LoRA-style adapters cannot reach. Across CLIP vision-language models and 7-8B parameter language models it reports broadly better out-of-distribution accuracy than LoRA and several newer full-rank adapters, at similar training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD claim is causally ambiguous: Table 2 shows KRAdapter has both the highest effective rank and the smallest update norm, so the reported generalization gains cannot yet be attributed to rank rather than update magnitude.","rationale":"I focused on the update-norm confound because it targets the abstract's explanatory claim, not merely the stability of the numbers. The reader identified this as the second fragile premise, so my agreement is partial rather than full: I set aside the single-run issue as secondary, since even replicated margins would leave the mechanism unidentified. I also considered two other real weaknesses: the Appendix A.2 proof of Theorem 3.1 contains a typo in Eq. A.15 and its grouping argument is not fully rigorous, and Section 3.2 initializes U to zero while Theorem 3.1 assumes i.i.d. entries, so the stated full-rank guarantee does not literally cover the deployed initialization. Both should be fixed, but they concern the theoretical packaging rather than the empirical causal story. The norm-matched experiment above is the single check that would settle whether the reported OOD improvements are due to high effective rank or simply to smaller updates; if it fails, the paper's main explanatory narrative would need to be revised even if the adapter remains practically useful.","tokens_in":27293,"tokens_out":18341,"duration_ms":201812,"concrete_test":"Run norm-matched OOD controls on the Table 2 protocol (ViT-B/32 and ViT-L/14, 5 seeds): train LoRA, SinLoRA, RandLoRA, Krona and KRAdapter with an added Frobenius-norm penalty on the update, tuning the penalty so every method ends at KRAdapter's norm; report ID accuracy, OOD accuracy and rgen. If KRAdapter's rgen advantage over norm-matched baselines is not significant, the OOD gains are attributable to update magnitude rather than effective rank; if the advantage persists, the rank-based mechanism survives this confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central explanatory claim—that Khatri–Rao updates improve OOD generalization because of their high effective rank—rests on a comparison that never separates rank from update norm. In Section 4.2.3, Table 2 reports nuclear/Frobenius norms of the learned updates alongside rgen; KRAdapter is best on rgen but also systematically smallest in norm (e.g., ViT-B/32 Fro 2.3 vs LoRA 4.1 and Krona 4.9; ViT-H/14 Fro 5.5 vs Krona 9.2). The text itself notes that small update norms have previously been linked to OOD robustness, so update magnitude is a plausible alternative mechanism. FT has an even smaller norm on ViT-H/14 (1.9) yet a lower rgen (0.39 vs 0.48), which suggests norm is not the whole story, but no controlled experiment varies norm while holding the construction fixed. Without such a control, the abstract's 'by construction high effective rank' explanation for the empirical gains is not established; the results are equally consistent with KRAdapter simply being a milder update. This is more load-bearing than the single-seed issue: even a 5-seed replication showing stable margins would not tell us why KRAdapter wins.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KRAdapter, a parameter-efficient fine-tuning (PEFT) method that constructs weight updates as a Khatri-Rao product of two trainable factor matrices, U and V. The authors argue that, unlike LoRA, the Khatri-Rao parameterization can produce full-rank updates and empirically exhibits flatter singular-value spectra (higher effective rank). They support this with a synthetic matrix-approximation benchmark, a theorem (Theorem 3.1) claiming that the Khatri-Rao product of random matrices has full column rank almost surely, and experiments on CLIP vision-language models, VTAB, ImageNet OOD benchmarks, and LLM commonsense reasoning tasks, reporting gains over LoRA, SinLoRA, RandLoRA, and Krona. The paper also reports training-time parity with LoRA and provides code.","tokens_in":27524,"tokens_out":16021,"duration_ms":143157,"significance":"If the empirical results hold across seeds and the theoretical claim is properly established, KRAdapter is a genuinely useful and simple PEFT method: it is competitive in cost, easy to implement, and appears to improve OOD generalization on several benchmarks. The synthetic matrix-approximation study is a valuable controlled way to compare the spectral inductive biases of PEFT methods, and the public code supports reproducibility. However, the causal claim that high effective rank drives the OOD improvements is not cleanly established, and the proof of the central theorem is currently invalid. The contribution is therefore potentially significant but needs substantial revision.","major_comments":[{"comment":"The proof of Theorem 3.1 is incorrect. The key assertion, 'since none of the column vectors of U are multiples of the others and none of the column vectors of V are multiples of the others, we have that the set {U1⊗V1,...,Unk+p⊗Vnk+p} are linearly independent,' is false. A concrete counterexample in R^2 with k=2, din=4 is U=V whose four columns are (1,0), (0,1), (1,1), (1,-1). These columns are pairwise non-collinear in both U and V, but the corresponding four Khatri-Rao columns A1=e1⊗e1, A2=e2⊗e2, A3=(e1+e2)⊗(e1+e2), A4=(e1-e2)⊗(e1-e2) satisfy A3+A4=2(A1+A2), so they are linearly dependent. The theorem itself may be true (e.g., the determinant of U⊙V as a polynomial in the i.i.d. entries is not identically zero, and a full-rank example exists), but the argument given in the paper is not valid. The authors must either supply a correct proof or revise the statement to the weaker claim that the Khatri-Rao parameterization has full-rank matrices in its hypothesis space.","section":"Appendix A.2 (proof of Theorem 3.1)"},{"comment":"There are internal inconsistencies between the main OOD table and the detailed appendix results for the same experiments. For ViT-B/32, Table 2 reports LoRA ID=72.9, OOD=58.2, rgen=0.27, while Table 10 reports LoRA ID=72.16, OOD=58.71, rgen=0.34. Similarly, Table 2 reports RandLoRA ID=73.0, rgen=0.28 while Table 10 reports ID=72.01, rgen=0.31, and Krona ID=72.4, rgen=0.32 vs Table 10 ID=71.88, rgen=0.38. Since both tables are stated to report the same ImageNet OOD evaluation, the discrepancies (particularly in ID accuracy, which should be identical) are unresolved and call into question the reliability of the central OOD comparison. Please reconcile the numbers or clarify whether these are different runs.","section":"Section 4.2.3, Table 2 vs Table 10"},{"comment":"The paper attributes KRAdapter's OOD advantage to its high effective rank, but the comparison never separates rank from update magnitude. Table 2 shows that KRAdapter has both the highest rgen and the smallest nuclear and Frobenius norms among PEFT methods (e.g., ViT-B/32 Frobenius norm 2.3 vs LoRA 4.1 and Krona 4.9; ViT-H/14 Frobenius norm 5.5 vs Krona 9.2). The text itself notes that small update norms have previously been linked to OOD robustness, citing refs. [33,59]. Full fine-tuning on ViT-H/14 has an even smaller norm (1.9) but a lower rgen (0.39 vs 0.48), which suggests norm is not the whole story, but no experiment varies update norm while holding the construction fixed. A controlled experiment (e.g., rescaling the scaling factor α to match Frobenius norms across methods, or applying spectral norm regularization to LoRA) is needed before the 'by construction high effective rank' explanation in the abstract can be regarded as established.","section":"Section 4.2.3, Table 2"},{"comment":"The headline comparisons in the main text are single-run results with no error bars or significance tests. Several reported advantages are small: for example, in Table 1 KRAdapter's ViT-H/14 average is 77.0 vs 76.9 for RandLoRA, and in Table 7 the ViT-H/14 100% average is 90.76 vs 90.81 for RandLoRA (KRAdapter is numerically lower). Without repeated seeds, the claim of consistent superiority is not supported. The GLUE experiments in Appendix H do report mean±std over 5 seeds; the same practice should be applied to the CLIP/VTAB and LLM tables, or the authors should at least provide a variance estimate for the main quantitative claims.","section":"Tables 1, 2, 3, 7"}],"minor_comments":[{"comment":"The paper states that U is initialized to zero and V to Kaiming uniform, but Theorem 3.1 assumes both U and V have i.i.d. Gaussian or uniform entries. At initialization U⊙V is the zero matrix, and after training the entries are not i.i.d. Please clarify that the theorem concerns the hypothesis space of the Khatri-Rao parameterization, not the trained or initialized adapter, and adjust the wording accordingly.","section":"Section 3.2 and Theorem 3.1"},{"comment":"The abstract says KRAdapter 'by construction, tends to produce matrix product with a high effective rank.' Theorem 3.1 only establishes full algebraic column rank, not high effective rank or a flat singular-value spectrum. The flat-spectrum behavior is empirical (Figure 4) rather than a formal guarantee; please restate the claim to distinguish algebraic rank from effective rank.","section":"Abstract and Section 3.4"},{"comment":"The derivative of f(x)=((m/x)+x)n is written incorrectly as f'(x)=n+(-mn/x^2); the correct derivative is n((m/x)+x)^{n-1}(1-m/x^2). The conclusion x=√m is still correct. Also, in the paragraph after Lemma A.3, 'k1 = ⌊dout⌋' should read 'k1 = ⌊√dout⌋'.","section":"Appendix A.1, Lemma A.3"},{"comment":"There are typographical artifacts in this section: '056' appears in the sentence about tail singular values, and 'Table 2 and 3 3' should likely be 'Tables 2 and 3'. Please correct these.","section":"Section 3.5"},{"comment":"The caption reads 'Adaptation performance on VTAB1k's structured datasets,' but the table reports natural, specialized, and structured groups and their average. Please change the caption to reflect the full benchmark.","section":"Table 1 caption"},{"comment":"The metric is described as 'the absolute nuclear reconstruction error (average absolute element-wise singular value difference).' This phrasing is confusing; the nuclear error of a matrix difference is the sum of the singular values of the difference, not an average of element-wise singular-value differences. Please define the metric precisely.","section":"Section 4.1"},{"comment":"The main text says the scaling factor α is 'typically set to 0.1,' but Appendix G states that the LLM experiments use a 'base scaling coefficient of 2 for α weights.' Please clarify how α is set across experiment families.","section":"Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising central idea and a broad empirical study, but the current version has a broken proof of the main theorem, internal inconsistencies between Table 2 and Table 10, and a causal claim that is confounded with update norm. These are fixable within the scope of a major revision, so I do not recommend rejection. I would ask the editor to specifically ensure that the proof of Theorem 3.1 is corrected before further review, since the theorem is a highlighted contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, KRAdapter is genuinely new: the Khatri-Rao product as a PEFT construction is not in the cited prior work, and the full-rank guarantee (Theorem 3.1) plus the spectral comparison against Krona (Figure 4) are real contributions. Second, the central explanatory claim—that high effective rank is why KRAdapter generalizes better OOD—is not established. The reported results are consistent with that story, but they do not test it.\n\nWhat the paper does well: the synthetic matrix-approximation benchmark is a nice way to isolate representational capacity, and KRAdapter does dominate on flat-spectrum targets. The evaluation spans three model scales and multiple benchmarks, and the baselines are independently developed. The Limitations section is honest about the parameter-count issue and low-rank cases. Code is provided. The paper ships a real theorem, even if the proof is rough.\n\nSoft spots, in order of concern. (1) The causal mechanism: Table 2 shows KRAdapter has both the highest effective rank and the smallest update norm. The text itself notes small norms have been linked to OOD robustness. FT has an even smaller norm on ViT-H/14 but a lower rgen, which suggests norm isn't the whole story, but there is no controlled experiment varying norm while holding the construction fixed. The abstract's 'by construction, high effective rank' explanation for the empirical gains is therefore not yet supported. (2) Headline comparisons are mostly single-run. Several margins are under one point (VTAB averages, +0.2 on ViT-H/14). Only GLUE has mean±std. Without seeded runs, 'consistently outperforms' is not defensible. (3) The proof in Appendix A.2 has a load-bearing gap: the claim that given two sets of p linearly independent vectors, the Kronecker products {x_i ⊗ y_i} are independent is false in general, and the proof's final linear-independence step does not follow from the stated premises. The theorem may well be true—the Khatri-Rao product of i.i.d. continuous random matrices should be almost surely full column rank under these dimensions—but the proof needs fixing, not just polishing. (4) 'Compute efficiency of LoRA' is supported only by wall-clock tables; no FLOP or memory accounting.\n\nWho is this for: anyone working on PEFT or fine-tuning large vision-language or LLMs. The method is simple, practical, and likely to be used. With seeded runs, a norm/rank ablation, and a repaired proof, this would be a solid paper. As it stands, it's a good paper with an over-reaching abstract.\n\nSend it to peer review. The contribution is real, and the weaknesses are addressable. A serious referee will ask for exactly the control experiments and the proof fix; the paper deserves that process.","headline":"Khatri-Rao PEFT is a genuine, useful addition with a real full-rank theorem, but the paper's rank-explains-OOD story is not yet supported; the update-norm confound and single-run tables need work.","tokens_in":28125,"tokens_out":2994,"would_cite":true,"duration_ms":27590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KRAdapter builds full-rank fine-tuning updates from the Khatri-Rao product and reports accuracy gains on vision-language models up to 1B parameters and language models up to 8B, especially on unseen commonsense tasks.","keywords":["parameter-efficient fine-tuning","Khatri-Rao product","effective rank","low-rank adaptation","out-of-distribution generalization","vision-language models","large language models","adapter methods"],"falsifier":"Rerun the ViT-H/14 VTAB-1k and ImageNet-shift comparisons with at least five seeds: if KRAdapter's average advantage over RandLoRA or Krona falls within the run-to-run spread, the claim that it consistently outperforms them fails. Alternatively, train LoRA with a spectral penalty matching KRAdapter's update norms on the same OOD suite; if the OOD gain ratio does not approach KRAdapter's, the effective-rank explanation is not sufficient.","tokens_in":27075,"feed_emoji":"🧩","tokens_out":11992,"duration_ms":73540,"temperature":0.7,"pith_summary":"Parameter-efficient fine-tuning normally leans on low-rank updates, and LoRA in particular is limited to updates of rank at most the chosen rank $r$. This paper argues that the real bottleneck is not rank itself but effective rank — whether the update actually uses all its singular directions — and proposes KRAdapter, which builds the weight update as a Khatri-Rao product, a column-wise Kronecker product, of two small matrices. The paper proves that with random initialization this product is almost surely full column rank, and shows empirically that its singular values decay more slowly than those of other full-rank adapters. Across CLIP vision-language models up to 1B parameters and large language models up to 8B parameters, KRAdapter matches LoRA's memory and compute while improving accuracy, especially on out-of-distribution and unseen commonsense reasoning tasks. If the paper is right, full-rank, high-effective-rank fine-tuning is available at low-rank cost, and that is what drives robust adaptation.","feed_headline":"One matrix product yields full-rank fine-tuning at LoRA cost","feed_subtitle":"KRAdapter keeps LoRA's memory and speed while improving out-of-distribution accuracy on models up to 8B.","key_machinery":"The central object is the Khatri-Rao product $U \\odot V$ of two matrices $U \\in \\mathbb{R}^{k_1 \\times d_{\\mathrm{in}}}$ and $V \\in \\mathbb{R}^{k_2 \\times d_{\\mathrm{in}}}$, which yields a $k_1 k_2 \\times d_{\\mathrm{in}}$ matrix whose $j$-th column is the Kronecker product of the $j$-th columns of $U$ and $V$. Choosing $k_1 \\approx k_2 \\approx \\sqrt{d_{\\mathrm{out}}}$ gives about $2\\sqrt{d_{\\mathrm{out}}} d_{\\mathrm{in}}$ trainable parameters, comparable to a rank-16 to rank-32 LoRA. The mechanism that carries the argument is Theorem 3.1: with random initialization the Khatri-Rao product has full column rank $d_{\\mathrm{in}}$ almost surely, and unlike a Kronecker product, its singular-value spectrum stays comparatively flat, so the update has high effective rank rather than merely high mathematical rank. The paper uses this construction as a drop-in adapter on attention projections, with $U$ initialized to zero and $V$ to a uniform initialization scaled for the factor shape.","core_discovery":"On the paper's own terms, the central discovery is that the Khatri-Rao product provides a parameter-efficient way to build weight updates that are genuinely full rank and spectrally flat. Theorem 3.1 states that for $U,V \\in \\mathbb{R}^{k \\times d_{\\mathrm{in}}}$ with i.i.d. Gaussian or uniform entries and $k \\le d_{\\mathrm{in}} \\le k^2$, one has $\\mathrm{rank}(U \\odot V) = d_{\\mathrm{in}}$ almost surely, in contrast to LoRA, whose update $BA$ has rank at most $k$. In a controlled matrix-reconstruction benchmark, KRAdapter approximates targets with flat spectra and high-frequency content better than LoRA, SinLoRA, RandLoRA, and Krona, and after fine-tuning its updates have higher effective rank and smaller norms. These spectral properties are linked to the paper's main empirical claim: KRAdapter improves accuracy on vision-language and commonsense reasoning benchmarks, particularly out-of-distribution, while keeping LoRA-level compute and memory. The paper also states the limitation that KRAdapter trains more parameters than the most extreme LoRA configurations and does not beat LoRA on very low-rank target matrices.","pith_inferences":["The paper does not separate the effects of high effective rank from the effects of the unusually small update norms it also reports; a direct test would be to train LoRA with a spectral penalty that matches KRAdapter's update norms and compare OOD gains on the same suite.","The full-rank theorem is stated for randomly initialized $U$ and $V$; the high effective ranks observed after training are empirical. Whether gradient descent preserves full rank throughout training is a separate question the paper leaves open.","Because KRAdapter reconstructs high-frequency and whitened matrices well, it may transfer to continual-learning or domain-adaptation settings where LoRA's low-rank bias is known to cause forgetting; the paper does not test this directly."],"forward_implications":["LoRA's rank ceiling is not a necessary price of parameter efficiency: a full-rank update with a flat spectrum can be trained with comparable memory and speed.","Out-of-distribution robustness in PEFT can be improved by choosing the update's spectrum rather than only its parameter count; KRAdapter reports the highest OOD-to-ID gain ratio on the ImageNet shift suite.","On commonsense reasoning after 4-bit quantized fine-tuning, full-rank high-effective-rank adapters, with KRAdapter leading, generalize better to unseen tasks than rank-restricted LoRA.","Effective rank, not nominal rank, is the quantity that separates strong full-rank adapters from weak ones; KRAdapter claims the highest post-training effective ranks across ViT-B/32, ViT-L/14, Llama3.1-8B, and Qwen2.5-7B.","For targets that are genuinely low rank, or under extreme parameter scarcity, KRAdapter does not improve over LoRA, so the method's advantage is specific to tasks that need broad spectral coverage."],"supporting_citations":[{"why":"Defines LoRA, the low-rank baseline whose rank ceiling and spectral behavior KRAdapter is designed to overcome.","marker":"[24]"},{"why":"Introduces RandLoRA, the full-rank random-basis method that motivates the full-rank argument and serves as the closest baseline in most comparisons.","marker":"[2]"},{"why":"Introduces Krona, the Kronecker-product adapter whose spectral profile is explicitly contrasted with KRAdapter's.","marker":"[16]"},{"why":"Introduces SinLoRA, a sine-activated full-rank adapter used as a baseline in the reconstruction and fine-tuning experiments.","marker":"[26]"},{"why":"Defines effective rank, the spectral measure the paper argues is the real differentiator among full-rank PEFT methods.","marker":"[47]"},{"why":"Supplies the CLIP vision-language models used for the image classification and OOD experiments.","marker":"[44]"},{"why":"Supplies the VTAB-1k benchmark used to evaluate adaptation on specialized and structured tasks.","marker":"[62]"}],"fun_headline_variants":["KRAdapter: full-rank updates at LoRA cost","Beat LoRA's rank ceiling with Khatri-Rao product","Full-rank PEFT without LoRA's rank limit","Khatri-Rao product yields high effective rank for PEFT","LoRA cost, full-rank power: KRAdapter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported performance advantages over the other full-rank methods are real rather than seed noise, because most headline comparisons are single runs with differences as small as a few tenths of a point.","fun_headline_variants_meta":{"raw":{"variants":["KRAdapter: full-rank updates at LoRA cost","Beat LoRA's rank ceiling with Khatri-Rao product","Full-rank PEFT without LoRA's rank limit","Khatri-Rao product yields high effective rank for PEFT","LoRA cost, full-rank power: KRAdapter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3568,"prompt_tokens":999,"completion_tokens":2569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2488}},"tokens_in":615,"tokens_out":2569,"duration_ms":17614,"temperature":1.0,"reasoning_tokens":2488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:19:49.103832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the ViT-H/14 VTAB-1k and ImageNet-shift comparisons with at least five seeds: if KRAdapter's average advantage over RandLoRA or Krona falls within the run-to-run spread, the claim that it consistently outperforms them fails. Alternatively, train LoRA with a spectral penalty matching KRAdapter's update norms on the same OOD suite; if the OOD gain ratio does not approach KRAdapter's, the effective-rank explanation is not sufficient.","supporting_citations":[{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"Defines LoRA, the low-rank baseline whose rank ceiling and spectral behavior KRAdapter is designed to overcome."},{"cited_title":"Zhang, Hemanth Saratchandran, Cris- tian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad","cited_arxiv_id":null,"evidence_quote":"Introduces RandLoRA, the full-rank random-basis method that motivates the full-rank argument and serves as the closest baseline in most comparisons."},{"cited_title":"Krona: Parameter efficient tuning with kronecker adapter","cited_arxiv_id":null,"evidence_quote":"Introduces Krona, the Kronecker-product adapter whose spectral profile is explicitly contrasted with KRAdapter's."},{"cited_title":"The effective rank: A mea- sure of effective dimensionality","cited_arxiv_id":null,"evidence_quote":"Defines effective rank, the spectral measure the paper argues is the real differentiator among full-rank PEFT methods."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP vision-language models used for the image classification and OOD experiments."}],"review_version":1}