Pith. sign in

REVIEW 4 major objections 7 minor 65 references

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KRAdapter builds full-rank fine-tuning updates from the Khatri-Rao product and reports accuracy gains on vision-language models up to 1B parameters and language models up to 8B, especially on unseen commonsense tasks.

desk verdict Khatri-Rao PEFT is a genuine, useful addition with a real full-rank theorem, but the paper's rank-explains-OOD story is not yet supported; the update-norm confound and single-run tables need work. read the letter →

arxiv 2508.00230 v1 pith:ZBSFJJDA submitted 2025-08-01 cs.LG cs.CLcs.CV

Paul Albert , Frederic Z. Zhang , Hemanth Saratchandran , Anton van den Hengel , Ehsan Abbasnejad This is my paper · ORCID
classification cs.LGcs.CLcs.CV
keywords parameter-efficientfine-tuningKhatri-Raoproducteffectiveranklow-rankadaptationout-of-distributiongeneralizationvision-languagemodelslargelanguageadaptermethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Parameter-efficient fine-tuning normally leans on low-rank updates, and LoRA in particular is limited to updates of rank at most the chosen rank $r$. This paper argues that the real bottleneck is not rank itself but effective rank — whether the update actually uses all its singular directions — and proposes KRAdapter, which builds the weight update as a Khatri-Rao product, a column-wise Kronecker product, of two small matrices. The paper proves that with random initialization this product is almost surely full column rank, and shows empirically that its singular values decay more slowly than those of other full-rank adapters. Across CLIP vision-language models up to 1B parameters and large language models up to 8B parameters, KRAdapter matches LoRA's memory and compute while improving accuracy, especially on out-of-distribution and unseen commonsense reasoning tasks. If the paper is right, full-rank, high-effective-rank fine-tuning is available at low-rank cost, and that is what drives robust adaptation.

What carries the argument

The central object is the Khatri-Rao product $U \odot V$ of two matrices $U \in \mathbb{R}^{k_1 \times d_{\mathrm{in}}}$ and $V \in \mathbb{R}^{k_2 \times d_{\mathrm{in}}}$, which yields a $k_1 k_2 \times d_{\mathrm{in}}$ matrix whose $j$-th column is the Kronecker product of the $j$-th columns of $U$ and $V$. Choosing $k_1 \approx k_2 \approx \sqrt{d_{\mathrm{out}}}$ gives about $2\sqrt{d_{\mathrm{out}}} d_{\mathrm{in}}$ trainable parameters, comparable to a rank-16 to rank-32 LoRA. The mechanism that carries the argument is Theorem 3.1: with random initialization the Khatri-Rao product has full column rank $d_{\mathrm{in}}$ almost surely, and unlike a Kronecker product, its singular-value spectrum stays comparatively flat, so the update has high effective rank rather than merely high mathematical rank. The paper uses this construction as a drop-in adapter on attention projections, with $U$ initialized to zero and $V$ to a uniform initialization scaled for the factor shape.

What would settle it

Rerun the ViT-H/14 VTAB-1k and ImageNet-shift comparisons with at least five seeds: if KRAdapter's average advantage over RandLoRA or Krona falls within the run-to-run spread, the claim that it consistently outperforms them fails. Alternatively, train LoRA with a spectral penalty matching KRAdapter's update norms on the same OOD suite; if the OOD gain ratio does not approach KRAdapter's, the effective-rank explanation is not sufficient.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the Khatri-Rao product provides a parameter-efficient way to build weight updates that are genuinely full rank and spectrally flat. Theorem 3.1 states that for $U,V \in \mathbb{R}^{k \times d_{\mathrm{in}}}$ with i.i.d. Gaussian or uniform entries and $k \le d_{\mathrm{in}} \le k^2$, one has $\mathrm{rank}(U \odot V) = d_{\mathrm{in}}$ almost surely, in contrast to LoRA, whose update $BA$ has rank at most $k$. In a controlled matrix-reconstruction benchmark, KRAdapter approximates targets with flat spectra and high-frequency content better than LoRA, SinLoRA, RandLoRA, and Krona, and after fine-tuning its updates have higher effective rank and smaller norms. These spectral properties are linked to the paper's main empirical claim: KRAdapter improves accuracy on vision-language and commonsense reasoning benchmarks, particularly out-of-distribution, while keeping LoRA-level compute and memory. The paper also states the limitation that KRAdapter trains more parameters than the most extreme LoRA configurations and does not beat LoRA on very low-rank target matrices.

Load-bearing premise

The load-bearing premise is that the reported performance advantages over the other full-rank methods are real rather than seed noise, because most headline comparisons are single runs with differences as small as a few tenths of a point.

Editorial extensions

If this is right

  • LoRA's rank ceiling is not a necessary price of parameter efficiency: a full-rank update with a flat spectrum can be trained with comparable memory and speed.
  • Out-of-distribution robustness in PEFT can be improved by choosing the update's spectrum rather than only its parameter count; KRAdapter reports the highest OOD-to-ID gain ratio on the ImageNet shift suite.
  • On commonsense reasoning after 4-bit quantized fine-tuning, full-rank high-effective-rank adapters, with KRAdapter leading, generalize better to unseen tasks than rank-restricted LoRA.
  • Effective rank, not nominal rank, is the quantity that separates strong full-rank adapters from weak ones; KRAdapter claims the highest post-training effective ranks across ViT-B/32, ViT-L/14, Llama3.1-8B, and Qwen2.5-7B.
  • For targets that are genuinely low rank, or under extreme parameter scarcity, KRAdapter does not improve over LoRA, so the method's advantage is specific to tasks that need broad spectral coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separate the effects of high effective rank from the effects of the unusually small update norms it also reports; a direct test would be to train LoRA with a spectral penalty that matches KRAdapter's update norms and compare OOD gains on the same suite.
  • The full-rank theorem is stated for randomly initialized $U$ and $V$; the high effective ranks observed after training are empirical. Whether gradient descent preserves full rank throughout training is a separate question the paper leaves open.
  • Because KRAdapter reconstructs high-frequency and whitened matrices well, it may transfer to continual-learning or domain-adaptation settings where LoRA's low-rank bias is known to cause forgetting; the paper does not test this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces KRAdapter, a parameter-efficient fine-tuning (PEFT) method that constructs weight updates as a Khatri-Rao product of two trainable factor matrices, U and V. The authors argue that, unlike LoRA, the Khatri-Rao parameterization can produce full-rank updates and empirically exhibits flatter singular-value spectra (higher effective rank). They support this with a synthetic matrix-approximation benchmark, a theorem (Theorem 3.1) claiming that the Khatri-Rao product of random matrices has full column rank almost surely, and experiments on CLIP vision-language models, VTAB, ImageNet OOD benchmarks, and LLM commonsense reasoning tasks, reporting gains over LoRA, SinLoRA, RandLoRA, and Krona. The paper also reports training-time parity with LoRA and provides code.

Significance. If the empirical results hold across seeds and the theoretical claim is properly established, KRAdapter is a genuinely useful and simple PEFT method: it is competitive in cost, easy to implement, and appears to improve OOD generalization on several benchmarks. The synthetic matrix-approximation study is a valuable controlled way to compare the spectral inductive biases of PEFT methods, and the public code supports reproducibility. However, the causal claim that high effective rank drives the OOD improvements is not cleanly established, and the proof of the central theorem is currently invalid. The contribution is therefore potentially significant but needs substantial revision.

major comments (4)
  1. [Appendix A.2 (proof of Theorem 3.1)] The proof of Theorem 3.1 is incorrect. The key assertion, 'since none of the column vectors of U are multiples of the others and none of the column vectors of V are multiples of the others, we have that the set {U1⊗V1,...,Unk+p⊗Vnk+p} are linearly independent,' is false. A concrete counterexample in R^2 with k=2, din=4 is U=V whose four columns are (1,0), (0,1), (1,1), (1,-1). These columns are pairwise non-collinear in both U and V, but the corresponding four Khatri-Rao columns A1=e1⊗e1, A2=e2⊗e2, A3=(e1+e2)⊗(e1+e2), A4=(e1-e2)⊗(e1-e2) satisfy A3+A4=2(A1+A2), so they are linearly dependent. The theorem itself may be true (e.g., the determinant of U⊙V as a polynomial in the i.i.d. entries is not identically zero, and a full-rank example exists), but the argument given in the paper is not valid. The authors must either supply a correct proof or revise the statement to the weaker claim that the Khatri-Rao parameterization has full-rank matrices in its hypothesis space.
  2. [Section 4.2.3, Table 2 vs Table 10] There are internal inconsistencies between the main OOD table and the detailed appendix results for the same experiments. For ViT-B/32, Table 2 reports LoRA ID=72.9, OOD=58.2, rgen=0.27, while Table 10 reports LoRA ID=72.16, OOD=58.71, rgen=0.34. Similarly, Table 2 reports RandLoRA ID=73.0, rgen=0.28 while Table 10 reports ID=72.01, rgen=0.31, and Krona ID=72.4, rgen=0.32 vs Table 10 ID=71.88, rgen=0.38. Since both tables are stated to report the same ImageNet OOD evaluation, the discrepancies (particularly in ID accuracy, which should be identical) are unresolved and call into question the reliability of the central OOD comparison. Please reconcile the numbers or clarify whether these are different runs.
  3. [Section 4.2.3, Table 2] The paper attributes KRAdapter's OOD advantage to its high effective rank, but the comparison never separates rank from update magnitude. Table 2 shows that KRAdapter has both the highest rgen and the smallest nuclear and Frobenius norms among PEFT methods (e.g., ViT-B/32 Frobenius norm 2.3 vs LoRA 4.1 and Krona 4.9; ViT-H/14 Frobenius norm 5.5 vs Krona 9.2). The text itself notes that small update norms have previously been linked to OOD robustness, citing refs. [33,59]. Full fine-tuning on ViT-H/14 has an even smaller norm (1.9) but a lower rgen (0.39 vs 0.48), which suggests norm is not the whole story, but no experiment varies update norm while holding the construction fixed. A controlled experiment (e.g., rescaling the scaling factor α to match Frobenius norms across methods, or applying spectral norm regularization to LoRA) is needed before the 'by construction high effective rank' explanation in the abstract can be regarded as established.
  4. [Tables 1, 2, 3, 7] The headline comparisons in the main text are single-run results with no error bars or significance tests. Several reported advantages are small: for example, in Table 1 KRAdapter's ViT-H/14 average is 77.0 vs 76.9 for RandLoRA, and in Table 7 the ViT-H/14 100% average is 90.76 vs 90.81 for RandLoRA (KRAdapter is numerically lower). Without repeated seeds, the claim of consistent superiority is not supported. The GLUE experiments in Appendix H do report mean±std over 5 seeds; the same practice should be applied to the CLIP/VTAB and LLM tables, or the authors should at least provide a variance estimate for the main quantitative claims.
minor comments (7)
  1. [Section 3.2 and Theorem 3.1] The paper states that U is initialized to zero and V to Kaiming uniform, but Theorem 3.1 assumes both U and V have i.i.d. Gaussian or uniform entries. At initialization U⊙V is the zero matrix, and after training the entries are not i.i.d. Please clarify that the theorem concerns the hypothesis space of the Khatri-Rao parameterization, not the trained or initialized adapter, and adjust the wording accordingly.
  2. [Abstract and Section 3.4] The abstract says KRAdapter 'by construction, tends to produce matrix product with a high effective rank.' Theorem 3.1 only establishes full algebraic column rank, not high effective rank or a flat singular-value spectrum. The flat-spectrum behavior is empirical (Figure 4) rather than a formal guarantee; please restate the claim to distinguish algebraic rank from effective rank.
  3. [Appendix A.1, Lemma A.3] The derivative of f(x)=((m/x)+x)n is written incorrectly as f'(x)=n+(-mn/x^2); the correct derivative is n((m/x)+x)^{n-1}(1-m/x^2). The conclusion x=√m is still correct. Also, in the paragraph after Lemma A.3, 'k1 = ⌊dout⌋' should read 'k1 = ⌊√dout⌋'.
  4. [Section 3.5] There are typographical artifacts in this section: '056' appears in the sentence about tail singular values, and 'Table 2 and 3 3' should likely be 'Tables 2 and 3'. Please correct these.
  5. [Table 1 caption] The caption reads 'Adaptation performance on VTAB1k's structured datasets,' but the table reports natural, specialized, and structured groups and their average. Please change the caption to reflect the full benchmark.
  6. [Section 4.1] The metric is described as 'the absolute nuclear reconstruction error (average absolute element-wise singular value difference).' This phrasing is confusing; the nuclear error of a matrix difference is the sum of the singular values of the difference, not an average of element-wise singular-value differences. Please define the metric precisely.
  7. [Appendix G] The main text says the scaling factor α is 'typically set to 0.1,' but Appendix G states that the LLM experiments use a 'base scaling coefficient of 2 for α weights.' Please clarify how α is set across experiment families.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KRAdapter's empirical claims are benchmarked externally and its mathematical lemmas are self-contained.

full rationale

The derivation chain is not circular. Theorem 3.1 establishes full column rank of a Khatri-Rao product of random matrices, and its proof in Appendix A.2 restates Lemma A.4 with a proof rather than merely citing the same authors' RandLoRA paper [2]; the lemma is a standard fact about continuous random vectors and does not assume the theorem's conclusion, so the self-citation is not load-bearing. The synthetic matrix-approximation experiments in Section 4.1 are controlled external benchmarks, and the real-data evaluations on VTAB, ImageNet-shift OOD sets, commonsense reasoning, and GLUE compare against independently developed baselines (LoRA, SinLoRA, Krona, RandLoRA) on public datasets. The measurements of effective rank and update norms in Table 4 and Table 2 are post-hoc descriptions of trained adapters, not fitted parameters renamed as predictions. The abstract's 'by construction' high-effective-rank wording describes the random Khatri-Rao construction, not an identity with the empirical OOD outcome; the possible confound between update norm and effective rank in Section 4.2.3 is a causal-identification weakness rather than a circular reduction. The proof of Theorem 3.1 has a rigor gap in its inference about linear independence of cross-batch Kronecker columns, but that is a correctness concern, not a circularity. No equation or fitted value in the paper makes the target result equal to its inputs by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on three chosen constants (alpha=0.1, the Kaiming slope chosen by experiment, and the k1 block-shape allocation), four background assumptions (the standard random-vector independence lemma, the transfer of the full-rank guarantee to the trained updates despite a U=0 initialization, the motivational SVD identity that the deployed method does not actually use, and the effective-rank hypothesis treated as a cause), and no invented entities.

free parameters (3)
  • scaling factor alpha = 0.1
    Global update scale (Section 3.2), set like LoRA's alpha; not fitted per layer.
  • Kaiming negative slope for V initialization = sqrt(1/k1)
    Chosen 'empirically ... optimal for convergence' (Section 3.2); a hand-tuned initialization choice.
  • block shape k1 = k1 = k2 = sqrt(dout) default; ablations use dout^(1/2), dout^(1/3), dout^(1/4)
    Parameter allocation that minimizes count (Lemma A.3) and drives the low/mid/high scaling study (Section 4.4).
assumptions (4)
  • standard math Random i.i.d. continuous vectors are almost surely linearly independent (Lemma A.4, credited to the authors' RandLoRA paper [2]).
    Invoked in the proof of Theorem 3.1 (Appendix A.2) to get batch-wise independence of the columns of U and V.
  • domain assumption The full-rank guarantee certifies the updates KRAdapter actually produces during training.
    Theorem 3.1 assumes both U and V have i.i.d. random entries, but the method sets U=0 and V Kaiming-uniform (Section 3.2), so the almost-sure full-rank statement holds for the parameterization family, not for the actual initialization (where U ⊙ V = 0). Table 4 supplies post-hoc evidence (high effective rank after training) that the paper does not derive from the theorem.
  • standard math vec(W) = (V ⊙ U) sigma for any rank-r W (Theorem A.2) motivates the Khatri-Rao parameterization.
    True by SVD, but the deployed method trains no sigma and restricts U to k1 x din with k1*k2 = dout, so the universal-representation result does not transfer to KRAdapter's actual hypothesis class; used only as motivation in Appendices A.1-A.2.
  • domain assumption High effective rank of the weight update causes better out-of-distribution generalization.
    The paper's causal hypothesis (Sections 1, 2.5, 4.2.3), supported by correlation (Table 4 vs Table 2) but confounded with update norm; not causally isolated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product." pith.science (2026). https://pith.science/paper/ZBSFJJDA

@misc{pith2026250800230,
  author       = {Pith},
  title        = {Pith review of: Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBSFJJDA}},
  note         = {Machine review of arXiv:2508.00230}
}
read the original abstract

Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large pre-trained models. Amongst PEFT methods, low-rank adaptation (LoRA) has achieved notable success. However, recent studies have highlighted its limitations compared against full-rank alternatives, particularly when applied to multimodal and large language models. In this work, we present a quantitative comparison amongst full-rank and low-rank PEFT methods using a synthetic matrix approximation benchmark with controlled spectral properties. Our results confirm that LoRA struggles to approximate matrices with relatively flat spectrums or high frequency components -- signs of high effective ranks. To this end, we introduce KRAdapter, a novel PEFT algorithm that leverages the Khatri-Rao product to produce weight updates, which, by construction, tends to produce matrix product with a high effective rank. We demonstrate performance gains with KRAdapter on vision-language models up to 1B parameters and on large language models up to 8B parameters, particularly on unseen common-sense reasoning tasks. In addition, KRAdapter maintains the memory and compute efficiency of LoRA, making it a practical and robust alternative to fine-tune billion-scale parameter models.

Figures

Figures reproduced from arXiv: 2508.00230 by the authors.

Figure 1
Figure 1. A visualization of the capacity of various PEFT methods to approximate the spectrum of different types of weight matrices. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Tuning CLIP’s vision and language backbone for image classification. Accuracy (%) averaged over 11 datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Increasing the number of trainable parameters in PEFT algorithms. Accuracy (%) averaged over 11 datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: We compare the singular value distribution and effective rank resulting from a parameter-efficient construction of a matrix of set [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Toy experiment. We evaluate the capacity of PEFT methods to produce specific types of weight matrices. We report the generated [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 48 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023. 1, 2

    arXiv 2023

  2. [2]

    Zhang, Hemanth Saratchandran, Cris- tian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad

    Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cris- tian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. RandLoRA: Full rank parameter-efficient fine- tuning of large models. In International Conference on Learn- ing Representations (ICLR), 2025. 1, 2, 3, 5, 8, 13, 16, 21

    work page 2025

  3. [3]

    Deepmind lab

    Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich K¨uttler, Andrew Lefrancq, Si- mon Green, V´ıctor Vald´es, Amir Sadik, et al. Deepmind lab. arXiv: 1612.03801, 2016. 18

    arXiv 2016

  4. [4]

    Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D

    Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, and Nicholas D. Lane. Flower: A Friendly Federated Learning Research Framework. arxiv :2007.14390,

    arXiv 2007

  5. [5]

    Piqa: Reasoning about physical commonsense in natural lan- guage

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural lan- guage. In Proceedings of the AAAI conference on Artificial Intelligence (AAAI), 2020. 20

    work page 2020

  6. [6]

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representa- tions. In International Conference on Machine Learning (ICML), 2020. 18

    work page 2020

  7. [7]

    Transferability vs

    Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral pe- nalization for adversarial domain adaptation. In International Conference on Machine Learning (ICML), 2019. 1

    work page 2019

  8. [8]

    Remote Sens- ing Image Scene Classification: Benchmark and State of the Art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sens- ing Image Scene Classification: Benchmark and State of the Art. Proceedings of the IEEE, 2017. 18

    work page 2017

Show all 65 references
  1. [9]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  2. [10]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing Textures in the Wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 17

    2014

  3. [11]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. 20

    1905 arXiv

  4. [12]

    Think you have solved question answering? try arc, the ai2 rea- soning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 rea- soning challenge. arXiv preprint arXiv:1803.05457, 2018. 21

    2018 arXiv

  5. [13]

    An image is worth 16x16 words: Transform- ers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. InInternational ...

    2021

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [15]

    Dia- betic retinopathy detection (2015)

    Emma Dugas, Jorge Jared, and Will Cukierski. Dia- betic retinopathy detection (2015). URL https://kaggle. com/competitions/diabetic-retinopathy-detection, 2015. 18

    2015

  8. [16]

    Krona: Parameter efficient tuning with kronecker adapter

    Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. In Advances in Neu- ral Information Processing Systems Workshops (NeurIPSW),

  9. [17]

    Geiger, P

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets Robotics: The KITTI Dataset. International Journal of Robotics Research (IJRR), 2013. 18

    2013

  10. [18]

    Parameter-efficient fine-tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 ,

    arXiv

  11. [19]

    Lora+: Effi- cient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Effi- cient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024. 2

    2024

  12. [20]

    Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015. 4

    2015

  13. [21]

    Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. In International Geoscience and Remote Sens- ing Symposium (IGARSS), 2018. 18

    2018

  14. [22]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Co...

    2021

  15. [23]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271, 2021. 6, 18

    2021

  16. [24]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2022. 1, 2, 3, 4, 5

    2022

  17. [25]

    Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. Llm-adapters: An adapter family for parameter- efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023. 6, 21

    2023 arXiv

  18. [26]

    Sine Activated Low-Rank Matrices for Parameter Efficient Learning

    Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey. Sine Activated Low-Rank Matrices for Parameter Efficient Learning. arXiv:2403.19243, 2024. 1, 2, 3, 5

    2024 arXiv

  19. [27]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 18

    2017

  20. [28]

    NOLA: Com- pressing LoRA using Linear Combination of Random Basis

    Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Noora- linejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Com- pressing LoRA using Linear Combination of Random Basis. In International Conference on Learning Representations (ICLR), 2024. 3

    2024

  21. [29]

    Vera: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. In In- ternational Conference on Learning Representations (ICLR),

  22. [30]

    Krizhevsky and G

    A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6, 17

    2009

  23. [31]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classifi- cation with deep convolutional neural networks. In Advances in neural information processing systems (NeurIPS) , 2012. 18

    2012

  24. [32]

    Learning methods for generic object recognition with invariance to pose and lighting

    Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Conference on Computer Vision and Pattern Recognition (CVPR), 2004. 18

    2004

  25. [33]

    Learning continually by spectral regularization

    Alex Lewandowski, Michał Bortkiewicz, Saurabh Kumar, Andr´as Gy ¨orgy, Dale Schuurmans, Mateusz Ostaszewski, and Marlos C Machado. Learning continually by spectral regularization. In International Conference on Learning Rep- resentations (ICLR), 2025. 6

    2025

  26. [34]

    Caltech 101, 2022

    Fei-Fei Li, Marco Andreeto, Marc’Aurelio Ranzato, and Pietro Perona. Caltech 101, 2022. 17

    2022

  27. [35]

    SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eun- sol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors. In International Conference on Machine Learning ...

    2024

  28. [36]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024. 2, 4, 6

    2024

  29. [37]

    RoBERTa: A robustly optimized BERT pretraining approach

    Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis, L Zettlemoyer, and V Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692,

    1907 arXiv

  30. [38]

    dSprites: Disentanglement testing Sprites dataset

    Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement testing Sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017. 18

    2017

  31. [39]

    Pissa: Prin- cipal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Prin- cipal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems (NeurIPS), 2024. 3

    2024

  32. [40]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sab- harwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. 21

    2018 arXiv

  33. [41]

    Spectral normalization for generative adver- sarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adver- sarial networks. In Advances in Neural Information Process- ing Systems (NeurIPS), 2018. 2

    2018

  34. [42]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS work- shop on deep learning and unsupervised feature learning (NeurIPSW), 2011. 18

    2011

  35. [43]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 18

    2012

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML),

  37. [45]

    On the expressive power of deep neural networks

    Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In International Conference on Machine Learning (ICML), 2017. 2

    2017

  38. [46]

    Do imagenet classifiers generalize to im- agenet? In International Conference on Machine Learning (ICML), pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International Conference on Machine Learning (ICML), pages 5389–5400. PMLR, 2019. 6, 18

    2019

  39. [47]

    The effective rank: A mea- sure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A mea- sure of effective dimensionality. In European Signal Process- ing Conference, 2007. 1, 3, 16

    2007

  40. [48]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM ,

  41. [49]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. 20

    1904 arXiv

  42. [50]

    Singular Value Penalization and Semantic Data Augmenta- tion for Fully Test-Time Adaptation

    Houcheng Su, Daixian Liu, Mengzhu Wang, and Wei Wang. Singular Value Penalization and Semantic Data Augmenta- tion for Fully Test-Time Adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), 2024. 1

    2024

  43. [51]

    Reclip: A strong zero-shot baseline for referring expression comprehen- sion

    Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehen- sion. In Association for Computational Linguistics (ACL) ,

  44. [52]

    HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

    2024

  45. [53]

    Rotation Equivariant CNNs for Digital Pathology

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Co- hen, and Max Welling. Rotation Equivariant CNNs for Digital Pathology. arxiv: 1806.03962, 2018. 18

    2018 arXiv

  46. [54]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language under- standing. In International Conference on Learning Represen- tations (ICLR), 2019. 22

    2019

  47. [55]

    Low-rank adaptation method for wav2vec2-based fake audio detection

    Chenglong Wang, Jiangyan Yi, Xiaohui Zhang, Jianhua Tao, Le Xu, and Ruibo Fu. Low-rank adaptation method for wav2vec2-based fake audio detection. In International Joint Conference on Artificial Intelligence Workshops (IJCAIW),

  48. [56]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), pages 10506–10518, 2019. 6, 18

    2019

  49. [57]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning

    Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. arXiv:2406.09044, 2024. 3

    2024 arXiv

  50. [58]

    Sun database: Large-scale scene recog- nition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. In Conference on Computer Vision and Pattern Recognition (CVPR), 2010. 18

    2010

  51. [59]

    Spectral norm regular- ization for improving the generalizability of deep learning

    Yuichi Yoshida and Takeru Miyato. Spectral norm regular- ization for improving the generalizability of deep learning. arXiv:1705.10941, 2017. 2, 6

    2017 arXiv

  52. [60]

    Low-Rank Few-Shot Adaptation of Vision-Language Models

    Maxime Zanella and Ismail Ben Ayed. Low-Rank Few-Shot Adaptation of Vision-Language Models. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR),

  53. [61]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. 20

    1905 arXiv

  54. [62]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv: 191...

    1910 arXiv

  55. [63]

    Knowledge Composition using Task Vectors with Learned Anisotropic Scaling

    Frederic Z Zhang, Paul Albert, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. Knowledge Composition using Task Vectors with Learned Anisotropic Scaling. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2024. 2, 4, 15, 16

    2024

  56. [64]

    AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), 2023. 4

    2023

  57. [65]

    Multi-lora composition for image generation

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. Transactions on Machine Learning Research (TMLR), 2024. 2 A. Mathematical supplement A.1. Minimizing the trainable ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.