Singular Value Decomposition on Kronecker Adaptation for Large Language Model

Peng Qu; Yee Hin Chong

REVIEW 5 major objections 3 minor 15 references

Singular Value Decomposition on Kronecker Adaptation for Large Language Model

T0 review · 5 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SoKA represents weight updates as sums of SVD-initialized Kronecker products with dynamic rank pruning, cutting trainable parameters on LLaMA2-7B to 0.99M while matching PiSSA.

desk verdict Incremental combination of known ideas whose abstract overclaims and whose KPSVD dimension slip means the parameter saving actually comes from rank selection, not Kronecker structure; still a plausible idea worth a revised look. read the letter →

arxiv 2506.15251 v1 pith:6PEHA6AI submitted 2025-06-18 cs.LG cs.AI

Yee Hin Chong , Peng Qu This is my paper · ORCID

classification cs.LGcs.AI

keywords parameter-efficientfine-tuningKroneckerproductsingularvaluedecompositionlow-rankadaptationdynamicrankselectionspectrum-awarepruningLLaMA-2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SoKA, a parameter-efficient fine-tuning method that writes each weight update as a sum of Kronecker products whose factors come from a singular value decomposition of the reshaped pretrained weight matrix. The claim is that this SVD-based initialization captures the principal directions of the update, while a spectrum-aware rank-selection rule prunes negligible components, so adaptation needs fewer trainable parameters than LoRA or PiSSA without sacrificing accuracy. On LLaMA2-7B over GSM8K, MATH, and MBPP, SoKA uses 0.99 million trainable parameters, about 25% fewer than the 1.33 million used by LoRA and PiSSA, and the reported scores trail PiSSA by less than one point on each benchmark while beating LoRA. The authors also report faster convergence and smaller gradient norms than PiSSA. If these results hold, SoKA is a more parameter-efficient alternative to SVD-initialized low-rank adaptation at comparable accuracy.

What carries the argument

The load-bearing object is Kronecker-Product SVD (KPSVD): reshape the pretrained weight matrix, take its truncated SVD, then reshape each singular vector into a small matrix so that each singular triple becomes one Kronecker-product term $\sigma_k U_k\otimes V_k$. This converts a dense SVD into a structured adapter with $r(mn+pq+1)$ parameters instead of LoRA's $2Nr_{\mathrm{lora}}$, and it makes the forward pass cheap through the identity $(U_k\otimes V_k)\operatorname{vec}(X)=\operatorname{vec}(V_k X U_k^\top)$. The supporting mechanism is dynamic rank selection, which sets the per-layer rank to $\min(r_{\mathrm{energy}}, r_{\mathrm{elbow}})$ using cumulative singular-value energy and the largest successive gap in the spectrum.

What would settle it

Train SoKA and a randomly initialized Kronecker adapter on a synthetic task whose target update is deliberately orthogonal to the top singular subspace of the reshaped pretrained matrix; if SoKA's convergence and accuracy advantage disappears, the initialization claim is falsified. A simpler ablation would keep the same rank-selection rule but replace KPSVD initialization with random factors, isolating how much of the gain comes from the SVD rather than from the Kronecker structure.

Watch

Extended reading notes

Core claim

SoKA's central discovery is that a Kronecker-structured adapter initialized by a truncated SVD of the reshaped weight matrix can carry most of the adaptation signal with fewer parameters than a plain low-rank adapter. For an $N\times N$ weight matrix with $N=mn=pq$, the method reshapes $W$ to $N\times N$, computes its truncated SVD, reshapes each singular vector pair into factor matrices $U_k\in\mathbb{R}^{m\times n}$ and $V_k\in\mathbb{R}^{p\times q}$, and approximates the update as $\Delta W\approx\sum_{k=1}^{r}\sigma_k U_k\otimes V_k$. The rank $r$ is chosen by taking the smaller of the energy-threshold rank and the elbow-point rank of the singular spectrum. The empirical claim is that this setup reaches the accuracy of PiSSA while using 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, and converges faster with more stable gradients.

Load-bearing premise

SoKA assumes that the strongest singular directions of the reshaped pretrained weight matrix are also the directions a downstream task needs to change; if that alignment fails, the SVD initialization and spectrum-based rank pruning give no advantage over random or fixed-rank adapters.

Editorial extensions

If this is right

Because each Kronecker term satisfies $(U_k\otimes V_k)\operatorname{vec}(X)=\operatorname{vec}(V_k X U_k^\top)$, SoKA avoids forming the full $N\times N$ update and keeps the structured computation proportional to the small factor sizes.
Per-layer rank is set to $\min(r_{\mathrm{energy}}, r_{\mathrm{elbow}})$, so layers with sharply decaying singular spectra automatically receive fewer parameters than layers with flat spectra.
SoKA's learned factors can be merged back into the frozen weights after training, so inference runs at the original model latency just as with LoRA.
On GSM8K, MATH, and MBPP with LLaMA2-7B, SoKA uses 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, trails PiSSA by less than one point per benchmark, and outperforms LoRA on all three tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the KPSVD initialization should transfer to any square block of a pretrained network, not just the attention projections tested here, because the reshape-and-SVD step only requires $N=mn=pq$.
Inference: the energy-threshold and elbow-point rules could be tested against an oracle rank chosen by validation performance; the paper reports aggregate parameter counts but does not isolate which criterion drives the savings.
Inference: the reported gradient stability suggests SoKA may be a good fit for low-precision or communication-limited fine-tuning, where large gradient spikes are costly, but the paper does not test those settings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the punchline: this is an incremental paper that deserves a serious referee only if it is heavily revised. The core idea is a plausible combination of existing pieces—PiSSA-style SVD initialization, a Kronecker-product update (KPSVD from Batselier and Wong), and dynamic rank selection—but the abstract overclaims, the KPSVD description is dimensionally inconsistent, and the parameter count cannot be reproduced without extra assumptions.

What is actually new: the specific arrangement. No single component is novel; KPSVD, SVD initialization, and energy/elbow heuristics are all cited properly. The paper does report standard deviations, and the body text honestly notes that SoKA trails PiSSA by less than a point on each benchmark. That honesty doesn't carry into the abstract, which says SoKA matches or exceeds baseline performance when Table 1 shows it behind PiSSA on all three tasks (GSM8K 52.19 vs 53.37, MATH 7.93 vs 8.17, MBPP 39.5 vs 40.4).

The soft spots, in order of severity. First, the dimensional slip: Algorithm 1 takes W ∈ R^{mn×pq}, reshapes to (mn)×(pq), and reconstructs U_k⊗V_k of shape (mp)×(nq). For that to equal the original N×N matrix you need p=n and q=m, which is never stated. If you use square factors, each Kronecker rank costs 2N+1 parameters, essentially identical to LoRA's 2N per rank. So the 25% parameter reduction has to be coming from dynamic rank selection producing an average rank near 96, not from Kronecker structure per se. A fair comparison would pit SoKA against LoRA/PiSSA at the same average rank and same budget. Second, the experimental disclosure is thin: no hyperparameters, no module selection, no residual handling, no code. The convergence and gradient-stability claims rest on two qualitative plots with no quantitative backing. Third, the abstract's 'matching or exceeding' claim is simply wrong against PiSSA.

I should say what the paper does well: it positions itself in the literature accurately, cites the prior work (Batselier, PiSSA, KronA) and uses established heuristics rather than inventing ad hoc rules. The combination is not crazy; a Kronecker adapter initialized from the principal components of W is a natural idea that could work if the details are fixed.

Who gets value: PEFT researchers looking for yet another structured adapter variant. It's not a landscape-changer, but it's a legitimate incremental proposal.

My recommendation: send it to peer review with major revision. As submitted I would reject it, but the idea is worth engaging with because the flaws are fixable. Ask for a corrected KPSVD specification, an equal-rank comparison, and full experimental details.

Referee Report

5 major / 3 minor

Summary. The paper proposes SoKA (SVD on Kronecker Adaptation), a parameter-efficient fine-tuning method for large language models. SoKA decomposes each weight update into a weighted sum of Kronecker products of factor matrices via a proposed Kronecker-Product SVD (KPSVD) procedure, initializes the adapter from principal singular vectors following PiSSA's heuristic, and selects the rank per layer using energy-threshold and elbow-point criteria. The experiments on LLaMA2-7B cover GSM8K, MATH, and MBPP, comparing against full fine-tuning, LoRA, and PiSSA. The paper claims 0.99M trainable parameters (25% fewer than LoRA/PiSSA), matching or exceeding baseline performance, faster convergence, and more stable gradients.

Significance. If the method worked as claimed, the combination of SVD-based initialization, Kronecker-structured adapters, and spectrum-aware rank selection would be a useful contribution to parameter-efficient fine-tuning, directly addressing the fixed-rank limitation of LoRA/PiSSA and the lack of spectrum adaptation in earlier Kronecker methods. The use of established PCA heuristics for rank selection is reasonable, and the algorithmic statement is explicit. However, the core Kronecker construction is dimensionally inconsistent as written, the reported numbers contradict the abstract's matching-or-exceeding claim, and the experimental section lacks the reproducibility details needed to verify the results. The paper also does not compare against the Kronecker-based PEFT baselines it cites. These issues are load-bearing for the central claims.

major comments (5)

[Experiments, Table 1] SoKA's results in Table 1 are below PiSSA on all three benchmarks: GSM8K 52.19 vs 53.37, MATH 7.93 vs 8.17, and MBPP 39.5 vs 40.4. The abstract's claim that SoKA achieves 'matching or exceeding baseline performance' is therefore contradicted by the paper's own data, and the text's statement that SoKA 'trails by less than 1 point' is inaccurate for GSM8K (a 1.18-point gap). Because the central contribution is a favorable parameter-efficiency/accuracy trade-off, this discrepancy is load-bearing.
[Kronecker-Product SVD for Adapter Initialization, Algorithm 1] The KPSVD construction is dimensionally inconsistent as written. The paper starts with W∈R^{mn×pq} and reshapes it to (mn)×(pq); the SVD gives x_k∈R^{mn} and y_k∈R^{pq}, which are reshaped to U_k∈R^{m×n} and V_k∈R^{p×q}. Then U_k⊗V_k has shape (mp)×(nq). For this to equal an N×N weight update, one requires mp=N and nq=N, which forces p=n and q=m (given N=mn=pq). The paper never states this constraint, and the later statement 'N=mp=nq' is a different condition unless p=n and q=m. The cited Batselier-Wong (2017) construction requires a specific permutation convention that is not described. In the valid general case, each Kronecker rank costs |U|+|V|+1 = 2N+1 parameters, essentially the same as LoRA's 2N parameters per rank; hence the reported 25% parameter saving (0.99M vs 1.33M) must come from the rank-selection step producing an average rank near 95 rather than 128, not from the Kronecker structure. Without stating this constraint and without a LoRA/PiSSA baseline at the same average rank, the parameter-efficiency claim is not supported.
[Dynamic Rank Selection] The singular-value sequence is defined as S=[σ1≥...≥σ_min(m,n)], but the KPSVD of a reshaped N×N matrix produces N singular values, not min(m,n). The energy-threshold criterion E(k) also sums over min(m,n) terms. This mismatch makes the rank-selection procedure ambiguous: it is unclear whether the spectrum is taken from the full KPSVD or from the factor dimensions, and how per-layer ranks are aggregated into the reported 0.99M trainable parameters. This is not a cosmetic issue, because the entire claimed parameter reduction rests on this selection rule.
[Experiments] The experimental section omits essential reproducibility information: learning rate, schedule, batch size, epochs, LoRA alpha and dropout, targeted modules, dataset versions and prompt formats, KPSVD block sizes (m,n,p,q), the energy threshold τ, rank bounds rmin and rmax, and hardware/software details. Figures 2 report loss and gradient-norm curves without training-configuration details or error bars. Given that the paper's central empirical claims cannot be checked or reproduced from the manuscript, this is a major omission.
[Experiments, Table 1] The empirical comparison includes only LoRA, PiSSA, and full fine-tuning; none of the Kronecker-based PEFT methods discussed in Related Works (Compacter, KronA, AdaKron, LyCORIS) is evaluated. The contribution bullet claiming 'state-of-the-art parameter efficiency' is therefore unsupported by the experiments, and the reader cannot assess whether SoKA improves over existing Kronecker-structured adapters.

minor comments (3)

[Kronecker-Product SVD for Adapter Initialization] The per-block parameter count is written as 'r(mn+pq+1)'; this should be r_kp(mn+pq+1) to match the notation r_kp and to avoid confusion with the rank bound r.
[Throughout] There are typos and stylistic inconsistencies: 'LLama2' in the contribution bullet should be 'LLaMA2'; 'an A,B' and 'an A,B ∈' in the complexity paragraph need grammatical correction; and the references list has inconsistent capitalization (e.g., 'mahabadi' vs 'Mahabadi').
[Kronecker-Product SVD for Adapter Initialization] The paper should state whether the singular values σ_k are trainable or fixed, and how the adapter output is scaled at initialization, since this affects the effective learning rate compared with LoRA and PiSSA.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SoKA's adapter initialization rests on an external SVD heuristic with user-set rank thresholds, and its main results are independent benchmark comparisons.

full rationale

The paper's derivation chain is not circular. SoKA's KPSVD initialization is defined by a truncated SVD of the reshaped pretrained matrix W (Algorithm 1), which is a known external linear-algebra operation; the dynamic rank r is chosen from user-set energy thresholds (e.g., tau = 0.90 or 0.95) and the spectrum elbow, not from fitting downstream accuracy. The 0.99M parameter count is an arithmetic consequence of the selected ranks and factor dimensions, and the performance comparisons use external benchmarks (GSM8K, MATH, MBPP) against independently published baselines (LoRA, PiSSA). No equation in the paper defines KPSVD in terms of the reported accuracy or parameter-count target, so no prediction reduces by construction to its inputs. There are serious technical concerns—Algorithm 1 is dimensionally inconsistent unless p = n and q = m, and it does not implement the cited Batselier-Wong Kronecker decomposition as written—but these are correctness and reproducibility issues, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The method uses existing mathematical tools and heuristics; the main free choices are the rank-selection thresholds and block sizes, which are not fully specified.

free parameters (3)

energy threshold tau
Used in rank selection; value not specified in the paper.
rank bounds rmin, rmax = r_soka <= 128
Constraint on dynamic rank selection; exact bounds not stated.
Kronecker block sizes m, n, p, q
Must be chosen so the Kronecker factors match W's shape; the paper does not state the values used.

assumptions (3)

standard math SVD of a real matrix exists and gives optimal low-rank approximations.
Used in Algorithm 1.
domain assumption Kronecker product SVD decomposition from Batselier and Wong (2017) is valid.
Cited but not derived; the paper relies on it for the KPSVD initialization.
domain assumption Top singular components of the pretrained weight W are a good initialization for task adaptation.
Borrowed from PiSSA; not justified by the authors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Singular Value Decomposition on Kronecker Adaptation for Large Language Model." pith.science (2026). https://pith.science/paper/6PEHA6AI

@misc{pith2026250615251,
  author       = {Pith},
  title        = {Pith review of: Singular Value Decomposition on Kronecker Adaptation for Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PEHA6AI}},
  note         = {Machine review of arXiv:2506.15251}
}

read the original abstract

Large pre-trained Transformer models achieve state-of-the-art results across diverse language and reasoning tasks, but full fine-tuning incurs substantial storage, memory, and computational overhead. Parameter-efficient fine-tuning (PEFT) methods mitigate these costs by learning only a small subset of task-specific parameters, yet existing approaches either introduce inference-time latency (adapter modules), suffer from suboptimal convergence (randomly initialized low-rank updates), or rely on fixed rank choices that may not match task complexity (Kronecker-based decompositions). We propose SoKA (SVD on Kronecker Adaptation), a novel PEFT strategy that combines Kronecker-product tensor factorization with SVD-driven initialization and spectrum-aware dynamic rank selection. Our Kronecker-Product SVD (KPSVD) procedure extracts principal components of the full weight update into compact Kronecker factors, while an adaptive rank selection algorithm uses energy-threshold and elbow-point criteria to prune negligible components. Empirical evaluation on LLaMA2-7B across arithmetic reasoning (GSM8K), formal mathematics (MATH), and code generation (MBPP) demonstrates that SoKA requires only 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, while matching or exceeding baseline performance. Moreover, SoKA exhibits faster convergence and more stable gradients, highlighting its robustness and efficiency for large-scale model adaptation.

Figures

Figures reproduced from arXiv: 2506.15251 by the authors.

**Figure 1.** Visualization of LoRA, PiSSA, and SoKA. Then each singular vector xk ∈ R mn and yk ∈ R pq is ’unvectorized’ into smaller factors. Uk = reshape(xk, (m, n)), Vk = reshape(yk, (p, q)), yielding the Kronecker-product approximation W ≈ Xr k=1 σk Uk ⊗ Vk. This process can be described by Algorithm 1. Algorithm 1: Kronecker-Product SVD (KPSVD) Require: W ∈ R mn×pq, integers m, n, p, q, r Ensure: {(σi , Ui , Vi)} r i=1 wit… view at source ↗

**Figure 2.** Training dynamics: (a) loss and (b) gradient norm [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

[1]

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; and Sutton, C. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732

arXiv 2021
[2]

Batselier, K.; and Wong, N. 2017. A constructive arbitrary-degree Kronecker product decomposition of tensors. Numerical Linear Algebra with Applications, 24(5): e2097. E2097 nla.2097

work page 2017
[3]

Braga, M.; Raganato, A.; and Pasi, G. 2024. A da K ron: An Adapter-based Parameter Efficient Model Tuning with Kronecker Product. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 350--3...

work page 2024
[4]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168

arXiv 2021
[5]

P.; Clark, J

Edalati, A.; Tahaei, M.; Kobyzev, I.; Nia, V. P.; Clark, J. J.; and Rezagholizadeh, M. 2022. KronA: Parameter Efficient Tuning with Kronecker Adapter. arXiv:2212.10650

arXiv 2022
[6]

He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; and Neubig, G. 2022. Towards a Unified View of Parameter-Efficient Transfer Learning. In International Conference on Learning Representations

2022
[7]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874

arXiv 2021
[8]

Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-Efficient Transfer Learning for NLP . In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2790--2799. PMLR

2019

Show all 15 references

[9]

J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

2022
[10]

Y.-C.; Su, M.; jui kuo, E.; Song, Z.; and Liu, H

Hu, J. Y.-C.; Su, M.; jui kuo, E.; Song, Z.; and Liu, H. 2025. Computational Limits of Low-Rank Adaptation (Lo RA ) Fine-Tuning for Transformer Models. In The Thirteenth International Conference on Learning Representations

2025
[11]

Jackson, D. A. 1993. Stopping Rules in Principal Components Analysis: A Comparison of Heuristical and Statistical Approaches. Ecology, 74(8): 2204–2214

1993
[12]

Jolliffe, I. T. 1986. Principal Component Analysis. Springer New York. ISBN 9781475719048

1986
[13]

K.; Henderson, J.; and Ruder, S

mahabadi, R. K.; Henderson, J.; and Ruder, S. 2021. Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. In Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Systems

2021
[14]

Meng, F.; Wang, Z.; and Zhang, M. 2024. Pi SSA : Principal Singular Values and Singular Vectors Adaptation of Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[15]

Yeh, S.-Y.; Hsieh, Y.-G.; Gao, Z.; Yang, B. B. W.; Oh, G.; and Gong, Y. 2024. Navigating Text-To-Image Customization: From Ly CORIS Fine-Tuning to Model Evaluation. In The Twelfth International Conference on Learning Representations

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; and Sutton, C. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732

arXiv 2021

[2] [2]

Batselier, K.; and Wong, N. 2017. A constructive arbitrary-degree Kronecker product decomposition of tensors. Numerical Linear Algebra with Applications, 24(5): e2097. E2097 nla.2097

work page 2017

[3] [3]

Braga, M.; Raganato, A.; and Pasi, G. 2024. A da K ron: An Adapter-based Parameter Efficient Model Tuning with Kronecker Product. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 350--3...

work page 2024

[4] [4]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168

arXiv 2021

[5] [5]

P.; Clark, J

Edalati, A.; Tahaei, M.; Kobyzev, I.; Nia, V. P.; Clark, J. J.; and Rezagholizadeh, M. 2022. KronA: Parameter Efficient Tuning with Kronecker Adapter. arXiv:2212.10650

arXiv 2022

[6] [6]

He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; and Neubig, G. 2022. Towards a Unified View of Parameter-Efficient Transfer Learning. In International Conference on Learning Representations

2022

[7] [7]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874

arXiv 2021

[8] [8]

Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-Efficient Transfer Learning for NLP . In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2790--2799. PMLR

2019