Pith. sign in

REVIEW 5 major objections 6 minor 23 references

Pruning-style scores can fix a tiny set of weights to train, and the lowest-scoring ones often make the best sparse adapters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 04:07 UTC pith:UHA5VG46

load-bearing objection Useful PEFT engineering with honest hedges: Wanda/magnitude fixed supports can work, but the headline ranking is best-of-grid, not fixed-protocol superiority. the 5 major comments →

arxiv 2607.09287 v1 pith:UHA5VG46 submitted 2026-07-10 cs.LG cs.CL

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

Ivan Ilin , Philip Zmushko , Peter Richt\'arik This is my paper
classification cs.LG cs.CL
keywords parameter-efficient fine-tuningsparse fine-tuningLoRAWanda scoreactivation-aware pruninghybrid sparse low-rank adaptersLLM adaptationbudget-matched PEFT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Fine-tuning huge language models is costly if every weight must move. This paper asks whether the same saliency scores used to prune models can instead mark which few weights should adapt. Super freezes almost everything and trains only a small fixed support ranked by an activation-weighted magnitude score from a short calibration pass; Supra splits the same parameter budget between that sparse support and a low-rank LoRA adapter. On single-seed Math17K arithmetic runs with 1B and 8B Llama models, the best Super/Supra variants (often BottomK low-score supports, alone or with LoRA) reach the highest average exact-answer accuracy among the matched-budget adapters tested. A magnitude-only PaFi-style mask is nearly as strong, so the practical message is simple: cheap pruning-inspired orderings can supply useful fixed sparse supports for parameter-efficient fine-tuning.

Core claim

Under matched rank-equivalent trainable-parameter budgets, fixed sparse supports chosen once by a training-free Wanda-style or magnitude score—especially BottomK low-score supports—yield competitive or best-in-comparison average arithmetic accuracy for PEFT, and combining those supports with LoRA under an explicit budget split (Supra) can preserve or improve that result without extra trainable parameters.

What carries the argument

The Wanda-style score A_ij = |W_ij| · ||X_j:||_2 (or magnitude-only |W_ij|), computed from a calibration pass, ranks layer weights so Super/Supra can lock a fixed TopK or BottomK sparse mask; Supra then converts a target parameter budget into a LoRA rank and residual sparse count via a simple floor rule on the low-rank budget fraction λ.

Load-bearing premise

That single-seed, validation-picked learning rates and best-of-one-or-three-epoch summaries on arithmetic-only benchmarks fairly show these fixed low-score supports remain useful when the task or model scale changes.

What would settle it

Multi-seed Math17K (or non-math) runs in which, under the same matched budget and validation LR protocol, BottomK Super/Supra and magnitude masks consistently fall below LoRA and random sparse supports on average exact-answer accuracy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can build sparse PEFT masks with one calibration pass and no gradients, then train only those entries.
  • Low-score (BottomK) pretrained coordinates are a strong default support to try before TopK or random masks.
  • Splitting a fixed parameter budget between sparse updates and LoRA can match or beat pure LoRA without raising the trainable count.
  • Magnitude-only ranking is a viable no-calibration alternative when activation statistics are unavailable.
  • Pruning metrics can be repurposed as adaptation-site selectors rather than only as deletion criteria.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If low-score weights are freer to move without wrecking pretrained behavior, other PEFT methods might deliberately protect high-score coordinates and concentrate learning on the low-score tail.
  • Dynamic reallocation of the sparse mask during training is a natural next test: fixed BottomK may be a strong start but not the last word when tasks drift mid-training.
  • The same ordering idea could be tried on structured supports (rows, heads, channels) to get real kernel speedups beyond the paper’s dense-matmul sparse implementation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Super, a sparse PEFT method that freezes a small trainable support chosen once by a Wanda-style activation-weighted magnitude score from a calibration pass, and Supra, a hybrid that splits a matched scalar-parameter budget between that fixed sparse support and LoRA via a transparent rank-conversion rule. On Math17K arithmetic fine-tuning of Llama-3.2-1B and Meta-Llama-3-8B under a rank-equivalent r0=8 budget, the authors report that the best schedule-selected Super/Supra (and Supra-Mag) variants attain the highest average exact-answer accuracy among the tested adapters. They also include PaFi-style magnitude-only supports, TopK/BottomK and hybrid-mask ablations, full one-/three-epoch grids, and implementation-level efficiency profiling, and carefully hedge the main claim as single-seed and schedule-selected.

Significance. If the finding holds under fixed protocols and broader evaluation, the work would be a useful practical contribution: it shows that training-free pruning-inspired orderings—especially low-score BottomK supports—can define competitive fixed sparse PEFT masks, alone or with LoRA, without gradient-based mask search. Strengths include explicit budget matching, a closest training-free magnitude baseline, reconstruction-identity motivation (Prop. 3.1), released code, and an honest Limitations section. The hybrid budget-splitting rule is transparent rather than adaptive. Significance is currently limited by arithmetic-only evaluation, single-seed results, and schedule/variant selection that inflate the headline ranking relative to fixed-schedule tables.

major comments (5)
  1. Tables 1–2 and §4.3 present schedule-selected summaries that, for each method family, take the better of one- vs three-epoch runs by average test accuracy and, for Super/Supra, the best support direction and λ from the appendix grids. This is load-bearing for the central superiority claim, yet it is not a fixed-protocol ranking: methods with more searchable configurations (λ grid, TopK/BottomK, Mag vs Wanda) are rewarded under single-seed noise. The paper’s own Limitations admit these are best-observed summaries. Primary claims and main tables should report a pre-specified protocol (fixed epoch budget, fixed default BottomK, fixed λ or a small pre-declared set) with appendix grids as secondary exploration.
  2. Fixed-schedule appendix tables undermine a universal Super/Supra advantage. On Llama-3.2-1B Math17K one-epoch (Table 11), Magnitude(BottomK) 59.31 and Supra-Mag(λ=0.3) 59.18 beat Super(BottomK) 37.00 and several Supra rows; on Meta-Llama-3-8B three-epoch (Table 19), Super(BottomK) 72.73 is strong but many Supra/Supra-Mag rows collapse (e.g. Supra-Mag λ=0.3 → 32.30). The abstract and §5 should be rewritten so the headline claim matches fixed-protocol evidence rather than post-hoc best-of-grid averages.
  3. The distinctive Super ingredient is the Wanda activation term Aij=|Wij|∥Xj:∥2 versus PaFi-style |Wij|. On the 8B schedule-selected comparison (Tables 2–3), Supra-Mag(BottomK, λ=0.3) 79.12 edges Magnitude(BottomK) 79.02 and beats Wanda-selected Supra 78.66; sparse-only Super often underperforms magnitude. §5 should state clearly that low-score pretrained-magnitude supports, not activation weighting per se, drive most of the positive sparse results, and either strengthen the case for Wanda (e.g., tasks/settings where calibration activations matter) or reframe Super as one instance of low-score fixed-support PEFT.
  4. All main results use a single seed (seed 0), while the 1B Supra vs LoRA gap is only 1.16 points (62.23 vs 61.07) and several 8B sparse/hybrid gaps are similarly small. §6 acknowledges this, but the abstract and §4.3 still read as method rankings. Either multi-seed means±std for the main matched-budget rows, or the claims must be demoted to single-run observations without ranking language. Small single-seed gaps under validation-selected LRs are not sufficient to establish superiority.
  5. Evaluation is confined to Math17K arithmetic fine-tuning and six arithmetic benchmarks on two Llama sizes (§4.1, §6). That is a narrow slice for a general PEFT method claim. At minimum, one non-arithmetic suite (e.g., commonsense or instruction-following) under the same matched budget would test whether BottomK fixed supports transfer beyond symbolic math; without it, the contribution should be scoped as arithmetic PEFT evidence rather than a general sparse-adapter recipe.
minor comments (6)
  1. Table 3 lists Supra(BottomK, λ=0.3) Acc. 55.03 on 1B while Table 1’s selected Supra row is λ=0.8 at 62.23; the dual λ rows are easy to misread as both being schedule-selected winners. Clarify which rows are compact winners vs diagnostic.
  2. Eq. (1) uses the squared Wanda score Skq=(|Wkq|∥Xq:∥2)2 while mask selection uses Aij=|Wij|∥Xj:∥2; the text notes monotonicity, but a single consistent score definition in §3.2 would reduce confusion.
  3. Figure 1 is schematic and helpful; adding the explicit budget split (ri=⌊λr0⌋, si=(r0−⌊λr0⌋)(ci+bi) from Eq. 22) on the figure would make Supra’s matching rule self-contained.
  4. Related work cites RoSA, PaFi, SIFT, and recent salience-aware sparse PEFT; a short explicit contrast table (score, static vs dynamic mask, hybrid with LoRA, training-free?) would sharpen novelty relative to RoSA and PaFi.
  5. Efficiency Table 5 notes dense matmul/grad gather for sparse layers; the abstract’s efficiency narrative should not imply sparse-kernel speedups beyond measured checkpoint/optimizer-state savings.
  6. Minor typos/style: “Supra-Mag denotes…” appears before formal definition in the intro tables; “SpA” is introduced then Super is used; ensure consistent method naming (Super vs Super-Tuning title).

Circularity Check

0 steps flagged

No circular derivation: Super/Supra are empirical PEFT methods whose accuracy claims are measured outcomes, not quantities forced by their own selection scores or budget rules.

full rationale

This paper does not claim a first-principles prediction that reduces to its inputs. Super fixes a sparse support via a Wanda-style score A_ij = |W_ij|∥X_j:∥_2 from a calibration pass (Algorithm 1, Eq. 7–9); Supra adds LoRA under an explicit budget split λ and rank conversion r_i = ⌊λ T_i/(c_i+b_i)⌋ (Eq. 14–22). Proposition 3.1 is a correct single-weight reconstruction identity used only to motivate a training-free ordering heuristic; the paper explicitly states the score is not an optimality criterion for sparse fine-tuning and that learned deltas are unconstrained. Reported accuracies are external measurements on held-out arithmetic benchmarks after Math17K training under matched scalar budgets, with LR chosen on validation NLL—not quantities reconstructed from the mask scores. Related work (Wanda, LoRA, PaFi, RoSA, SIFT) is cited from external authors; there is no load-bearing self-citation uniqueness chain, no fitted constant renamed as a prediction of the main claim, and no renaming of a known closed-form result. Schedule/variant selection in compact Tables 1–2 is a reporting protocol issue (fairness of comparison), not circular reasoning by construction. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central claim rests on empirical PEFT protocol choices and the hypothesis that pruning-style orderings identify useful fixed adaptation supports. Free parameters are design/selection knobs (budget rank, λ, LR, epochs, calibration). Axioms are standard PEFT and Wanda reconstruction heuristics, not new physics. Invented entities are the Super/Supra methods themselves.

free parameters (5)
  • rank-equivalent budget r0
    Main experiments fix r0=8 to match LoRA scalar-parameter count; all adapter comparisons depend on this budget choice.
  • Supra low-rank budget fraction λ
    λ ∈ {0.3,0.5,0.8} splits trainable parameters between LoRA and sparse support; best reported rows select among these values.
  • validation-selected learning rate
    Each method picks one LR from a fixed grid using held-out Math17K NLL; reported test accuracy depends on this selection.
  • training schedule (1 vs 3 epochs)
    Compact tables report the better of one- and three-epoch runs by average accuracy; schedule is a free experimental choice affecting the headline ranking.
  • Wanda calibration set size/source
    Default masks use 128 C4 samples; magnitude-only needs no calibration. Mask construction depends on this choice (ablated only lightly).
axioms (4)
  • domain assumption Wanda single-weight reconstruction identity (Prop. 3.1) is a useful training-free ordering heuristic for sparse fine-tuning supports, not an optimality criterion for learned deltas.
    §3.2 explicitly treats the score as a diagonal reconstruction heuristic for support selection.
  • domain assumption Matching trainable scalar-parameter counts is a fair primary PEFT comparison axis even when memory/runtime differ by implementation.
    §4.1–4.2 match r0=8 scalar budgets and separately report efficiency profiling.
  • domain assumption Held-out Math17K validation NLL is an adequate selector of learning rate and checkpoint for arithmetic exact-answer evaluation.
    §4.1: LR selected by validation NLL, not test accuracy.
  • standard math Standard linear algebra and PEFT parameterizations (LoRA rank, sparse support cardinality, Hadamard mask updates).
    Equations (4)–(22) use ordinary rank/sparsity constraints and budget conversion.
invented entities (2)
  • Super (Wanda-ordered fixed sparse PEFT) independent evidence
    purpose: Select a fixed sparse trainable support once via activation-weighted magnitude scores and train only those entries.
    New method name/algorithm; independent evidence is the reported Math17K experiments, not external theory.
  • Supra (budget-matched Super+LoRA hybrid) independent evidence
    purpose: Split a fixed scalar-parameter budget between Super sparse updates and LoRA under a simple floor-based rank rule.
    Hybrid PEFT design with explicit budget conversion (Eq. 19–22); evidence is empirical comparison tables.

pith-pipeline@v1.1.0-grok45 · 37948 in / 3363 out tokens · 38752 ms · 2026-07-13T04:07:25.798857+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Figures

Figures reproduced from arXiv: 2607.09287 by Ivan Ilin, Peter Richt\'arik, Philip Zmushko.

Figure 1
Figure 1. Figure 1: Supra combines a sparse adapter based on Super weights with a low-rank adapter. To select the parameters for training in the sparse adapter, we employ the Wanda metric (Equation 1). The rank r of the low-rank adapter is determined from the matched parameter-count budget using Equation 19. 3.3 Supra: Budget-Matched Low-Rank and Sparse Adaptation. Supra is a hybrid fine-tuning strategy that combines two comp… view at source ↗
Figure 2
Figure 2. Figure 2: Optimization curves for Meta-Llama-3-8B Math17K fine-tuning over three epochs. Left: training loss, plotted with a log-scaled y-axis. Right: held-out Math17K validation loss. Each curve uses the validation-selected learning rate for the corresponding method under the rank-equivalent r0 = 8 budget. Evaluation metric. We report exact-answer accuracy on each math benchmark and the arithmetic mean over benchma… view at source ↗
Figure 3
Figure 3. Figure 3: Optimization curves for Llama-3.2-1B Math17K fine-tuning over three epochs. Left: training loss, plotted with a log-scaled y-axis. Right: held-out Math17K validation loss. Each curve uses the validation-selected learning rate for the corresponding method under the rank-equivalent r0 = 8 budget [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 1 canonical work pages

  1. [1]

    Scaling sparse fine-tuning to large language models.arXiv preprint arXiv:2401.16405,

    Alan Ansell, Ivan Vuli ´c, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models.arXiv preprint arXiv:2401.16405,

    Pith/arXiv arXiv

  2. [2]

    Gallop: Gradient-based sparse learning on low-magnitude parameters.arXiv preprint arXiv:2510.19778,

    Anand Choudhary, Yasser Sulaıman, Lukas Mauch, Ghouthi Boukli Hacene, Fabien Cardinaux, and Antoine Bosselut. Gallop: Gradient-based sparse learning on low-magnitude parameters.arXiv preprint arXiv:2510.19778,

    arXiv

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al

    doi: 10.48550/arXiv.2510.19778. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page doi:10.48550/arxiv.2510.19778

  4. [4]

    Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

    Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. Sltrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.02214,

    Pith/arXiv arXiv

  5. [5]

    Learning to solve arithmetic word problems with verb categorization

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 523–533,

    2014

  6. [6]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    Pith/arXiv arXiv

  7. [7]

    An efficient sparse fine-tuning with low quantization error via neural network pruning.arXiv preprint arXiv:2502.11439,

    Cen-Jhih Li and Aditya Bhaskara. An efficient sparse fine-tuning with low quantization error via neural network pruning.arXiv preprint arXiv:2502.11439,

    Pith/arXiv arXiv

  8. [8]

    Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems.arXiv preprint arXiv:1705.04146,

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems.arXiv preprint arXiv:1705.04146,

    Pith/arXiv arXiv

  9. [9]

    Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353,

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353,

    Pith/arXiv arXiv

  10. [10]

    org/2025.acl-long.1541/

    URL https://aclanthology. org/2025.acl-long.1541/. 16 Da Ma, Lu Chen, Pengyu Wang, Hongshen Xu, Hanqi Li, Liangtai Sun, Su Zhu, Shuai Fan, and Kai Yu. Sparsity-accelerated training for large language models.arXiv preprint arXiv:2406.01392,

    Pith/arXiv arXiv 2025

  11. [11]

    Rosa: Accurate parameter-efficient fine-tuning via robust adaptation.arXiv preprint arXiv:2401.04679,

    Mahdi Nikdan, Soroush Tabesh, Elvir Crnˇcevi´c, and Dan Alistarh. Rosa: Accurate parameter-efficient fine-tuning via robust adaptation.arXiv preprint arXiv:2401.04679,

    Pith/arXiv arXiv

  12. [12]

    Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191,

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191,

    Pith/arXiv arXiv

  13. [13]

    Solving general arithmetic word problems.arXiv preprint arXiv:1608.01413,

    Subhro Roy and Dan Roth. Solving general arithmetic word problems.arXiv preprint arXiv:1608.01413,

    Pith/arXiv arXiv

  14. [14]

    Sparse is enough in fine-tuning pre-trained large language models.arXiv preprint arXiv:2312.11875,

    Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du. Sparse is enough in fine-tuning pre-trained large language models.arXiv preprint arXiv:2312.11875,

    Pith/arXiv arXiv

  15. [15]

    A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695,

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695,

    Pith/arXiv arXiv

  16. [16]

    Let’s focus on neuron: Neuron-level supervised fine-tuning for large language model.arXiv preprint arXiv:2403.11621,

    Haoyun Xu, Runzhe Zhan, Derek F Wong, and Lidia S Chao. Let’s focus on neuron: Neuron-level supervised fine-tuning for large language model.arXiv preprint arXiv:2403.11621,

    Pith/arXiv arXiv

  17. [17]

    S2ft: Efficient, scalable and generalizable llm fine-tuning by structured sparsity

    Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. S2ft: Efficient, scalable and generalizable llm fine-tuning by structured sparsity. arXiv preprint arXiv:2412.06289,

    Pith/arXiv arXiv

  18. [18]

    Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512,

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512,

    Pith/arXiv arXiv

  19. [19]

    Gift-sw: Gaussian noise injected fine-tuning of salient weights for llms.arXiv preprint arXiv:2408.15300,

    Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov, Mariya Krylova, Aleksandr Zuev, and Evgeny Burnaev. Gift-sw: Gaussian noise injected fine-tuning of salient weights for llms.arXiv preprint arXiv:2408.15300,

    Pith/arXiv arXiv

  20. [20]

    All trainable masks use C4 calibration and approximately 5.6M trainable sparse parameters

    The Base row is frozen. All trainable masks use C4 calibration and approximately 5.6M trainable sparse parameters. Best average within each epoch block is bold. Results are exact-answer accuracy (%). Method EpochsβSelected LR AddSub MultiArith SingleEq GSM8K AQuA SV AMP Average BASE(frozen) – – – 13.67 4.67 21.46 2.81 21.65 11.60 12.64 Super(TopK) 1 1.05·...

    2050

  21. [21]

    Full fine-tuning is an unbudgeted reference row separated by rules

    The Base row is frozen. Full fine-tuning is an unbudgeted reference row separated by rules. Bold marks the lowest matched-budget adapter average PPL. TheCalib.column gives the calibration source used to construct fixedWanda-style sparse supports; “–” denotes methods that do not use a calibration pass. Lower is better. Perplexities are computed on up to 12...

    2000

  22. [22]

    Each row uses the learning rate selected by the held-out validation split of the fine-tuning set

    Adapter rows use approximately the rank-8LoRAtrainable-parameter budget for this model. Each row uses the learning rate selected by the held-out validation split of the fine-tuning set. Results are exact-answer accuracy (%). Method Calib. Selected LR AddSub MultiArith SingleEq GSM8K AQuA SV AMP Average BASE(frozen) – – 22.53 21.00 36.42 10.24 24.02 24.10 ...

    2072

  23. [23]

    Lower is better

    TheCalib.column gives the calibration source used to construct fixedWanda-style sparse supports; “–” denotes methods that do not use a calibration pass. Lower is better. Perplexities are computed on up to 120 examples per benchmark. Method Calib. Selected LR AddSub MultiArith SingleEq GSM8K AQuA SV AMP Average BASE(frozen) – – 2.15 2.28 2.30 1.98 2.64 2.4...

    2079