Pith. sign in

REVIEW 5 major objections 6 minor 46 references

A one-pass mask that tunes 0.1% of weights, selected by gradient-to-weight ratio, beats full fine-tuning on seven GLUE/SuperGLUE tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GEM selects fine-tuning parameters by gradient-to-weight ratio and distributes the budget by layer entropy, reaching 0.1% parameter updates with small accuracy gains on several NLP tasks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection GEM is a plausible incremental PEFT recipe, but the paper's own ablation undercuts the 'scale-aware' headline; the useful part is entropy-guided layer allocation. the 5 major comments →

arxiv 2508.16191 v1 pith:GGQBLEOD submitted 2025-08-22 cs.LG

GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation

Sungmin Kang , Jisoo Kim , Salman Avestimehr , Sunwoo Lee This is my paper
classification cs.LG
keywords parameter-efficient fine-tuninggradient-to-weight ratiosparse fine-tuningentropy-guided maskingstatic masklayer-wise budget allocationGLUESuperGLUE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GEM (Gradient-to-Weight Ratio and Entropy-guided Masking) is a sparse fine-tuning method that picks which weights to update by the size of the gradient relative to the weight's own pre-trained value, |gradient|/|weight|, instead of the absolute gradient. It then decides how many parameters each layer may update using the entropy of those ratio scores inside the layer, so layers with a concentrated learning signal get fewer updates and layers with a spread-out signal get more. The paper reports that with only 0.1% of parameters updated, this static one-pass mask beats full fine-tuning on average across seven GLUE/SuperGLUE tasks on OPT-125M, OPT-1.3B, and Phi-2, and beats the PEFT baselines compared (LoRA, BitFit, Adapters, AdaLoRA, random and top-gradient masking). The paper also reports that GEM is the best PEFT method on GSM8k and MBPP code/math tasks, though below full fine-tuning there. If the result holds, relative change, not raw update magnitude, is the right signal for deciding what to tune under a tight budget.

Core claim

GEM's central claim is that fine-tuning capacity is best spent on parameters whose updates are largest relative to their initial scale. The paper formalizes this as the gradient-to-weight ratio rho(i) = |gradient of L wrt w(i)| / |w(i)| and argues, through a Taylor-expansion argument and loss-reduction measurements, that selecting top-rho parameters produces both the largest proportional weight shifts and the largest loss reduction per unit of gradient norm. To allocate the total 0.1% budget across layers, it computes the entropy H(p_l) of normalized rho scores in each layer and scores the layer by alpha_l = ||rho_l||_2 * H(p_l), tuning more parameters where the signal is strong or broadly s

What carries the argument

The gradient-to-weight ratio (GWR) is the per-parameter score rho(i) = |gradient of L wrt w(i)| / |w(i)|, computed at the initial weights, used to rank parameters inside each layer. The second piece is the entropy-guided layer importance alpha_l = ||rho_l||_2 * H(p_l), where p_l is the normalized GWR distribution within layer l; the entropy term measures whether the learning signal is concentrated in a few parameters or spread across many, and the norm term measures the layer's overall update strength. Together they build a static binary mask M_l per layer: select the top k_l = floor(r * N * gamma_l) parameters, with gamma_l = alpha_l / sum_j alpha_j. The framework's load-bearing assumption

Load-bearing premise

GEM chooses its mask once, from the first backward pass, and never revisits it; if the early gradients misidentify which weights will matter after training, the whole advantage vanishes.

What would settle it

Run GEM against a version that re-computes the mask every few epochs at the same 0.1% budget. If refreshing the mask improves accuracy materially, the static-mask premise is wrong. A cheaper diagnostic: track the rank correlation between gradient-to-weight ratios at epoch 0 and at later epochs; if it decays toward zero, the initial mask is effectively random by the end of training.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A fixed budget of 0.1% tunable parameters can match or exceed full fine-tuning on the GLUE/SuperGLUE tasks tested, on models from 125M to 2.7B parameters.
  • Scale-aware selection (GWR) is the reason: it beats top-gradient masking by 4.48% (OPT-125M) and 6.05% (OPT-1.3B) at the same budget.
  • Entropy-guided layer allocation beats both uniform allocation and norm-only allocation on every task and model tested, suggesting per-layer budgets should reflect signal spread, not equal counts.
  • Because the mask is static and derived from one backward pass, GEM adds almost no compute over ordinary fine-tuning: no extra trainable modules, no extra inference parameters, no adaptive updates.
  • On domain-shifted tasks (GSM8k, MBPP), GEM remains the strongest PEFT baseline, meaning the selection criterion carries over outside GLUE-style tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not refresh or validate its one-shot mask. A natural test the authors leave implicit: re-select the mask every few epochs and compare; if refreshes help, the static mask is not actually near-optimal, and the reported gains likely come mostly from the GWR criterion rather than the static design.
  • Because GWR divides by weight magnitude, it resembles a scale-invariant normalization; one testable extension is to combine it with curvature information to see whether the top-rho set coincides with high-curvature directions — if so, GWR could serve as a cheap proxy for Hessian-based selection.
  • Entropy-guided allocation could be applied to any importance score, not just GWR, and could be re-computed periodically during training; the paper only tests it as a one-time layer budget.
  • In settings with heterogeneous parameter scales — quantized models, or models with mixed precision — absolute-gradient selection is biased toward large-scale weights; GWR's scale-awareness may be even more advantageous there, a regime the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes GEM, a static sparse fine-tuning framework that selects tunable parameters by a gradient-to-weight ratio (GWR), rho_i = |grad_i L| / |w_i|, and allocates per-layer tuning budgets using an entropy-weighted layer score alpha_l = ||rho_l||_2 * H(p_l). The mask is built once from a backward pass at initialization, and only the selected 0.1% of parameters are updated. The authors evaluate GEM on seven GLUE/SuperGLUE tasks and two domain-specific tasks across OPT-125M, OPT-1.3B, and Phi-2, reporting average accuracy above full fine-tuning and above several PEFT baselines, along with ablations on layer-allocation strategies and tuning-ratio sensitivity.

Significance. If its central claims held, GEM would be a useful, simple sparse PEFT baseline: it is model-agnostic, has no extra trainable or inference parameters, and the paper provides a fairly broad empirical protocol (three model sizes, seven general-domain tasks, two reasoning/code tasks, three seeds, learning-rate grid search, and full-training-set robustness checks). However, the scale-aware GWR contribution is currently confounded with the entropy-based layer allocation, and the paper's own ablation data suggest GWR may be harmful under uniform allocation. The theoretical justification in Section 3.1.2 is a global vector-norm identity that does not support per-parameter selection. These issues must be resolved before the title-level claim about 'scale-aware' adaptation can be accepted.

major comments (5)
  1. [§4.3.1, Table 4 vs. Table 3] The ablation in Table 4 fixes the in-layer selection metric to GWR and only varies the layer-allocation strategy, so it cannot isolate the contribution of the GWR metric. Worse, comparing Table 4 with Table 3 under uniform allocation shows GWR performing below Top Gradient Masking and Random Masking: for OPT-125M on SST-2, GWR+Uniform gives 85.34 vs. Top Gradient Masking 88.07 and Random Masking 89.33; on SQuAD the values are 59.52 vs. 61.03 and 61.69. Thus the reported advantage of GEM over Top Gradient Masking could be entirely due to entropy-guided layer allocation, and the 'scale-aware' contribution named in the title and abstract is unsupported. Please run a full 2x2 ablation: gradient magnitude vs. GWR, crossed with uniform vs. entropy allocation, plus a random in-layer selection condition, and report all combinations.
  2. [§3.1.2, Eqs. (3)-(4)] The theoretical interpretation is not valid for the proposed per-parameter criterion. Equations (3)-(4) and the subsequent norm manipulation establish a relationship among global quantities: with Delta w = -eta grad L, one gets |Delta L| / ||grad L|| divided by ||w|| approximately equals ||Delta w|| / ||w||. This vector-norm identity says nothing about the coordinate-wise ratio rho_i = |grad_i L| / |w_i| used in Eq. (2) and Algorithm 1. Ranking parameters by rho_i does not follow from this identity, and the |w_i| denominator can amplify small-weight coordinates regardless of their contribution to loss reduction. The authors should either provide a per-coordinate sensitivity analysis or explicitly frame GWR as a heuristic and rely on controlled experiments.
  3. [§3.1.3, Figure 2] The empirical 'validation' of GWR is partly circular. Figure 2(a) shows that GWR masking yields the largest relative weight change; this is close to true by construction, because the mask is built by selecting parameters with the largest |Delta w|/|w| at initialization, and the plotted quantity is the same relative-change measure. Figure 2(b) also uses the first-order approximation |Delta L| approx | - grad L_0 . (w_t - w_0) |, which measures alignment with the initial gradient rather than the actual loss reduction. Please report the true training/validation loss trajectories for the compared masks, and avoid presenting the relative-weight-change result as independent evidence.
  4. [Algorithm 1] Algorithm 1 constructs the mask from a single backward pass at W0 and never refreshes or updates the mask during training. All reported gains therefore rely on the untested assumption that the initial GWR ranking remains near-optimal throughout optimization. Please compare the static mask against masks refreshed at regular intervals, masks built from gradients averaged over the first several steps, or a dynamic-mask variant, and report whether GEM's advantage persists. Without this, the method's strong results could be an artifact of a favorable coincidence at initialization.
  5. [§4.2, Table 3] The text states that GEM 'outperforms all other SOTA methods' and, for Phi-2, 'ranking first on all seven tasks.' Table 3 does not support these per-task claims: on Phi-2, GEM is below full fine-tuning on MultiRC (83.46 vs. 83.51) and COPA (91.33 vs. 92.00), and on OPT-125M COPA it ties with FFT and Adapter at 69.67. The claims should be restricted to average accuracy. Moreover, no paired significance tests are reported, and many differences are within one standard deviation of three seeds; given that each method uses a separately tuned per-task learning rate (Table 7), the paper should report pairwise significance tests or confidence intervals to support its comparative conclusions.
minor comments (6)
  1. [§4.1.1] The citation for OPT appears as '[?]' in the text ('OPT-125M, OPT-1.3B [?]') and should be replaced with the proper reference [Zhang et al., 2022].
  2. [Appendix A.2] The learning-rate grid description is ambiguous: 'a grid search over learning rates in {1e-2, ..., 1e-6}, each scaled by factors of 1 and 5' should be made explicit (e.g., {1e-2, 5e-2, 1e-3, 5e-3, ...}).
  3. [§3.1 and §3.1.3] Notation is inconsistent: |w| is used both for the absolute value of a scalar parameter and for a vector/norm quantity. Define the norm explicitly, especially in Eqs. (1)-(2) and the relative-change formula in Section 3.1.3.
  4. [Table 1] The Top-Gradient Mask row marks 'Layer-wise Importance' as partially true with a footnote, while the text and Section 4.2.1 state that the baseline uses a uniform per-layer allocation. Clarify what the implemented baseline actually does.
  5. [§4.3.3] There is a typo on page 10: 'only about0.1%' should be 'only about 0.1%'.
  6. [Reproducibility] No code, configuration files, or model checkpoints are provided. Given the method is a simple masking scheme, a public implementation would substantially strengthen reproducibility.

Circularity Check

1 steps flagged

No significant circularity; one motivation step is self-confirming by construction

specific steps
  1. self definitional [Section 3.1.1, Eq. (1)-(2); Section 3.1.3, Figure 2(a)]
    "Motivated by the analysis above, we define a scale-invariant signal for parameter prioritization, referred to as the gradient-to-weight ratio (GWR), as ρ(i) := |∇w(i)L / w(i)|. ... Figure 2(a) shows that selecting parameters based on the gradient-to-weight ratio, ρ, leads to the largest relative weight change, indicating that this metric enables us to effectively prioritize the parameters that undergo meaningful updates relative to their scale. This observation is consistent with the intended design of our method."

    From Eq. (1), |Δw|/|w| = η·|∇L|/|w| = ηρ for one gradient-descent step. Thus choosing parameters with the largest ρ is, by the paper's own update rule, exactly the rule that maximizes the one-step relative weight change. Figure 2(a) therefore does not provide independent empirical support; it restates the selection criterion. This is a motivation-level tautology, not the paper's main empirical claim, which is tested against external benchmarks.

full rationale

The paper's central empirical claim—that GEM with 0.1% parameters outperforms full fine-tuning and PEFT baselines on GLUE/SuperGLUE and domain tasks—is evaluated against external benchmarks and does not involve fitting a parameter and then predicting a closely related quantity. The method is a fixed, static mask computed from one backward pass; there is no fitted input renamed as a prediction. The paper contains no load-bearing self-citations; references to prior work are external and not used to justify GEM's core selection rule. The only identifiable circular element is the motivation in Section 3.1.3 / Figure 2(a): showing that GWR-based selection yields the largest relative weight change is true by construction from Eq. (1)-(2). The separate loss-reduction claim and final task accuracies are not forced by the metric definition. The skeptic's concern that the ablation in Table 4 confounds the GWR metric with entropy-based allocation is a legitimate experimental-design concern about attribution, not circularity; per the review rules it does not raise the circularity score. Overall, the derivation chain is mostly self-contained, with one minor self-confirming motivation step, so the score is 2.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method's empirical claims rest mostly on standard benchmark evaluation, but its design choices (GWR definition, entropy product, static mask, budget r) are assumptions rather than derived results. No new entities are introduced.

free parameters (4)
  • tuning ratio r = 0.1% default; 0.5%, 0.05%, 0.01% in Figure 3 and Table 8
    The total fraction of parameters to tune is chosen from a small grid; it is a design choice, not fitted to the reported test sets, but it directly controls the mask budget.
  • entropy weighting in alpha_l = ||rho_l||_2 * H(p_l) = 1 (multiplicative, no exponent tuning)
    The layer importance score combines norm and entropy with equal weight; no ablation over alternative functional forms is provided, so the form is chosen by hand.
  • per-method per-task learning rates = values in Table 7, e.g. GEM 5e-5 for RTE
    Selected by grid search over {1e-2..1e-6, times 1 and 5} for every method and task; this is standard hyperparameter selection but is part of the empirical protocol.
  • dataset subset caps = 1,000 (most tasks), 8,000 SQuAD, 6,000 SST-2, 2,000 GSM8k
    Training sets are subsampled with caps; sampling method and seeds are not given, which affects reproducibility.
axioms (4)
  • domain assumption First-order Taylor approximation of the loss is accurate for the update scales used.
    Used in §3.1.2 Eq. (3) to connect ΔL and ∇L·Δw; no error bound or update-scale verification is given.
  • domain assumption Gradients at initialization are a sufficient oracle for selecting parameters to update throughout training.
    Algorithm 1 builds the binary mask from ∇L(W0) and Eq. (10) keeps it static; the paper never tests whether refreshing the mask changes results.
  • ad hoc to paper Entropy of the normalized GWR distribution measures how many parameters a layer needs.
    Eqs. (5)-(7) define layer importance as alpha_l = ||rho||_2 * H; no derivation connects this specific functional form to optimal budget allocation.
  • ad hoc to paper Relative weight change |Δw|/|w| is the right definition of downstream adaptation.
    The paper asserts this in the introduction; it is a modeling choice, not implied by any theorem.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation." pith.science (2026). https://pith.science/paper/GGQBLEOD

@misc{pith2026250816191,
  author       = {Pith},
  title        = {Pith review of: GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGQBLEOD}},
  note         = {Machine review of arXiv:2508.16191}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Parameter-efficient fine-tuning (PEFT) has become a popular way to adapt large pre-trained models to new tasks. Most PEFT methods update only a small subset of parameters while freezing the rest, avoiding redundant computation. As they maximize the absolute size of the updates without regard to the parameters' original scale, the resulting changes in model behavior can be minimal. In contrast, we maximize updates relative to each parameter's scale, yielding more meaningful downstream adaptation. We propose Gradient-to-Weight Ratio and Entropy-guided Masking (GEM), a parameter scale-aware, distribution-sensitive sparse fine-tuning framework. GEM prioritizes parameters whose updates are significant in proportion to their initial pre-trained values. It also adaptively determines how many parameters to tune at each layer based on the entropy of parameter values, thereby making the most effective use of the computational budget in PEFT. Our empirical study demonstrates the efficacy of GEM on both general-domain tasks (GLUE and SuperGLUE) and domain-specific tasks (GSM8k and MBPP), achieving up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 13 canonical work pages · 5 internal anchors

  1. [1]

    Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models

    Aradhye Agarwal, Suhas K Ramesh, Ayan Sengupta, and Tanmoy Chakraborty. Step-by-step unmasking for parameter-efficient fine-tuning of large language models, 2025. URL https://arxiv.org/abs/2408.14470

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  4. [4]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT 2019, 2019

    work page 2019

  5. [5]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  6. [6]

    The pascal recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177--190. Springer, 2005

    work page 2005

  7. [7]

    Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning, 2023

    Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning, 2023. URL https://arxiv.org/abs/2311.03748

    Pith/arXiv arXiv 2023

  8. [8]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314

    Pith/arXiv arXiv 2023

  9. [9]

    Sparse low-rank adaptation of pre-trained language models, 2023

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models, 2023. URL https://arxiv.org/abs/2311.11696

    Pith/arXiv arXiv 2023

  10. [10]

    Xo RA : Expander adapted lo RA finetuning

    Amaljith EV, Arindam Biswas, Suryam Arnav Kalra, Pabitra Mitra, and BISWAJIT BASU. Xo RA : Expander adapted lo RA finetuning. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. URL https://openreview.net/forum?id=4MTGxMHBgg

    work page 2024

  11. [11]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL https://arxiv.org/abs/1803.03635

    Pith/arXiv arXiv 2019

  12. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv 2023

  13. [13]

    Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning

    Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, and Weiping Wang. Light-peft: Lightening parameter-efficient fine-tuning via early pruning, 2024. URL https://arxiv.org/abs/2406.03792

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv....

    Pith/arXiv arXiv 2023

  15. [15]

    Rush, and Yoon Kim

    Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning, 2021. URL https://arxiv.org/abs/2012.07463

    Pith/arXiv arXiv 2021

  16. [16]

    Gora: Gradient-driven adaptive low rank adaptation, 2025

    Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, and Lei Chen. Gora: Gradient-driven adaptive low rank adaptation, 2025. URL https://arxiv.org/abs/2502.12171

    arXiv 2025

  17. [17]

    Parameter-efficient transfer learning for nlp, 2019

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. URL https://arxiv.org/abs/1902.00751

    Pith/arXiv arXiv 2019

  18. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    Pith/arXiv arXiv 2021

  19. [19]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018

    work page 2018

  20. [20]

    Kopiczko, Tijmen Blankevoort, and Yuki M

    Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024. URL https://arxiv.org/abs/2310.11454

    Pith/arXiv arXiv 2024

  21. [21]

    Enhancing Large Language Model Performance with Gradient-Based Parameter Selection

    Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, and Peng Cheng. Enhancing large language model performance with gradient-based parameter selection, 2025. URL https://arxiv.org/abs/2406.15330

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Prefix-tuning: Optimizing continuous prompts for generation, 2021

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021. URL https://arxiv.org/abs/2101.00190

    Pith/arXiv arXiv 2021

  23. [23]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. URL https://arxiv.org/abs/2306.00978

    Pith/arXiv arXiv 2024

  24. [24]

    Alora: Allocating low-rank adaptation for fine-tuning large language models, 2024

    Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models, 2024. URL https://arxiv.org/abs/2403.16187

    Pith/arXiv arXiv 2024

  25. [25]

    Compacter: Efficient low-rank hypercomplex adapter layers, 2021

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021. URL https://arxiv.org/abs/2106.04647

    Pith/arXiv arXiv 2021

  26. [26]

    Phi-2: The surprising power of small language models

    Microsoft . Phi-2: The surprising power of small language models. https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/, 2023. Accessed: 2025-07-03

    work page 2023

  27. [27]

    Adapterhub: A framework for adapting transformers, 2020

    Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers, 2020. URL https://arxiv.org/abs/2007.07779

    Pith/arXiv arXiv 2020

  28. [28]

    Adapterfusion: Non-destructive task composition for transfer learning, 2021

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning, 2021. URL https://arxiv.org/abs/2005.00247

    Pith/arXiv arXiv 2021

  29. [29]

    Wic: 10,000 example pairs for evaluating context-sensitive representations

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: 10,000 example pairs for evaluating context-sensitive representations. CoRR, abs/1808.09121, 2018. URL http://arxiv.org/abs/1808.09121

    Pith/arXiv arXiv 2018

  30. [30]

    Know what you don ' t know: Unanswerable questions for SQ u AD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don ' t know: Unanswerable questions for SQ u AD . In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784--789, Melbourne, Australia, July 2018. Association for Computational Linguist...

    doi:10.18653/v1/p18-2124 2018

  31. [31]

    Choice of plausible alternatives: An evaluation of commonsense causal reasoning

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011

    work page 2011

  32. [32]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631--1642, 2013

    work page 2013

  33. [33]

    Sparse is enough in fine-tuning pre-trained large language models, 2024

    Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du. Sparse is enough in fine-tuning pre-trained large language models, 2024. URL https://arxiv.org/abs/2312.11875

    Pith/arXiv arXiv 2024

  34. [34]

    Training neural networks with fixed sparse masks, 2021

    Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks, 2021. URL https://arxiv.org/abs/2111.09839

    Pith/arXiv arXiv 2021

  35. [35]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  36. [36]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP, 2018

    work page 2018

  37. [37]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019

    Pith/arXiv arXiv 1905

  38. [38]

    Random masking finds winning tickets for parameter efficient fine-tuning, 2024

    Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine-tuning, 2024. URL https://arxiv.org/abs/2405.02596

    Pith/arXiv arXiv 2024

  39. [39]

    Raise a child in large language model: Towards effective and generalizable fine-tuning, 2021

    Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning, 2021. URL https://arxiv.org/abs/2109.05687

    Pith/arXiv arXiv 2021

  40. [40]

    Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection

    Shiyun Xu and Zhiqi Bu. Adaptive parameter-efficient fine-tuning via hessian-informed subset selection, 2025. URL https://arxiv.org/abs/2505.12579

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022. URL https://arxiv.org/abs/2106.10199

    arXiv 2022

  42. [42]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023 a

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023 a . URL https://arxiv.org/abs/2308.03303

    Pith/arXiv arXiv 2023

  43. [43]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023 b

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023 b . URL https://arxiv.org/abs/2303.10512

    Pith/arXiv arXiv 2023

  44. [44]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/ab...

    Pith/arXiv arXiv 2022

  45. [45]

    Gradient-based Parameter Selection for Efficient Fine-Tuning

    Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. Gradient-based parameter selection for efficient fine-tuning, 2024. URL https://arxiv.org/abs/2312.10136

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Masking as an efficient alternative to finetuning for pretrained language models, 2020

    Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking as an efficient alternative to finetuning for pretrained language models, 2020. URL https://arxiv.org/abs/2004.12406

    Pith/arXiv arXiv 2020

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.