Pith. sign in

REVIEW 4 major objections 5 minor 21 references

FineGates: LLMs Finetuning with Compression using Stochastic Gates

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Stochastic row and column gates trained during fine-tuning can remove 20–40% of a frozen transformer's parameters with little accuracy loss while matching or beating LoRA.

desk verdict Novel gate-based PEFT with competitive numbers, but the missing binarization rule makes the core compression claim unverifiable as written. read the letter →

arxiv 2412.12951 v1 pith:YYALA35X submitted 2024-12-17 cs.LG

Jonathan Svirsky , Yehonathan Refael , Ofir Lindenbaum This is my paper · ORCID
classification cs.LG
keywords stochasticgatesstructuredsparsityparameter-efficientfinetuninglow-rankadaptationLLMcompressionGLUEbenchmarkRoBERTa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FineGates is a finetuning method that trains stochastic gate vectors on the frozen weights of a transformer while the model is being adapted to a downstream task. Each gate multiplies an entire row or column of a projection or feed-forward matrix, so once training ends the learned values can be converted into structured masks that remove whole dimensions. The paper claims this removes 20–40% of base-model parameters with little accuracy loss, using about 0.17M trainable parameters on RoBERTa-base, and that accuracy is comparable to LoRA and often better than full finetuning on GLUE tasks. The payoff, if the claim holds, is that task adaptation and compression happen in one training pass with no separate pruning or distillation stage.

What carries the argument

The load-bearing object is the stochastic gates vector $\omega(\mu) = \max(0, \min(1, 0.5 + \mu + \epsilon))$ with $\epsilon \sim \mathcal{N}(0, \sigma^2)$ and $\sigma = 0.5$, a Gaussian relaxation of Bernoulli variables; the trainable parameter $\mu$ lives in $[-1,1]$ and the relaxed gate is regularized by the expected $\ell_0$ norm of $\omega$, $L_{\text{sparse}} = \max(\mathbb{E}\|\omega\|_0, s)$. A separate $\omega_r$ and $\omega_c$ multiply rows and columns of each $W_0$, turning adaptation into a search for which matrix dimensions the task actually needs. Because the gates are continuous during training, gradients flow through the reparameterized noise; because the regularizer counts the expected number of active gates, the model is pushed toward a target sparsity ratio $s$. The optional low-rank extension $W_B W_A$ from LoRA is multiplied by the same gates, giving a combined adapter-pruner.

What would settle it

Re-run the FineGates experiments, then hard-threshold the trained gate vectors at a grid of values in [0,1], count exactly how many parameters remain, and measure GLUE validation accuracy after removal; if no threshold yields the reported accuracy at the claimed removed-parameter count, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured sparsity can be learned at the same time as task adaptation by multiplying each base weight matrix $W_0$ by two trainable stochastic gate vectors, $\omega_r$ and $\omega_c$, so the forward pass becomes $h = [\omega_r \cdot W_0 \cdot \omega_c] x$. The gates are relaxed Bernoulli variables (Eq. 5), trained with an expected-$\ell_0$ regularizer that pushes them toward the target sparsity ratio $s$ (Eq. 6). In experiments on GLUE with at most 10K training samples per task, FineGates matches or beats LoRA accuracy with fewer trainable parameters (0.17M vs 0.3M on RoBERTa-base) and removes 10–20% of base-model parameters at negligible accuracy cost, with the sparsification sweep showing 20–40% removal at a cost of at most a few metric points. The method also comes in a simplified variant that trains only the gates and task head, and an extended variant that adds LoRA-style low-rank matrices.

Load-bearing premise

The paper assumes that the continuous gate values learned during training can be converted into binary row/column masks that actually remove the stated 10–40% of parameters, but the conversion procedure is never specified.

Editorial extensions

If this is right

  • FineGates reaches LoRA-comparable or better accuracy on the subsampled GLUE tasks while training only about 0.17M parameters on RoBERTa-base, roughly 0.14% of the base model.
  • The same run that adapts the model also compresses it: removing 10–20% of base-model parameters costs at most a few tenths of a point on most tasks, and the sparsification sweep shows 20–40% removal with a drop of at most 4 accuracy points.
  • No post-training pruning pass or distillation stage is needed; compression is enforced in the finetuning objective itself.
  • Removing whole rows and columns yields measured wall-clock speedups for matrix multiplication and single-epoch inference, proportional to the achieved column reduction.
  • Optional low-rank matrices can be combined with the gates (FineGates+LoRA), and gating can be restricted to attention projections only (FineGates w/o $W_{\text{mlp}}$) with small accuracy trade-offs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the gate-to-binary-mask conversion were specified, the 20–40% numbers could be verified exactly; until then, the reported sparsity should be read as a bound on what the learned gates allow, not a measured property of a deployed pruned model.
  • Editorial inference: the same row/column gate mechanism could be reused across multiple tasks or during continued pretraining to identify reusable sparse sub-networks, which the paper lists as future work but does not test.
  • Editorial inference: because the simplified variant gates only the frozen base weights, the selected dimensions could be compared against task-specific attention or probing statistics to see whether the gates find semantically meaningful subspaces; the paper does not analyze what the gates select.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FineGates, a parameter-efficient fine-tuning method that learns stochastic gates on the rows and columns of the frozen base model's weight matrices, optionally combined with LoRA-style low-rank updates. The method is evaluated on GLUE tasks with RoBERTa-base and RoBERTa-large, reporting accuracy comparable to LoRA and full fine-tuning while removing up to 20-40% of base-model parameters. A convergence proof for the relaxed objective is also provided.

Significance. If the central claim holds, the method would be valuable because it achieves task adaptation and structured compression in a single training pass with very few trainable parameters (0.17M for RoBERTa-base), avoiding post-hoc pruning. The use of stochastic gates for structured sparsity is an interesting and relevant idea. However, the empirical claim is under-supported: the paper does not specify how continuous gates are binarized to {0,1} masks, reports only median scores without variance, and its convergence proof applies to a smoothed objective under unverified assumptions. The paper does not release code, which further limits reproducibility.

major comments (4)
  1. [§4, Eqs. (5)-(6) and Table 2] The paper never specifies the procedure that converts the continuous relaxed gates ω(µ) = clip(0.5+µ+ε) into the binary {0,1} row/column masks needed to physically remove parameters. The objective uses max(||ω||_0, s), but for continuous ω the ℓ0 norm is not differentiable and is not the expected Bernoulli count used in the convergence section; the training objective is therefore ambiguous. Without a thresholding, top-k, or rounding rule, the reported parameter removals (-12M, -25M, -70M in Table 2) and the sparsity-accuracy curves in Figure 2 cannot be reproduced, and the central claim of 'removal of up to 20-40%' is not verifiable.
  2. [§5.7 and Figure 3] The speedup experiments measure the time of (W^T·ω)(X·ω) with the same ω used during training, which is continuous. If ω is not binarized, this operation does not delete any matrix entries; it only multiplies by a dense mask and adds indexing overhead. The reported wall-clock speedups therefore do not demonstrate that the compressed model exists or that the stated sparsity levels translate into real inference gains. The authors must clarify whether the ω used in the speedup measurements is the binary mask and how it was obtained.
  3. [§6, Proposition 1] The convergence proof analyzes the smoothed objective with h(µ) equal to the expected ℓ0 norm, not the actual objective (9) which contains the non-smooth term λ·max(||ω||_0, s) with a target sparsity s. The proof also assumes bounded W and L-smooth bounded f, but boundedness of W during training is not verified, and the argument does not account for the constraint s or the non-smooth max. As written, the proof establishes smoothness of a related relaxed objective, not convergence of the method as implemented.
  4. [§5.1] Only median accuracies over five seeds are reported, with no standard deviations, confidence intervals, or statistical tests. Given that several comparisons in Table 2 differ by less than one point (e.g., FineGates vs LoRA on STS-B, RTE, QQP), the claim that FineGates is 'comparable' or 'better' than baselines is not supported by the reported statistics.
minor comments (5)
  1. [§1] The phrase 'like like quantization and pruning' contains a duplicated word.
  2. [§5.6] The sentence 'We observe, that the rank varies' has an unnecessary comma; also, the statement that FineGates 'obtains comparable results without pruning attention heads' is confusing because the method is supposed to prune the base model, so clarify what 'without extensive pruning' means.
  3. [§5.4 and Figure 2] The caption says 'reducing only 4% of accuracy' while the text says 'a loss of 4% in accuracy'; accuracy differences should be reported as absolute percentage points, not percent relative, to avoid ambiguity.
  4. [§5.4] The claim that the method provides '>40% of structured sparsity' on SST2 is not tied to any explicit sparsity enforcement procedure; clarify how the sparsity level is set and measured.
  5. [§7] The conclusion says 'removal of up to 20−30% parameters in the attention matrices', while the abstract and Section 1 claim '20-40%'; the inconsistency should be resolved.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: FineGates' accuracy and compression claims are empirical and benchmarked against external baselines; self-citations supply method components, not evidence.

full rationale

I walked the derivation chain from Eqs. (3)-(9) through Section 5 and Section 6. No result claimed as a prediction is equivalent to an input by construction. The sparsity constraint Lsparse = max(||omega||_0,s) (Eq. 6) does make sparsity a training target rather than a discovered outcome; however, the paper's actual claim is that this enforced sparsity can reach 20-40% without significant accuracy loss, and those accuracies are empirical measurements compared against full finetuning, LoRA, VeRA, LoRA-XS, LoRA-FA, and APT (Tables 2, 3, and 5). The convergence proof (Proposition 1) is a standard smoothness argument: it assumes L-smoothness of f and cites Yamada et al. (2020) only for the closed-form Gaussian CDF of the relaxed gate expectation; it does not import the accuracy or sparsity conclusion. Self-citations to Yamada et al., Jana et al., Svirsky-Lindenbaum, and Lindenbaum et al. are used as building blocks for the stochastic-gate relaxation, not as verification of the FineGates results. The manuscript does leave the continuous-to-binary conversion unspecified (no threshold or top-k rule for omega(mu) after Eq. 5), which is a reproducibility and verifiability gap, but it is not a case where a quantity is defined in terms of the quantity it is supposed to predict. Therefore no specific circular step can be exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The empirical central claim is benchmarked against external baselines, but several hyperparameters are unreported and the convergence argument relies on unverified smoothness, boundedness, and a smoothed objective that differs from the implemented one.

free parameters (4)
  • lambda (sparsity regularization weight) = not reported
    Hyperparameter in Eq. (9); no value or sensitivity analysis is given.
  • sigma (noise std in gate relaxation) = 0.5
    Fixed by hand in Eq. (5); controls the stochastic gate behavior.
  • s (target sparsity) = 10%, 20%, 40%
    User-set sparsity target in Eq. (6); defines the reported compression levels.
  • LoRA rank r = 8 (16/32/64 in Table 5)
    Rank for the FineGates+LoRA variant; not swept for the main results.
assumptions (4)
  • domain assumption The task loss f = L_task is L-smooth, non-convex, and bounded by M.
    Stated in Proposition 1 (Section 6). No verification that transformer cross-entropy loss satisfies this, and no Lipschitz constant is estimated.
  • ad hoc to paper Weight matrices W are bounded throughout training.
    Needed in Section 6 to bound the gradient Lipschitz terms, but not stated as an assumption and not guaranteed by the training setup.
  • ad hoc to paper The relaxed expected l0 norm is equivalent to the Lsparse term max(||omega||_0,s).
    Eq. (6) defines Lsparse with max(||omega||_0,s), while Section 6 proves properties of the expected Bernoulli count; no equivalence is shown.
  • domain assumption The stochastic gate relaxation converges to binary values.
    Inherited from Yamada et al. 2020; the paper does not prove or verify convergence to exactly {0,1}.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FineGates: LLMs Finetuning with Compression using Stochastic Gates." pith.science (2026). https://pith.science/paper/YYALA35X

@misc{pith2026241212951,
  author       = {Pith},
  title        = {Pith review of: FineGates: LLMs Finetuning with Compression using Stochastic Gates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYALA35X}},
  note         = {Machine review of arXiv:2412.12951}
}
read the original abstract

Large Language Models (LLMs), with billions of parameters, present significant challenges for full finetuning due to the high computational demands, memory requirements, and impracticality of many real-world applications. When faced with limited computational resources or small datasets, updating all model parameters can often result in overfitting. To address this, lightweight finetuning techniques have been proposed, like learning low-rank adapter layers. These methods aim to train only a few additional parameters combined with the base model, which remains frozen, reducing resource usage and mitigating overfitting risks. In this work, we propose an adaptor model based on stochastic gates that simultaneously sparsify the frozen base model with task-specific adaptation. Our method comes with a small number of trainable parameters and allows us to speed up the base model inference with competitive accuracy. We evaluate it in additional variants by equipping it with additional low-rank parameters and comparing it to several recent baselines. Our results show that the proposed method improves the finetuned model accuracy comparatively to the several baselines and allows the removal of up to 20-40\% without significant accuracy loss.

Figures

Figures reproduced from arXiv: 2412.12951 by the authors.

Figure 1
Figure 1. Two versions of our method: (a) In the first one, we train an adaptor with additional weights WA,WB. After training we compute the updated and pruned weight matrix W˜ = ωr · (W0 + WBWA) · ωc. (b) In the simplified version, the adaptor is based only on the trainable gates vectors ωl , ωr that enforce structured sparsity. Assuming that ω is a Bernoulli variable, we calculate its expected ℓ0 norm as follows: E||ω||0 = … view at source ↗
Figure 2
Figure 2. Sparsification-Accuracy trade-off measured on CoLA, SST2, and STSB datasets. Our model provides > 40% of structured sparsity while sacrificing only 4% of accuracy compared to the model without sparsification on the SST2 dataset where we train ωr, ωc with total 166K parameters. On CoLA the method reduces up to 20% of parameters without significant loss in accuracy, and 40% on the STSB dataset with only 3% drop in acc… view at source ↗
Figure 3
Figure 3. (a) Measuring relative time reduction in multiplication (WT · ω)(X · ω) compared to full matrices multiplication WTX. We measure CPU time by repeating the operation 100K times and reporting the average time (vertical line) for each sparsity level (horizontal line). (b) Measuring inference time for a single validation epoch with varying sparsity levels. 5.7 Inference Speedup Matrix multiplication speedup We now asses… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 linked inside Pith

  1. [1]

    Lora-xs: Low-rank adaptation with extremely small number of parameters

    Klaudia Ba lazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604,

    arXiv

  2. [3]

    One-for-all: Gen- eralized lora for parameter-efficient fine-tuning

    Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Gen- eralized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.07967 ,

    arXiv

  3. [5]

    Sparseadapter: An easy approach for improving the parameter-efficiency of adapters

    Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. arXiv preprint arXiv:2210.04284,

    arXiv

  4. [9]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 ,

    arXiv

  5. [11]

    Parameter-efficient sparsity for large language models fine-tuning

    Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. Parameter-efficient sparsity for large language models fine-tuning. arXiv preprint arXiv:2205.11005,

    arXiv

  6. [12]

    Nora: Nested low-rank adaptation for efficient fine-tuning large models

    Cheng Lin, Lujun Li, Dezhi Li, Jie Zou, Wenhan Luo, Wei Xue, and Yike Guo. Nora: Nested low-rank adaptation for efficient fine-tuning large models. arXiv preprint arXiv:2408.10280, 2024a. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantiz...

    arXiv

  7. [13]

    Roberta: A robustly optimized bert pretraining approach

    Y Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

    arXiv 1907

  8. [14]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

Show all 21 references
  1. [15]

    Adapterfusion: Non-destructive task composition for transfer learning

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R¨ uckl´ e, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247,

    2005 arXiv

  2. [17]

    Knowledge editing in lan- guage models via adapted direct preference optimization.arXiv preprint arXiv:2406.09920,

    Amit Rozner, Barak Battash, Lior Wolf, and Ofir Lindenbaum. Knowledge editing in lan- guage models via adapted direct preference optimization.arXiv preprint arXiv:2406.09920,

    arXiv

  3. [19]

    Qa-lora: Quantization-aware low-rank adaptation of large language models

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717 ,

    arXiv

  4. [20]

    Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning

    15 Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023a. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He,...

    arXiv

  5. [21]

    Apt: Adaptive pruning and tun- ing pretrained language models for efficient training and inference

    Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: Adaptive pruning and tun- ing pretrained language models for efficient training and inference. arXiv preprint arXiv:2401.12200,

    arXiv

  6. [2016]

    Structured pruning learns compact and accurate models

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408 ,

    arXiv

  7. [2017]

    Vera: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454 ,

    arXiv

  8. [2019]

    Lora: Low-rank adaptation of large language models

    13 Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  9. [2020]

    Adarankgrad: Adaptive gradient-rank and moments for memory-efficient llms training and fine-tuning

    Yehonathan Refael, Jonathan Svirsky, Boris Shustin, Wasim Huleihel, and Ofir Lindenbaum. Adarankgrad: Adaptive gradient-rank and moments for memory-efficient llms training and fine-tuning. arXiv preprint arXiv:2410.17881 ,

  10. [2021]

    Prefix-tuning: Optimizing continuous prompts for genera- tion

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for genera- tion. arXiv preprint arXiv:2101.00190 ,

    arXiv

  11. [2022]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  12. [2023]

    Mixture-of-loras: An efficient multitask tuning for large language models

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models. arXiv preprint arXiv:2403.03432,

    arXiv

  13. [2024]

    Low-rank quantization-aware training for llms

    Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel. Low-rank quantization-aware training for llms. arXiv preprint arXiv:2406.06385 ,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.