{"id":"eb8d7b40-86df-4288-9856-d5f0d02b8752","arxiv_id":"2412.12951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FineGates uses stochastic gates to prune structured rows and columns of a transformer during fine-tuning, matching LoRA accuracy while removing up to 20-40% of weights.","lead":"FineGates learns row and column gates on a frozen language model during fine-tuning, so the model is adapted and compressed in one pass. A reader might care because it promises LoRA-level accuracy while removing up to 20-40% of parameters for faster inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method never specifies how relaxed gates become binary masks, so the reported 20–40% parameter-removal accuracies may not correspond to an actually pruned model; the central compression claim is unverifiable as written.","rationale":"The reader's weakest_assumption is exactly the load-bearing point: the paper never specifies how the relaxed stochastic gates from Eq. (5) are converted into the binary row/column masks that would justify the claimed parameter removal. This is not a cosmetic omission; it sits between the trained objective and every empirical claim about compression. If the reported accuracies at 10–20% and 20–40% sparsity were obtained by evaluating with continuous soft gates, then the 'removed parameters' do not exist at inference time, and the speedup measurements in Section 5.7 would be measuring a different operation than actual pruning. If a threshold is applied post hoc, the accuracy at the claimed sparsity could be materially worse than Table 2 and Figure 2 suggest, because the training objective never optimized the discrete mask. The ambiguity of Lsparse in Eq. (6) versus the expected-ℓ0 relaxation used in the convergence proof compounds the problem: it is unclear what objective actually produced the reported gate values. I do not see this as grounds for rejection, because the method is plausible and the missing step is readily fixable by the authors, but it is precisely the kind of unverifiable procedural gap that makes the central claim conditional rather than established. The convergence proof has additional gaps, but the binarization issue is more load-bearing because it directly determines whether the headline compression result is real. I agree with the reader's conditional verdict and recommend no change.","tokens_in":11900,"tokens_out":4305,"duration_ms":40830,"concrete_test":"Obtain or reimplement the training loop and add an explicit binarization step: after training, set ω_j=1 if the learned gate exceeds a specified threshold (or keep the top (1-s)d gates), form diag(ω_r) W0 diag(ω_c) plus diag(ω_r) W_B W_A diag(ω_c), physically remove zeroed rows/columns, and rerun the GLUE evaluations of Table 2 and Figure 2 on this pruned network. Compare against the reported numbers; if accuracy at 20–40% sparsity drops by more than the stated small margins (e.g., >2–4 points on MRPC/RTE), the central removal claim fails. Also report the distribution of learned ω values to confirm that gates are near 0/1 and not mostly intermediate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FineGates 'allows the removal of up to 20–40%' of base-model parameters with little accuracy loss. For that claim to hold, the continuous relaxed gates ω(µ)=clip(0.5+µ+ε) produced by Eq. (5) must be converted to {0,1} row/column masks that are then actually deleted from W0 (and from W_BW_A in the +LoRA variant), and the accuracies in Table 2 / Figure 2 must be measured on that binarized, physically pruned network. The paper never states this conversion: no threshold, no top-k selection, no description of how the sparsity ratio s in Eq. (6) is enforced, and no statement that the reported accuracies are post-binarization rather than with continuous soft masks. Lsparse is written as λ·max(||ω||0,s), but ||ω||0 of a continuous vector is not the expected Bernoulli count used in the convergence section; the paper silently switches to E||ω||0 = Σ P(ω_j>0), leaving the actual training objective ambiguous. Section 5.7 measures wall-clock speedup of 'W^T·ω' and '(W^T·ω)(X·ω)' operations, but if ω remains continuous, multiplying by it does not remove parameters and only adds indexing overhead; the speedup plot is therefore not evidence that the compressed model exists. Without a precise binarization rule, the reader cannot tell whether the 10–20%/20–40% rows of Table 2 and Figure 2 reflect a real pruned model, a soft-mask approximation, or a post-hoc threshold that might degrade accuracy well beyond the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FineGates, a parameter-efficient fine-tuning method that learns stochastic gates on the rows and columns of the frozen base model's weight matrices, optionally combined with LoRA-style low-rank updates. The method is evaluated on GLUE tasks with RoBERTa-base and RoBERTa-large, reporting accuracy comparable to LoRA and full fine-tuning while removing up to 20-40% of base-model parameters. A convergence proof for the relaxed objective is also provided.","tokens_in":12331,"tokens_out":5433,"duration_ms":47058,"significance":"If the central claim holds, the method would be valuable because it achieves task adaptation and structured compression in a single training pass with very few trainable parameters (0.17M for RoBERTa-base), avoiding post-hoc pruning. The use of stochastic gates for structured sparsity is an interesting and relevant idea. However, the empirical claim is under-supported: the paper does not specify how continuous gates are binarized to {0,1} masks, reports only median scores without variance, and its convergence proof applies to a smoothed objective under unverified assumptions. The paper does not release code, which further limits reproducibility.","major_comments":[{"comment":"The paper never specifies the procedure that converts the continuous relaxed gates ω(µ) = clip(0.5+µ+ε) into the binary {0,1} row/column masks needed to physically remove parameters. The objective uses max(||ω||_0, s), but for continuous ω the ℓ0 norm is not differentiable and is not the expected Bernoulli count used in the convergence section; the training objective is therefore ambiguous. Without a thresholding, top-k, or rounding rule, the reported parameter removals (-12M, -25M, -70M in Table 2) and the sparsity-accuracy curves in Figure 2 cannot be reproduced, and the central claim of 'removal of up to 20-40%' is not verifiable.","section":"§4, Eqs. (5)-(6) and Table 2"},{"comment":"The speedup experiments measure the time of (W^T·ω)(X·ω) with the same ω used during training, which is continuous. If ω is not binarized, this operation does not delete any matrix entries; it only multiplies by a dense mask and adds indexing overhead. The reported wall-clock speedups therefore do not demonstrate that the compressed model exists or that the stated sparsity levels translate into real inference gains. The authors must clarify whether the ω used in the speedup measurements is the binary mask and how it was obtained.","section":"§5.7 and Figure 3"},{"comment":"The convergence proof analyzes the smoothed objective with h(µ) equal to the expected ℓ0 norm, not the actual objective (9) which contains the non-smooth term λ·max(||ω||_0, s) with a target sparsity s. The proof also assumes bounded W and L-smooth bounded f, but boundedness of W during training is not verified, and the argument does not account for the constraint s or the non-smooth max. As written, the proof establishes smoothness of a related relaxed objective, not convergence of the method as implemented.","section":"§6, Proposition 1"},{"comment":"Only median accuracies over five seeds are reported, with no standard deviations, confidence intervals, or statistical tests. Given that several comparisons in Table 2 differ by less than one point (e.g., FineGates vs LoRA on STS-B, RTE, QQP), the claim that FineGates is 'comparable' or 'better' than baselines is not supported by the reported statistics.","section":"§5.1"}],"minor_comments":[{"comment":"The phrase 'like like quantization and pruning' contains a duplicated word.","section":"§1"},{"comment":"The sentence 'We observe, that the rank varies' has an unnecessary comma; also, the statement that FineGates 'obtains comparable results without pruning attention heads' is confusing because the method is supposed to prune the base model, so clarify what 'without extensive pruning' means.","section":"§5.6"},{"comment":"The caption says 'reducing only 4% of accuracy' while the text says 'a loss of 4% in accuracy'; accuracy differences should be reported as absolute percentage points, not percent relative, to avoid ambiguity.","section":"§5.4 and Figure 2"},{"comment":"The claim that the method provides '>40% of structured sparsity' on SST2 is not tied to any explicit sparsity enforcement procedure; clarify how the sparsity level is set and measured.","section":"§5.4"},{"comment":"The conclusion says 'removal of up to 20−30% parameters in the attention matrices', while the abstract and Section 1 claim '20-40%'; the inconsistency should be resolved.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim depends on an unspecified binarization of the stochastic gates. I would urge the editor to require the authors to provide the exact thresholding procedure, report post-binarization accuracy, and ideally release code before acceptance. The convergence proof, as written, does not cover the actual objective and should either be revised or removed. The comparison to prior stochastic-gate work is appropriately cited; the novelty lies in applying it to frozen LLM weight matrices, but the missing implementation details currently block verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: FineGates is a genuinely new combination — stochastic row/column gates learned on frozen transformer weights during fine-tuning, with an optional LoRA path — and the small-scale GLUE numbers look competitive. But the paper's central claim, that you can actually remove 20–40% of base-model parameters, is under-specified to the point of being unverifiable. No binarization rule for turning the relaxed gates ω(µ) into {0,1} masks is given, and the speedup experiment as described appears to measure soft-mask multiplication, not pruned inference.\n\nThe paper does some things well. The ablations in Table 4 (with/without W_mlp, with/without LoRA) are useful. The comparison to APT is honest: they acknowledge their sparsity doesn't include heads or embeddings, and they still get comparable numbers at fixed sparsity with a simpler training loop. The trainable-parameter counts (0.17M on RoBERTa-base) are legitimately small. The convergence proof, while not deep, is standard for a smoothed relaxation.\n\nThe soft spots are real and load-bearing. The stress-test is right: Lsparse in Eq. (6) is written with an ℓ0 norm on continuous ω, while the convergence section switches to the expected Bernoulli count. That's a different objective, and the paper doesn't say which one is actually optimized. More importantly, Table 2's sparsity rows report `s >= 10%` and `s >= 20%`, but there is no description of how the target sparsity is enforced exactly — top-k thresholding? projected gradient? — and no statement that the accuracies are measured after binarizing and physically removing rows/columns. If the gates stay continuous, then `ω·W·ω` is just a scaled weight, not a pruned one, and Figure 3's speedup is measuring dense matrix multiplication with a scaling factor. That's a significant omission, not a stylistic one.\n\nAlso, only median scores over five seeds are reported, with no variance or significance, and no code is provided. These are fixable, but they currently prevent an independent check of the main claim.\n\nBottom line: the idea is plausible and worth taking seriously, but the paper as written does not demonstrate that a pruned model with 20–40% fewer parameters achieves the reported accuracies. I'd send it to review — a good referee could push the authors to specify the binarization and release code — but I wouldn't trust the headline number until that's clarified.\n\nWho's this for: PEFT and structured-pruning researchers. I'd bring it to reading group to talk about what counts as actual pruning.\n\nRecommendation: peer review, with major revision required.","headline":"Novel gate-based PEFT with competitive numbers, but the missing binarization rule makes the core compression claim unverifiable as written.","tokens_in":12773,"tokens_out":2815,"would_cite":true,"duration_ms":25166,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stochastic row and column gates trained during fine-tuning can remove 20–40% of a frozen transformer's parameters with little accuracy loss while matching or beating LoRA.","keywords":["stochastic gates","structured sparsity","parameter-efficient finetuning","low-rank adaptation","LLM compression","GLUE benchmark","RoBERTa"],"falsifier":"Re-run the FineGates experiments, then hard-threshold the trained gate vectors at a grid of values in [0,1], count exactly how many parameters remain, and measure GLUE validation accuracy after removal; if no threshold yields the reported accuracy at the claimed removed-parameter count, the central claim fails.","tokens_in":1825,"feed_emoji":"✂️","tokens_out":2575,"duration_ms":85829,"temperature":0.7,"pith_summary":"FineGates is a finetuning method that trains stochastic gate vectors on the frozen weights of a transformer while the model is being adapted to a downstream task. Each gate multiplies an entire row or column of a projection or feed-forward matrix, so once training ends the learned values can be converted into structured masks that remove whole dimensions. The paper claims this removes 20–40% of base-model parameters with little accuracy loss, using about 0.17M trainable parameters on RoBERTa-base, and that accuracy is comparable to LoRA and often better than full finetuning on GLUE tasks. The payoff, if the claim holds, is that task adaptation and compression happen in one training pass with no separate pruning or distillation stage.","feed_headline":"Fine-tuning with stochastic gates drops up to 40% of LLM weights","feed_subtitle":"One pass learns which weight columns to keep, matching LoRA accuracy with a fraction of the trainable parameters.","key_machinery":"The load-bearing object is the stochastic gates vector $\\omega(\\mu) = \\max(0, \\min(1, 0.5 + \\mu + \\epsilon))$ with $\\epsilon \\sim \\mathcal{N}(0, \\sigma^2)$ and $\\sigma = 0.5$, a Gaussian relaxation of Bernoulli variables; the trainable parameter $\\mu$ lives in $[-1,1]$ and the relaxed gate is regularized by the expected $\\ell_0$ norm of $\\omega$, $L_{\\text{sparse}} = \\max(\\mathbb{E}\\|\\omega\\|_0, s)$. A separate $\\omega_r$ and $\\omega_c$ multiply rows and columns of each $W_0$, turning adaptation into a search for which matrix dimensions the task actually needs. Because the gates are continuous during training, gradients flow through the reparameterized noise; because the regularizer counts the expected number of active gates, the model is pushed toward a target sparsity ratio $s$. The optional low-rank extension $W_B W_A$ from LoRA is multiplied by the same gates, giving a combined adapter-pruner.","core_discovery":"The paper's central claim is that structured sparsity can be learned at the same time as task adaptation by multiplying each base weight matrix $W_0$ by two trainable stochastic gate vectors, $\\omega_r$ and $\\omega_c$, so the forward pass becomes $h = [\\omega_r \\cdot W_0 \\cdot \\omega_c] x$. The gates are relaxed Bernoulli variables (Eq. 5), trained with an expected-$\\ell_0$ regularizer that pushes them toward the target sparsity ratio $s$ (Eq. 6). In experiments on GLUE with at most 10K training samples per task, FineGates matches or beats LoRA accuracy with fewer trainable parameters (0.17M vs 0.3M on RoBERTa-base) and removes 10–20% of base-model parameters at negligible accuracy cost, with the sparsification sweep showing 20–40% removal at a cost of at most a few metric points. The method also comes in a simplified variant that trains only the gates and task head, and an extended variant that adds LoRA-style low-rank matrices.","pith_inferences":["Editorial inference: if the gate-to-binary-mask conversion were specified, the 20–40% numbers could be verified exactly; until then, the reported sparsity should be read as a bound on what the learned gates allow, not a measured property of a deployed pruned model.","Editorial inference: the same row/column gate mechanism could be reused across multiple tasks or during continued pretraining to identify reusable sparse sub-networks, which the paper lists as future work but does not test.","Editorial inference: because the simplified variant gates only the frozen base weights, the selected dimensions could be compared against task-specific attention or probing statistics to see whether the gates find semantically meaningful subspaces; the paper does not analyze what the gates select."],"forward_implications":["FineGates reaches LoRA-comparable or better accuracy on the subsampled GLUE tasks while training only about 0.17M parameters on RoBERTa-base, roughly 0.14% of the base model.","The same run that adapts the model also compresses it: removing 10–20% of base-model parameters costs at most a few tenths of a point on most tasks, and the sparsification sweep shows 20–40% removal with a drop of at most 4 accuracy points.","No post-training pruning pass or distillation stage is needed; compression is enforced in the finetuning objective itself.","Removing whole rows and columns yields measured wall-clock speedups for matrix multiplication and single-epoch inference, proportional to the achieved column reduction.","Optional low-rank matrices can be combined with the gates (FineGates+LoRA), and gating can be restricted to attention projections only (FineGates w/o $W_{\\text{mlp}}$) with small accuracy trade-offs."],"supporting_citations":[{"why":"Supplies the LoRA low-rank decomposition $W_B W_A$ and the zero/random initialization scheme that FineGates wraps with its gates.","marker":"[Hu et al., 2021]"},{"why":"Provides the stochastic-gate relaxation and the expected $\\ell_0$ regularization that Eq. (5) and Eq. (6) are built on.","marker":"[Yamada et al., 2020]"},{"why":"Defines the RoBERTa-base and RoBERTa-large backbones used in every experiment.","marker":"[Liu, 2019]"},{"why":"Supplies the GLUE benchmark datasets and evaluation metrics in Tables 2 and 3.","marker":"[Wang et al., 2019]"},{"why":"APT is the closest adaptive-pruning baseline; the MRPC comparison in Table 5 frames the sparsity-accuracy trade-off claim.","marker":"[Zhao et al., 2023]"},{"why":"CoFi is the structured-pruning predecessor whose attention-head pruning extension the paper says it plans to adopt.","marker":"[Xia et al., 2022]"},{"why":"Supplies the vanilla SGD convergence result invoked in the Proposition 1 proof for FineGates.","marker":"[Ketkar and Ketkar, 2017]"}],"fun_headline_variants":["Stochastic gates cut up to 40% of LLM weights while fine-tuning","Fine-tune and prune: learned gates drop 40% of LLM parameters","FineGates: prune LLM weights during fine-tuning, up to 40% off","Learn which weight columns to keep: gates prune 40% in fine-tuning","Prunes 40% of LLM weights while matching LoRA accuracy"],"cache_read_input_tokens":14848,"weakest_assumption_plain":"The paper assumes that the continuous gate values learned during training can be converted into binary row/column masks that actually remove the stated 10–40% of parameters, but the conversion procedure is never specified.","fun_headline_variants_meta":{"raw":{"variants":["Stochastic gates cut up to 40% of LLM weights while fine-tuning","Fine-tune and prune: learned gates drop 40% of LLM parameters","FineGates: prune LLM weights during fine-tuning, up to 40% off","Learn which weight columns to keep: gates prune 40% in fine-tuning","Prunes 40% of LLM weights while matching LoRA accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2999,"prompt_tokens":970,"completion_tokens":2029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":586,"tokens_out":2029,"duration_ms":12718,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:33:22.918454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the FineGates experiments, then hard-threshold the trained gate vectors at a grid of values in [0,1], count exactly how many parameters remain, and measure GLUE validation accuracy after removal; if no threshold yields the reported accuracy at the claimed removed-parameter count, the central claim fails.","supporting_citations":[],"review_version":1}