Pith. sign in

REVIEW 4 major objections 8 minor 2 cited by

You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PruneNet claims that a learned stochastic policy can prune LLM feed-forward layers without calibration data by preserving the singular-value spectrum of each weight matrix.

desk verdict A novel calibration-free pruning policy with a real transfer result, but the core reward is off by one and the tables disagree; fix those before trusting the numbers. read the letter →

arxiv 2501.15296 v3 pith:WGBE4C7I submitted 2025-01-25 cs.CL

Ayan Sengupta , Siddhant Chaudhary , Tanmoy Chakraborty This is my paper · ORCID
classification cs.CL MSC 68T0768T5015A1862G10
keywords calibration-freepruningstructuredpolicylearningspectralpreservationKolmogorov-SmirnovdistanceLLMcompressionREINFORCEsingularvaluedistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PruneNet tries to establish that structured pruning of large language models can be done without any calibration dataset by treating pruning as a policy-learning problem over the model's own weights. The policy learns which rows of each feed-forward (FFN1) weight matrix to keep, and is trained with REINFORCE to minimize the Kolmogorov-Smirnov distance between the singular-value distributions of the original and pruned matrices. The paper reports that this spectral-preservation objective is enough to retain over 80% of LLaMA-2-7B's zero-shot accuracy at 30% compression, with pruning finished in about 15 minutes, while the calibration-dependent baseline SliceGPT retains about 75% at the same compression. A reader should care because calibration data is currently a bottleneck and a data-quality vulnerability in LLM compression, and PruneNet removes it by relying on intrinsic model structure alone.

What carries the argument

The central object is a stochastic row-selection policy: a lightweight MLP that maps each FFN1 weight matrix to row-importance scores and samples the rows to keep. It is trained with the REINFORCE algorithm, using as penalty the Kolmogorov-Smirnov distance between the empirical cumulative distributions of singular values of the original and pruned FFN1 matrices, with a discounted future penalty across layers. Corollary 3.3, derived from the Poincaré separation theorem, supplies the theoretical motivation: row slicing shrinks the range of the spectrum, so the policy is asked to minimize the resulting distributional shift. Because the policy learner is decoupled from the LLM, the same learned policy can be reused at different compression ratios without retraining.

What would settle it

Run PruneNet at 30% compression on LLaMA-2-7B with the KS penalty inverted, so the policy maximizes rather than minimizes the singular-value distribution shift; if the five-task zero-shot average stays near the reported 55.45, then preserving the spectral distribution is not what carries the result.

Watch

Extended reading notes

Core claim

The central claim is that a small, model-independent policy can decide which rows of an LLM's feed-forward weight matrices to prune by learning to keep the empirical singular-value distribution of each FFN1 matrix nearly unchanged. For each layer, the policy MLP computes row-importance scores, samples the rows to retain through the reparametrization trick, and is optimized by REINFORCE with a penalty equal to the Kolmogorov-Smirnov distance between the singular-value distributions of the uncompressed and compressed matrices; the same sampled indices are then applied to FFN2 and the biases. The paper motivates this objective with Corollary 3.3, derived from the Poincaré separation theorem, which says that slicing rows shrinks the range of the spectrum, so minimizing that distributional shift is a way to limit information loss. The paper reports that with this objective LLaMA-2-7B compresses in about 15 minutes without calibration data, retaining over 80% of average zero-shot accuracy at 30% compression (55.45 versus 69.00 dense and 51.50 for SliceGPT).

Load-bearing premise

The method assumes that keeping the spread of singular values of the feed-forward weight matrices nearly unchanged is enough to keep the model's downstream task accuracy, even though the pruning policy never sees any task examples.

Editorial extensions

If this is right

  • Structured compression no longer requires a calibration dataset, so the pruning step can be run in about 15 minutes for a 7B-parameter model on a single GPU, with no task labels or external data.
  • One learned policy transfers across compression ratios: reusing a policy trained at 40% or 10% on the same model costs less than 1% average zero-shot accuracy relative to a ratio-matched policy, and still beats SliceGPT by roughly 3%.
  • Pruning FFN layers alone yields real inference speedups: at 30% compression, the compressed LLaMA-2-7B generates about 20.74 tokens per second versus 11.96 for the dense model and 12.82 for SliceGPT.
  • Recovery fine-tuning is nearly optional: fine-tuning on WikiText2 improves the compressed LLaMA-2-7B by only about 1.5%, and varying the fine-tuning dataset or data size changes results only slightly.
  • At equal compression ratios, PruneNet surpasses SVD-based spectral methods that require calibration (ASVD and SVD-LLM) by about 8.5% average accuracy on LLaMA-1-7B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The KS objective constrains only the marginal distribution of singular values per matrix, not how the remaining rows interact with the rest of the network. A natural test is whether adding a constraint on singular vectors or on the layer's input-output map closes the remaining gap to the dense model; if it does not, the marginal spectrum alone may be the limiting factor.
  • Because the policy never sees task data, the method's reliability rests on spectral similarity being a universal proxy for downstream accuracy. A stress test on out-of-distribution domains (e.g., code or multilingual benchmarks) would show whether the proxy holds where pre-training and evaluation distributions differ.
  • The reusable-policy result suggests a further extension the paper leaves implicit: one policy trained on one model of a family might transfer to larger or smaller siblings without retraining, turning compression into a one-time cost per architecture family rather than per model and per ratio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. PruneNet proposes a calibration-free structured-pruning method for LLMs. A shared MLP policy (Winter, Wproj) trained with REINFORCE scores the rows of each FFN1 weight matrix; a subset of rows is sampled via a Gumbel-style reparametrization (Eq. 4), and the same indices are applied to FFN2 and the biases. The immediate penalty is the Kolmogorov-Smirnov distance between the empirical singular-value distributions of the original and pruned FFN1 matrices (Eq. 6), accumulated into a discounted return (Eq. 7). Experiments on LLaMA-1/2-7B, Phi-2, and OPT models report that at 30% nominal compression PruneNet retains about 80% of LLaMA-2-7B zero-shot accuracy (55.45 average vs 51.50 for SliceGPT, Table 2), compresses in about 15 minutes (Table 9a), transfers policies across compression ratios (Table 6), and makes recovery fine-tuning largely unnecessary (Table 3). The claimed novelty is decoupling pruning from calibration data via a spectral-preservation objective.

Significance. If the central claims survived scrutiny, the paper would be a solid practical contribution: a calibration-free pipeline with a reusable policy, a 15-minute compression time on LLaMA-2-7B, broad model coverage (LLaMA-1/2-7B, Phi-2, OPT-125M/2.7B/6.7B), and credit-worthy ablations (random selection, deterministic top-k, alternative reward metrics, FFN1 vs FFN2). The public code link and the falsifiable spectral observations (Fig. 1) are strengths, as is the explicit policy-transfer experiment (Table 6). However, the significance is currently undercut by three load-bearing problems: the stated return (Eq. 7) does not credit the action that produced the penalty; the headline numbers are internally inconsistent across Tables 2, 3, 8, and 11; and the advantage over SliceGPT reverses when SliceGPT is given its preferred Alpaca calibration (Table 10). The paper also lacks a direct test linking the KS objective to output or task loss, so the central mechanism, while plausible, is not yet established.

major comments (4)
  1. [Section 4.3, Eq. (7); Algorithm 1 lines 17-19] Eq. (7) defines G_l = sum_{k=0}^inf gamma^k D_{l+k+1}, and Algorithm 1 line 19 repeats this. Because D_l is the KS distance caused by action S_l, the return G_l credits S_l with the penalties of layers l+1,...,L while excluding D_l; consequently D_1 never enters any return, and G_L refers to a nonexistent D_{L+1}. The REINFORCE update (Algorithm 1 line 22) therefore does not optimize the stated objective of minimizing the spectral shift of the layer being pruned, and the last layer's action receives no reward from its own compression. This is load-bearing because the paper's central claim is that the policy learns to minimize the KS distance of Eq. (6). The authors must either correct the return to G_l = sum_{k=0}^{L-l} gamma^k D_{l+k} and confirm that the reported experiments (including the ablations in Tables 16-18) used that objective, or, if the shifted reward was intended, explain it and recompute the attribution of the reported 55.45 average accordingly.
  2. [Table 3 vs Tables 2, 8, and 11] The caption of Table 3 says the rows without RFT are the same as in Table 2, but they are not: for LLaMA-2-7B at 20%, ARC-e is 63.80 in Table 2 and 62.25 in Table 3, and at 30% the average is 55.45 (Table 2) versus 55.33 (Table 3) with ARC-c 33.53 versus 32.94. The RFT rows of Table 3 also disagree with Tables 8 and 11 for the identical WikiText2 condition (at 30%, average 57.08 versus 59.15, ARC-e 53.37 versus 60.14). These inconsistencies change the paper's quantitative conclusions: the 'marginal impact of 1.5%' attributed to RFT in Section 5.2 becomes up to 3.7 points at 30% if Tables 8/11 are correct. The authors should unify the numbers, state which configuration the released code reproduces, and re-derive the RFT claims from the corrected tables.
  3. [Table 2 vs Appendix Table 10; Abstract] The headline comparison against SliceGPT never states which calibration dataset was used for SliceGPT in Table 2, although Section 2 documents that SliceGPT's zero-shot accuracy varies by roughly 6 points with the calibration set. Appendix Table 10 shows that with Alpaca calibration (SliceGPT's default), SliceGPT has a higher average than PruneNet on LLaMA-2-7B at all three ratios (63.68 vs 61.71 at 20%; 60.91 vs 58.63 at 25%; 57.93 vs 55.45 at 30%), and the appendix text concedes this. The abstract's claim that PruneNet 'outperforms existing methods' is therefore contingent on the calibration configuration used in Table 2, and the paper must state that configuration in the main text, report the strong-baseline comparison there, and qualify the superiority claim accordingly.
  4. [Section 4.2 and Section 5.2 / Appendix E.2] The sufficiency of the spectral proxy, that minimizing the KS distance between singular-value ECDFs (Eq. 6) preserves downstream zero-shot performance, is asserted rather than established. Corollary 3.3 only shows that row slicing shrinks the spectrum's range; it does not imply that a smaller KS distance yields smaller output or task error, and no other theorem connects Eq. (6) to the model's function. The random-policy and deterministic-policy ablations (Tables 16-17) show the learned policy is better than these baselines but do not identify the KS objective as the operative cause (Table 18 shows the alternative AD metric performs comparably). A concrete test would be: over many random row subsets, correlate per-matrix KS distance with the induced change in hidden states or logits, and compare PruneNet's policy against a policy trained on output-reconstruction error; if the correlation is weak, the reported advantage cannot be attributed to the stated spectral objective.
minor comments (8)
  1. [Section 4.1 and Algorithm 1] Section 4.1 and Algorithm 1 never specify how LLaMA's gate projection Wgate is pruned, even though Section 3.1 introduces it and Table 9a lists 'Up, Down & Gate Proj' as the compressed modules; the authors should state that Wgate is pruned with the same row indices as Wup.
  2. [Algorithm 1 line 18] Algorithm 1 line 18 assigns F1,n and F2,¯n to the compressed and original matrices, respectively, the reverse of the definitions in Section 4.2; the KS distance is symmetric so the values are unaffected, but the notation should be aligned.
  3. [Section 5.2] Section 5.2 reports a one-sided KS test (p<0.05) for the claim that PruneNet's performance drop is significantly lower than SliceGPT's, but neither the null hypothesis nor the sample over which the test is computed is described; a paired test over the per-task, per-ratio differences would be easier to interpret.
  4. [Appendix C.3 and Algorithm 1 lines 10-11] The reparametrization proof in Appendix C.3 establishes a per-index Bernoulli variable, but Algorithm 1 samples a subset of size (1-r)n from a multinomial; the paper should specify whether the subset is drawn with or without replacement and give the exact log-probability used in the REINFORCE gradient.
  5. [Table 1] Table 1 lacks effective-sparsity entries for both PruneNet rows, and its caption claims PruneNet achieves higher effective sparsity than SliceGPT, which is not true at 30% (SliceGPT 21.4% vs PruneNet 19.0%); the table and caption should be reconciled.
  6. [Section 2] Section 2 says the policy learner 'can be reused to compress different models at different compression ratios,' but the transfer experiments in Table 6 only reuse a policy within the same model across ratios; the claim should be scoped to same-model reuse.
  7. [Tables 2 and 21; Figure 4] Dense-model scores differ slightly between Table 2 and Table 21 (e.g., LLaMA-2-7B WinoGrande 69.06 vs 69.30), and the lower panel of Figure 4 appears garbled with overlapping axis labels; the authors should state whether dense baselines are re-evaluated per table and clean up the figure.
  8. [Section 5.1 and Appendix E.2] The paper does not report the policy learner's own training dynamics or the KS distances achieved after training, so it is unclear whether the 20-episode budget suffices for convergence; a training curve and final per-layer KS distances would also help resolve the attribution issue in Major 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PruneNet is trained on an intrinsic KS-distance objective and evaluated on held-out zero-shot tasks, so the reported accuracies are not an input to the method.

full rationale

The paper's derivation chain is not circular. PruneNet learns row-selection policies via REINFORCE on the Kolmogorov-Smirnov distance between the singular-value ECDFs of original and pruned FFN1 matrices (Eq. 6), and no calibration data or task labels are used in the policy update. The central claim, that this spectral-preservation objective transfers to zero-shot accuracy, is an empirical proxy assumption rather than an equation-level identity: the downstream accuracy numbers in Tables 1-5 come from held-out commonsense and MMLU evaluations, so they are not by construction equal to the KS penalty being optimized. The supporting theory relies on the Poincaré separation theorem, cited to an external textbook (Magnus, 2019) and proved in the appendix, and there are no load-bearing self-citations. The only notable anomaly is the index offset in Eq. (7)/Algorithm 1, where G_l sums D_{l+k+1}, excluding the current layer's D_l and referencing an undefined D_{L+1}; this is an internal implementation inconsistency that affects whether the stated objective is optimized, but it does not make the method's predictions equivalent to its inputs, so it does not constitute circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a learned policy network optimized against an intrinsic spectral objective, plus several hand-chosen hyperparameters and domain assumptions. The ledger is empty of invented entities because PruneNet introduces no new physical entities.

free parameters (3)
  • Policy learner weights (Winter, Wproj) = Trained; exact values only in released repository
    The 45M parameters for LLaMA-2-7B are learned via REINFORCE against the spectral KS objective. All reported compression results depend on these fitted weights, and the paper gives no seed-specific values or convergence analysis.
  • Discount factor gamma = 0.99
    Chosen in Section 5.1 for the future penalty in Eq. 7. The cross-layer weighting of spectral penalties changes with gamma, and no sensitivity study is reported.
  • Maximum training episodes = 20
    Set in Section 5.1 as the training budget for the policy learner. The results depend on this budget; no convergence curves or variance across seeds are shown for policy training.
assumptions (5)
  • standard math Poincaré Separation Theorem (Theorem 3.2)
    Invoked in Corollary 3.3 to claim that slicing rows from a weight matrix contracts the range of its singular values.
  • domain assumption KS distance between singular-value distributions is a sufficient proxy for preserving downstream task performance
    This is the entire reward signal of PruneNet (Section 4.2, Eq. 6). The paper gives no theorem or calibration linking spectral KS distance to zero-shot accuracy.
  • domain assumption Compressing only FFN layers is sufficient for effective LLM compression
    Section 4 states this as a logical assumption based on FFN parameter share and density; self-attention is not pruned in the main results.
  • domain assumption Later layers are more semantically important and have higher singular values
    Section 4.3 and Figure 1 motivate the discounted future penalty, but no task-level evidence is given that later-layer spectral preservation should be weighted more heavily.
  • domain assumption REINFORCE with 20 episodes and a single trajectory yields a good row-selection policy
    No baseline reduction, variance control, or seed analysis is described; the empirical claims rely on this optimization succeeding reliably.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning." pith.science (2026). https://pith.science/paper/WGBE4C7I

@misc{pith2026250115296,
  author       = {Pith},
  title        = {Pith review of: You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGBE4C7I}},
  note         = {Machine review of arXiv:2501.15296}
}
read the original abstract

The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model pruning techniques attempt to alleviate these issues by relying heavily on external calibration datasets to determine which parameters to prune or compress, thus limiting their flexibility and scalability across different compression ratios. Moreover, these methods often cause severe performance degradation, particularly in downstream tasks, when subjected to higher compression rates. In this paper, we propose PruneNet, a novel model compression method that addresses these limitations by reformulating model pruning as a policy learning process. PruneNet decouples the pruning process from the model architecture, eliminating the need for calibration datasets. It learns a stochastic pruning policy to assess parameter importance solely based on intrinsic model properties while preserving the spectral structure to minimize information loss. PruneNet can compress the LLaMA-2-7B model in just 15 minutes, achieving over 80% retention of its zero-shot performance with a 30% compression ratio, outperforming existing methods that retain only 75% performance. Furthermore, on complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80% performance of the original model, proving itself a superior alternative to conventional structured compression techniques.

Figures

Figures reproduced from arXiv: 2501.15296 by the authors.

Figure 1
Figure 1. This observation encourages us to learn a compression model to minimize the distribu [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. A schematic diagram of PruneNet. A policy learner is used to learn the row indices of an FFN1 weight matrix to prune. The policy learner is trained with policy gradient with penalty calculated with the Kolmogorov-Smirnov (KS) distance between the uncompressed and compressed matrix singular value distributions. FFN2 is pruned by columns with the same indices learned by the policy learner. Biases, bup and bdown, are e… view at source ↗
Figure 3
Figure 3. We highlight the performance recov￾ery (w.r.t. the dense uncompressed model) for the LLaMA-2-7B model under different compres￾sion ratios after recovery fine-tuning with varying training data sizes. Does the RFT dataset matter? [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: LLaMA-2-7B RFT training and validation loss [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Spectrum of FFN1 layer for Phi-2 (a) and OPT-2.7B (b) models at different compression [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.

  2. EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    EfficientVLA combines LLM layer pruning, task-aware visual token selection, and diffusion-head feature caching to cut CogACT's inference cost to 28.9% of baseline FLOPs with a 0.6% SIMPLER success drop.

Reference graph

Works this paper leans on

39 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    The falcon series of open language models, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023. URL https://arxiv.org/abs/2311.16867

    arXiv 2023

  2. [2]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL https://arxiv.org/abs/2401.15024

    arXiv 2024

  3. [3]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  6. [6]

    Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. URL https://arxiv.org/abs/2301.00774

    arXiv 2023

  7. [7]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    arXiv 2024

  8. [8]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv....

    arXiv 2023

Show all 39 references
  1. [9]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

    2015 arXiv

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  3. [11]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112, 2022

    2022 arXiv

  4. [12]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  5. [13]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  6. [14]

    Levesque, Ernest Davis, and Leora Morgenstern

    Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR'12, pp.\ 552–561. AAAI Press, 2012. ISBN 9781577355601

    2012

  7. [15]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  8. [16]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023 a

    2023

  9. [17]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023 b

    2023

  10. [18]

    The concrete distribution: A continuous relaxation of discrete random variables

    Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016

    2016 arXiv

  11. [19]

    Matrix Differential Calculus with Applications in Statistics and Econometrics

    J.R Magnus. Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley Series in Probability and Statistics, 2019

    2019

  12. [20]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of E nglish: The P enn T reebank. Computational Linguistics, 19 0 (2): 0 313--330, 1993. URL https://aclanthology.org/J93-2004

    1993

  13. [21]

    Shortgpt: Layers in large language models are more redundant than you expect, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https://arxiv.org/abs/2403.03853

    2024 arXiv

  14. [22]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  15. [23]

    Relu strikes back: Exploiting activation sparsity in large language models, 2023

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URL https://arxiv.org/abs/2310.04564

    2023 arXiv

  16. [24]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  17. [25]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  18. [26]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît S...

    2023 arXiv

  19. [27]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  20. [28]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  21. [29]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  22. [31]

    Svd-llm: Truncation-aware singular value decomposition for large language model compression, 2024 b

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression, 2024 b . URL https://arxiv.org/abs/2403.07378

    2024 arXiv

  23. [32]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8: 0 229--256, 1992. doi:10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696

    1992 doi

  24. [33]

    Laco: Large language model pruning via layer collapse, 2024

    Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse, 2024. URL https://arxiv.org/abs/2402.11187

    2024 arXiv

  25. [34]

    Asvd: Activation-aware singular value decomposition for compressing large language models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023

    2023 arXiv

  26. [35]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  27. [36]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022 arXiv

  28. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  29. [38]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  30. [39]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  31. [40]

    We report the results with LLaMA-2-7B with RFT on Wikitext2 and Alpaca datasets in Table wikitext_with_rft and Table alpaca_with_rft , respectively

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.