Pith. sign in

REVIEW 4 major objections 5 minor 47 references

This paper claims that one uniform error tolerance per model, applied to per-layer SVD truncation, yields Pareto-optimal heterogeneous ranks, making LLM/VLM rank allocation a one-knob problem and supporting a zero-shot pipeline that beats u

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:20 UTC pith:WJZYQB3G

load-bearing objection The empirical recipe—uniform per-layer error tolerance with ALS refinement—is a real, useful contribution, but the proof of Theorem 2 overclaims: uniform tolerance is only minimax-optimal for a symmetric worst-case surrogate, not Pareto-optimal for the actual heterogeneous layer profiles, and a two-layer counterexample shows the claim fails as stated. the 4 major comments →

arxiv 2510.05544 v2 pith:WJZYQB3G submitted 2025-10-07 cs.CL cs.LG

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

Ryan Solgi , Parsa Madinei , Jiayi Tian , Rupak Swaminathan , Jing Liu , Nathan Susanj , Zheng Zhang This is my paper
classification cs.CL cs.LG
keywords low-rank compressionsingular value decompositionPareto optimalityactivation-aware compressionzero-shot compressionlarge language modelsvision-language modelsalternating least squares
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that when each weight layer of a language or vision-language model is compressed by truncating its singular value decomposition to the smallest rank whose relative error is below a common tolerance ε, the resulting per-layer ranks sit on a Pareto-optimal trade-off between parameter savings and network-loss change. The proof chain has two load-bearing pieces: a first-order bound showing how layer-wise activation distortion propagates to the final loss, and an equivalence between rank allocation and error-tolerance allocation that makes the uniform-ε choice optimal in a surrogate sense. The practical payoff is PGSVD, a zero-shot pipeline that picks ranks with this one-knob rule and then refines factors with alternating least squares. If the claim is right, adaptive per-layer rank search can be replaced by a single calibrated scalar per model, with better perplexity and reasoning accuracy at matched compression than uniform-ratio compression.

Core claim

The central claim is Theorem 2: under the paper's assumptions, every uniform tolerance ε corresponds to a point on the surrogate Pareto frontier of the bi-objective problem that trades total parameters against loss change. Concretely, for each layer one computes the minimal rank whose truncated-SVD relative error is at most ε; because layer spectra differ, the same ε induces different ranks per layer, and this allocation is claimed to be optimal for the parameter budget it realizes. The paper also proves a loss-sensitivity bound (Theorem 1) showing that activation-aware per-layer error, weighted by the Jacobian norms of downstream layers, upper-bounds the absolute change in network loss — wh

What carries the argument

The load-bearing object is the ε–parameter mapping h_l(ε), which assigns to each tolerance the minimal parameter count of a truncated SVD whose relative Frobenius error is ≤ ε. Lemma 1 bounds all layer profiles between shared nonincreasing convex envelopes and assumes homogeneous per-layer sensitivity, reducing the robust allocation problem to a convex ε-allocation problem whose optimality conditions force a uniform ε. Theorem 2 then imports that uniform ε as a surrogate Pareto point for the original bi-objective rank-allocation problem. The algorithmic workhorse is an alternating least-squares solver that updates the low-rank factors A and B against the empirical activation covariance matri

Load-bearing premise

The Pareto-optimality of a single uniform tolerance assumes all layers share one convex envelope for their SVD error profiles and equal sensitivity to perturbation, so if real layers deviate from that envelope — as the paper's own exclusion of outlier layers suggests — a non-uniform allocation can beat the uniform one at the same parameter budget.

What would settle it

Take two layers with known, very different singular-value decay (e.g., one fast exponential, one slow linear), both within the common convex envelope. Choose ε so the shared-tolerance rank allocation meets a target total parameter budget, then minimize total surrogate error over all per-layer ε_l subject to that budget. If the constrained optimum uses ε_1 ≠ ε_2 and strictly improves on the shared-ε point, then Theorem 2 as stated fails for that instance.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If uniform ε is surrogate-optimal, rank allocation needs no search: calibrating one tolerance per model produces a heterogeneous rank vector that is Pareto-optimal in the surrogate problem.
  • Because the same ε yields different ranks across layers, compression automatically spares hard-to-compress layers and squeezes easy ones.
  • For vision-language models, separate tolerances for the vision and text towers extend the same theory to cross-modality imbalance using only two knobs.
  • The loss-bound result implies that activation fidelity, not raw weight fidelity, is the right per-layer objective, explaining performance gains over plain weight-SVD compression.
  • The ALS refinement makes implementing the chosen ranks fast and numerically stable, avoiding the Cholesky failures reported for prior activation-aware solvers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the Pareto claim is testable against real layer profiles — the paper's own convex-envelope evidence excludes about 2% of layers, and a direct check is whether hand-tuned non-uniform ε beats uniform ε at equal parameter count on those layers.
  • Beyond the paper: the single-knob principle should transfer to any compression module with a monotone ε-parameter map, such as quantization step size or pruning threshold, giving a family of one-knob Pareto-optimal compressors.
  • Beyond the paper: the framework permits clustered uniform tolerances (e.g., one ε for attention, one for MLP) without leaving the theory, a degree of freedom the paper notes but does not exploit; calibrating those clusters from data could close much of the gap between the surrogate and the true Pareto frontier.
  • Beyond the paper: an ablation that replaces only the rank-selection step while keeping the solver fixed would clarify whether the gains come from the Pareto-guided ranks or from the activation-aware ALS refinement — a natural way to test where PGSVD's advantage lives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PGSVD, a zero-shot low-rank compression method for LLMs and VLMs. The theoretical component (Sec. 3) derives a first-order bound connecting activation-based layer compression error to network loss (Theorem 1), formulates layer-wise rank allocation as a bi-objective problem (B), and claims that a single uniform error tolerance ε yields surrogate Pareto-optimal heterogeneous layer ranks (Theorem 2). The algorithm first selects ranks via uniform ε, then refines low-rank factors with an alternating least-squares (ALS) solver (Eq. 6). Experiments compare PGSVD with SVD-LLM, a uniform-ratio variant SVD-ALS, and pruning baselines on LLaMA-2-7B/13B, Mistral-7B, and CLIP, reporting perplexity and zero-shot accuracy gains, with an honest acknowledgment that SliceGPT outperforms PGSVD at 50% compression on WikiText-2.

Significance. If the headline Pareto-optimality claim were valid, the paper would make a useful conceptual contribution: reducing the layer-wise rank search for zero-shot LLM/VLM compression to one (or two) calibrated scalar tolerances, with a theoretical justification. The empirical work is also valuable in a more modest way: the SVD-ALS control isolates the effect of heterogeneous rank allocation from the ALS solver, and the consistent PPL/accuracy improvements over uniform-ratio compression are credible. The paper deserves credit for clearly naming its robust/minimax assumptions (Remark 1, Appendix D), for including a limitations section, and for reporting the SliceGPT comparison where the proposed method loses. However, the central theoretical guarantee is not established as stated; the proof of Theorem 2 relies on an unproven transfer from a symmetric worst-case surrogate to the actual heterogeneous layer profiles.

major comments (4)
  1. [Appendix E (Proof of Theorem 2; Lemma 1, Appendix C)] The central inference is invalid. Lemma 1 establishes optimality of the uniform allocation only for the symmetric surrogate using a common envelope h̄ (and a minimax statement for hl≡h̄). It does not establish Pareto optimality for the actual heterogeneous profiles hl. The sentence in Appendix E that 'the optimal solution of this surrogate for any budget b is attained by a uniform tolerance' is an unproven import step. In fact, the claim is false even inside the lemma's assumptions: take L=2, α=1, h1(ε)=0.6(1−ε)^2, h2(ε)=0.9(1−ε)^2, with envelopes h(ε)=0.25(1−ε)^2 and h̄(ε)=(1−ε)^2. Uniform ε=0.5 gives budget 0.375 and objective 1.0, while (ε1,ε2)=(0.388,0.592) has budget ≈0.3745 and objective 0.980. Thus uniform ε is not Pareto-optimal even for the surrogate unless all hl coincide. The abstract and Section 3 therefore overclaim; the defensible statement is that uniform ε is a minimax-ro
  2. [Lemma 1 and Appendix G (convex-envelope premise)] Lemma 1 requires every layer's parameter function hl to lie between common nonincreasing convex envelopes h and h̄. Appendix G supports this only for 'the remaining 98% of layers' after excluding outliers. Outlier layers are exactly the ones where heterogeneous rank allocation matters most, so the premise is not established for the real networks evaluated. Without a universal bound, Theorem 2's Pareto conclusion does not apply to the models tested. This is not just a technicality: the proof of Theorem 2 and the abstract's 'prove' language depend on this premise.
  3. [Proposition 1 / Formulation 3 (discrete-to-continuous equivalence)] The paper treats the ε-allocation problem (E) as a continuous convex problem, but the actual hl are right-continuous step functions defined by minimal SVD ranks, so (E) is not convex in the original variables. Lemma 1's KKT argument applies to the convex envelope h̄, not to the true step-function profiles. There is no explicit bound on the gap between the envelope-relaxed solution and the original discrete rank allocation. This weakens the claimed 'same optimal value' equivalence in Proposition 1 and the closed-form optimality statement, independent of the heterogeneity issue raised above.
  4. [Theorem 1 / Appendix A] Theorem 1 is presented as an upper bound on |ΔL|, but the proof drops second-order terms in the first-order expansion. The displayed inequality is therefore an approximate first-order bound, not a rigorous upper bound unless the remainder is controlled. The text in §3.2 says it 'upper bounds' the loss change; this should be qualified, and a remainder term or a formal assumption on the perturbation size should be supplied.
minor comments (5)
  1. [Abstract and §1] The wording 'prove that a single uniform tolerance yields surrogate Pareto-optimal heterogeneous ranks' is too strong given the proof gap. Suggest rephrasing to 'is minimax-optimal for a symmetric surrogate and provides a robust default allocation' unless the theorem is replaced by a valid statement.
  2. [Fig. 3] The legend appears to repeat 'PGSVD (20%)' for the right panel; one of the entries should presumably be 'PGSVD (40%)'.
  3. [§5.1] The claim 'gains of up to 30% and an average improvement of 14%' is not tied to a specific table or metric. Please clarify whether the percentage is relative accuracy improvement, relative PPL reduction, or some other measure; the current phrasing is ambiguous.
  4. [References] The SVD-LLM citations (Wang et al., 2024; 2025) are used but the reference list includes 'SVD-LLM V2' under Wang et al. 2025; please ensure the in-text references match the bibliographic entries and include page/venue details for all entries.
  5. [General notation] The symbol h is used both for the lower envelope in Lemma 1 and for the ε–parameter mapping in Definition 1. Please disambiguate (e.g., h̲/h̄ for envelopes), since this makes the statements in Appendix G and Lemma 1 harder to follow.

Circularity Check

2 steps flagged

Uniform-ε Pareto-optimality is manufactured by a symmetrized surrogate and then re-imported as a theorem about the actual per-layer profiles.

specific steps
  1. other [Lemma 1 / Appendix C (with Remark 1, Appendix D)]
    "Consider the ε-allocation problem (E) for a homogeneous network (where α_l ≡ α, ∀l) ... Assume each layer’s parameter function h_l is bounded by common nonincreasing convex envelopes h, h̄ ... Then the ε-allocation problem admits a uniform solution ε1 = · · · = εL that (i) is optimal for the symmetric surrogate using h ..."

    The uniform solution is not derived from the actual per-layer profiles; it is produced by construction, replacing every α_l with the same α and every h_l with a common symmetric envelope h. The KKT argument then mechanically returns equal ε_l. That is a minimax/worst-case statement about a symmetrized surrogate, not an optimality statement for the actual heterogeneous h_l used in Formulation (B). The later claim that uniform ε is a Pareto point therefore inherits its content from the symmetrizing assumption rather than from a derivation about the true profiles.

  2. other [Theorem 2 and Appendix E]
    "Under the homogeneous sensitivity and bounded-profile assumption (Theorem 1), the optimal solution of this surrogate for any budget b is attained by a uniform tolerance across layers."

    Appendix E uses this sentence as the bridge from Lemma 1 to Theorem 2, but Lemma 1 only proved optimality for the symmetric h-surrogate and minimax optimality at the worst-case profile hl ≡ h̄. It never proved that a uniform allocation is optimal for the actual heterogeneous functions hl. The proof therefore assumes, in effect, the Pareto-optimality conclusion it is meant to establish. This is not a mere missing detail: within Lemma 1's own assumptions, taking h1(ε)=0.6(1−ε)^2, h2(ε)=0.9(1−ε)^2 with common convex bounds yields a nonuniform allocation that strictly dominates the uniform allocation at the same parameter budget.

full rationale

The empirical and algorithmic components are largely self-contained: Theorem 1 gives a genuine first-order loss bound, Proposition 1 is a real equivalence between rank and ε formulations, the ALS updates are derived from the stated objective, and the benchmark comparisons are external and reproducible. There is no fitted parameter renamed as a prediction and no load-bearing self-citation. The circularity is concentrated in the theoretical headline: Lemma 1 proves uniform ε is optimal only for a symmetrized worst-case surrogate, and Theorem 2's proof re-imports that statement as Pareto-optimality for the actual heterogeneous per-layer problem. Appendix G's convex-envelope evidence is post hoc and explicitly excludes outliers ('after excluding a small fraction of outlier layers'), so it does not supply an independent derivation of the uniform-tolerance Pareto guarantee. The score reflects this construction-level circularity in the central theorem, not data fitting or self-citation; the PGSVD recipe may remain viable as a minimax-safe default, but the claimed Pareto-optimality guarantee is not independently established.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claim rests on two load-bearing structural premises: homogeneous sensitivity α_l ≡ α (Remark 1, App. D) and common convex envelopes h ≤ h_l ≤ h̄ (Lemma 1, App. G, empirically claimed only for 98% of layers after excluding outliers). The first-order linearization in Theorem 1's proof is also dropped from the stated bound. The algorithm itself has one calibration knob per model (ε) and two per VLM (ε_v, ε_t), values undisclosed; α_l is defined but never estimated or used. No invented physical/formal entities.

free parameters (3)
  • ε — uniform per-layer error tolerance (LLM) = not reported
    Algorithm 1 chooses each layer's rank as the minimum rank with relative SVD error e_l(r) ≤ ε; ε is the single knob. To report 20%/40% parameter reduction, ε must be calibrated per model; the chosen values and calibration protocol are not disclosed (§4, §5.1).
  • ε_v, ε_t — separate tolerances for CLIP vision/text towers = not reported
    §4/§5.2 assign different tolerances to image and text towers; the values are not given, so the reported 20%/40% CLIP compression cannot be reproduced exactly.
  • τ — ALS iteration count = 10 (all experiments); 1 is 'acceptable' (Fig. 2)
    Chosen from the perplexity-plateau curve in Fig. 2; a tuning choice adopted for all runs (§5).
axioms (6)
  • domain assumption First-order Taylor expansion: higher-order terms in ΔW are dropped, so the bound is on the linearized loss change (Appendix A: 'dropping second-order terms')
    Theorem 1 is stated as |ΔL| ≤ G Σ ... c||ΔW_l X_l||_F but the proof bounds only the first-order term; for 20–40% compression ΔW is not infinitesimal, so the inequality is not established for the true loss.
  • ad hoc to paper Homogeneous loss sensitivity α_l ≡ α across layers (Remark 1, App. D)
    Adopted so the ε-allocation problem becomes symmetric and uniform ε becomes optimal; actual α_l = ||∇Y L||_F (Π_{m=l+1}^L K_m) c ||X_l||_F ||W_l||_F is depth- and data-dependent. The authors' own CLIP experiments use ε_v ≠ ε_t, conceding sensitivity differs between towers.
  • domain assumption All SVD profiles h_l are pointwise bounded by common convex nonincreasing envelopes h ≤ h_l ≤ h̄ (Lemma 1, App. G)
    Required for Lemma 1's minimax result; Fig. 4 supports it only after excluding 'a small fraction of outlier layers' (98% band) and for the weight-space spectra of LLaMA-2, not for arbitrary models.
  • domain assumption Activation covariance M = XX^T from a small sample approximates the true second moment; activations X display low-rank structure (Sec. 2, Eq. (2); Sec. 4)
    The ALS updates and the objective ||WX − ABX||_F use an empirical covariance; the sample (dataset, length, batch size) is not specified, and the low-rank-property claim is cited from Zhang et al. 2024 rather than demonstrated here.
  • standard math Elementwise-activation Lipschitz bound |σ'| ≤ c < ∞ (Theorem 1)
    Holds for bounded-slope activations (ReLU/GELU in subgradient sense); standard.
  • standard math Eckart–Young–Mirsky: truncated SVD is the minimum-Frobenius-error rank-r approximation, giving the h_l profiles (Definition 1, proof of Theorem 2)
    Standard linear algebra the paper relies on to define r*(ε) and P(r).

pith-pipeline@v1.3.0-alltime-deepseek · 15569 in / 33679 out tokens · 225340 ms · 2026-08-04T11:20:14.547562+00:00 · methodology

0 comments
read the original abstract

Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment. We present a novel low-rank compression framework to address this challenge. First, we upper bound the change of network loss via layer-wise activation-based compression errors, filling a theoretical gap in the literature. We then formulate low-rank model compression as a bi-objective optimization and prove that a single uniform tolerance yields surrogate Pareto-optimal heterogeneous ranks. Based on our theoretical insights, we propose Pareto-Guided Singular Value Decomposition (PGSVD), a zero-shot pipeline that improves activation-aware compression via Pareto-guided rank selection and alternating least-squares implementation. We apply PGSVD to both LLM and VLM, showing better accuracy at the same compression levels and inference speedup.

Figures

Figures reproduced from arXiv: 2510.05544 by Jiayi Tian, Jing Liu, Nathan Susanj, Parsa Madinei, Rupak Swaminathan, Ryan Solgi, Zheng Zhang.

Figure 1
Figure 1. Figure 1: Overview of PGSVD: (left) unimodal model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Compression times of different solvers for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Inference throughput of LLaMA-2-7b (left) and Mistral 7b (right) for 20% and 40% compression using [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: SVD profiles for LLaMA-2 7B (left) and 13B (right). [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 17 linked inside Pith

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mistral AI. 2023. Mistral 7b: A 7-billion parameter language model. arXiv preprint arXiv:2310.06825

    Pith/arXiv arXiv 2023

  4. [4]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://arxiv.org/abs/2401.15024 Slicegpt: Compress large language models by deleting rows and columns . In International Conference on Learning Representations (ICLR)

    Pith/arXiv arXiv 2024

  5. [5]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of AAAI

    2020

  6. [6]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, and et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    Pith/arXiv arXiv 2021

  7. [7]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 -- mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), pages 446--461

    2014

  8. [8]

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2014

  9. [9]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT

    2019

  10. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  11. [11]

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2023. Spqr: A sparse-quantized representation for efficient generative inference. arXiv preprint arXiv:2306.03078

    Pith/arXiv arXiv 2023

  12. [12]

    Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPR Workshop on Generative-Model Based Vision

    2004

  13. [13]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774

    Pith/arXiv arXiv 2023

  14. [14]

    Elias Frantar, Eldar Kurtic, and Dan Alistarh. 2022. Optimal brain compression: A framework for practical pruning of pretrained transformers. arXiv preprint arXiv:2208.11580

    Pith/arXiv arXiv 2022

  15. [15]

    Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. Adaptive rank selections for low-rank approximation of language models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics

    2024

  16. [16]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, volume 12, pages 2217--2226

    2019

  17. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of ICLR

    2021

  18. [18]

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022 a . Language model compression with weighted low-rank factorization. International Conference on Learning Representations (ICLR)

    2022

  19. [19]

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022 b . Language model compression with weighted low-rank factorization. In International Conference on Learning Representations (ICLR)

    2022

  20. [20]

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops (ICCVW)

    2013

  21. [21]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE : Large-scale reading comprehension dataset from examinations. In Proceedings of EMNLP

    2017

  22. [22]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. International Conference on Learning Representations (ICLR)

    2020

  23. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. https://proceedings.mlr.press/v202/li23q.html Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research

    2023

  24. [24]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. https://arxiv.org/abs/2305.11627 Llm-pruner: On the structural pruning of large language models . In Advances in Neural Information Processing Systems (NeurIPS)

    Pith/arXiv arXiv 2023

  25. [25]

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853

    Pith/arXiv arXiv 2024

  26. [26]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    Pith/arXiv arXiv 2016

  27. [27]

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of ACL

    2016

  28. [28]

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3498--3505

    2012

  29. [29]

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Luis Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350

    Pith/arXiv arXiv 2021

  30. [30]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748--8763. PMLR

    2021

  31. [31]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of AAAI

    2020

  32. [32]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: Smaller, faster, cheaper, and lighter. arXiv:1910.01108

    Pith/arXiv arXiv 2019

  33. [33]

    Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems (NeurIPS)

    2020

  34. [34]

    Mahoney, and Kurt Keutzer

    Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. AAAI Conference on Artificial Intelligence

    2020

  35. [35]

    Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. 2023. https://proceedings.mlr.press/v202/shi23e/shi23e.pdf Upop: Unified and progressive pruning for compressing vision-language transformers . In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research

    2023

  36. [36]

    Masao Someki, Yifan Peng, Siddhant Arora, Markus M \"u ller, Athanasios Mouchtaris, Grant Strimel, Jing Liu, and Shinji Watanabe. 2025. https://openreview.net/forum?id=u2QdCiOgwA Context-aware dynamic pruning for speech foundation models . In The Thirteenth International Conference on Learning Representations

    2025

  37. [37]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)

    2019

  38. [38]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT

    2019

  39. [39]

    Hugo Touvron, Matthieu Lavril, Gautier Izacard, Xavier Martinet, Marie‐Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Aurélien Azhar, Justin Rodriguez, Armand Joulin, and Edouard Grave. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    Pith/arXiv arXiv 2023

  40. [40]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS)

    2017

  41. [41]

    Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. 2025. https://doi.org/10.18653/v1/2025.naacl-long.217 SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies...

    doi:10.18653/v1/2025.naacl-long.217 2025

  42. [42]

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024. https://arxiv.org/abs/2403.07378 Svd-llm: Truncation-aware singular value decomposition for large language model compression . arXiv preprint arXiv:2403.07378

    Pith/arXiv arXiv 2024

  43. [43]

    Shuohang Yang, Xuguang Meng, Haoran Li, Jiahui Wang, Hongxia Tang, Shuo Lin, Xinyang Chen, Zheng Zhang, Weijia Liu, and Jingdong Wang. 2024. https://arxiv.org/abs/2404.12345 Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models . In Proceedings of the 41st International Conference on Machine Learning (ICML)

    Pith/arXiv arXiv 2024

  44. [44]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. https://arxiv.org/abs/2304.14178 mplug-owl: Modularization empowers large language models with multimodality . arXiv preprint arXiv:...

    Pith/arXiv arXiv 2023

  45. [45]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. https://arxiv.org/abs/2312.05821 Asvd: Activation-aware singular value decomposition for compressing large language models . arXiv preprint arXiv:2312.05821

    Pith/arXiv arXiv 2023

  46. [46]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL

    2019

  47. [47]

    Zhiyuan Zhang, Xuefei Ning Sun, Song Han, Jie Tang, and Bolin Ding. 2024. https://ojs.aaai.org/index.php/AAAI/article/view/26304 Compressing transformers: Features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13714--13722

    2024