Pith. sign in

REVIEW 4 major objections 5 minor 64 references

Compressing an LLM along both parameter and token dimensions preserves more capability under the same total sparsity than either mechanism alone, though a cross-dimensional sparsity wall remains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:39 UTC pith:FAUMVMJ3

load-bearing objection Good question, plausible answer, but the evidence is thin and the sparsity metric is a convention — needs revision before the strong claims can be trusted. the 4 major comments →

arxiv 2607.18280 v1 pith:FAUMVMJ3 submitted 2026-06-29 cs.LG cs.AI

Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

Chao Han , Haozhe Hu , Xiaoyu Shen This is my paper
classification cs.LG cs.AI
keywords LLM compressionparameter pruningtoken skippingcompound sparsityessential sparsity boundarycross-dimensional sparsitylow-rank approximationdynamic routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a large language model can be compressed more gracefully by splitting the compression burden between two independent sources of redundancy: static parameter pruning and dynamic per-token layer skipping. It builds a controllable pipeline that first applies low-rank and channel pruning to the weights, then adds lightweight routers that skip full computation for some tokens. Across understanding and modeling benchmarks, the compound approach consistently beats either single mechanism at the same nominal sparsity—delaying the sharp performance collapse and retaining higher relative accuracy. The catch: no allocation escapes a fundamental cross-dimensional sparsity boundary, suggesting a universal limit on how much an LLM can be squeezed regardless of how the burden is distributed.

Core claim

The central discovery is that sparsity applied in two orthogonal dimensions—parameter space and token-computation space—interacts productively: for a fixed global sparsity Scomp = 1 − (1−sp)(1−st), distributing the budget across both dimensions yields better downstream performance than concentrating it in one. Concretely, the paper reports that compound sparsity delays the sharp decay point on understanding tasks from roughly 20% to 30% sparsity, and at 30% sparsity it retains about 70% relative performance versus about 60% for token skipping alone and 56% for parameter pruning alone. The paper also finds a cross-dimensional interference effect—a lightly pruned backbone is more sensitive to

What carries the argument

The load-bearing object is the compound sparsity formula Scomp = 1 − (1−sp)(1−st), which converts a pair of per-dimension sparsities into a single scalar 'total sparsity.' That scalar lets the paper compare single-mechanism and compound compression on a common scale, and it underpins every claim about delaying the decay point or optimal budget allocation. The pipeline itself—static low-rank/channel pruning followed by per-token routing that skips layer computation—realizes the two dimensions independently, so each can be set without affecting the other.

Load-bearing premise

The load-bearing assumption is that a unit of parameter sparsity and a unit of token sparsity reduce effective inference computation to the same degree, so that comparing 'same total sparsity' across allocations is a fair operational comparison; if they do not, the claimed delay of the decay point could vanish under a real-cost metric.

What would settle it

Measure wall-clock latency and energy for models matched at the same Scomp but different (sp, st); if the balanced allocation is not the cheapest at the same reported sparsity, or if a single-mechanism model at the same real cost outperforms the compound one, the 'same sparsity' equivalence is broken.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If compound sparsity holds, practitioners can push effective compression ratios higher before quality collapses, simply by allocating the budget across parameters and tokens.
  • The near-balanced allocation result gives a simple rule: for any fixed sparsity budget, spread the reduction roughly evenly across the two dimensions to minimize per-dimension stress.
  • The observed cross-dimensional interference implies that pruning in one dimension changes how sensitive the model is to the other; this should be factored into loss functions or adaptive schedules.
  • The existence of a universal sparsity wall means that no combination of these two mechanisms can push past the fundamental limit, so further gains require better training or different architectures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'same total sparsity' comparisons assume one unit of parameter sparsity is as costly as one unit of token sparsity; if they reduce real inference cost differently (e.g., token skipping adds router overhead), the optimal allocation under a true cost budget might shift. This is not validated in the paper.
  • The experiments cover a single model family and scale; the claimed universality of the cross-dimensional sparsity wall is an extrapolation that could be tested on smaller and larger models.
  • If the benefit of compound sparsity stems from tapping heterogeneous redundancy, then combining other orthogonal mechanisms—such as quantization or distillation—might postpone decay further, though the paper's own 'one wall' principle predicts diminishing returns.
  • The balanced-allocation optimum could be formalized as a min-max principle: under a fixed compound sparsity, minimize the maximum per-dimension sparsity; this is a testable optimization objective beyond the paper's empirical snapshots.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a 'compound sparsity' framework for LLMs that combines static parameter pruning (LoRAP plus channel pruning) with dynamic per-token layer skipping (SkipGPT). It defines a global sparsity metric Scomp = 1 − (1−sp)(1−st) and reports experiments on Llama-3.1-8B across understanding and modeling benchmarks. The central claim is that, at the same total sparsity, the compound method outperforms either single mechanism, delaying the sharp performance-decay point from about 20% to about 30% sparsity on understanding tasks while retaining higher modeling performance. A further claim is that under a fixed 50% total sparsity budget, near-balanced allocation is optimal, and that cross-dimensional interference between the two mechanisms reveals a universal 'cross-dimensional sparsity wall.'

Significance. If the headline results are correct, the paper makes a useful practical and conceptual contribution: it shows that distributing compression across two orthogonal redundancy dimensions can improve the compression-performance trade-off, and that a single mechanism is not the only way to approach the sparsity boundary. The framework has the merit of combining two established and publicly validated methods, which reduces the risk that bespoke designs drive the findings. However, the empirical claims as presented are not fully verifiable: Figure 2 is an unreadable encoding artifact in the provided text, the 20%→30% delay and 70/60/56% figures appear only in prose, and the only quantitative supporting table (Table 2) covers only the 50% total-sparsity setting. Moreover, the central comparison depends on Eq. (4), which treats parameter sparsity and token sparsity as interchangeable units of computation without FLOP, latency, or energy validation. These issues are load-bearing for the paper's main claims.

major comments (4)
  1. [§2.2, Eq. (4)] The global sparsity metric Scomp = 1 − (1−sp)(1−st) treats a unit of parameter sparsity as equivalent to a unit of token sparsity in reducing 'effective inference computation.' But LoRAP/channel pruning and SkipGPT token skipping do not reduce inference cost commensurately: parameter pruning changes the FLOP profile non-uniformly across attention and MLP sublayers and does not remove embeddings, layer norms, residuals, or the router overhead; token skipping leaves the model footprint unchanged and adds router cost. All 'same total sparsity' comparisons, the claimed delay of the decay point, and the balanced-allocation optimum are defined on this algebraic scale. The paper provides no FLOP, latency, or energy measurements to validate the equivalence. Please either validate the metric with measured compute or explicitly reframe the claims as nominal sparsity comparisons and add an ablation
  2. [§3.2, Figure 2] Figure 2 is not interpretable in the manuscript text: instead of readable curves, the caption area contains an unicode-escape-like token stream ('/uni00000013/uni00000014/...'). The quantitative conclusions in the same section — the delay of the essential-sparsity collapse from 20% to 30%, and the 70%/60%/56% relative performance numbers at 30% sparsity — appear only in prose and are not backed by a tabular listing. The reader cannot verify the core empirical claim. Please provide a properly rendered figure and, more importantly, full per-configuration tables (sp, st, accuracy, PPL) across the entire sparsity range.
  3. [§3.3, cross-dimensional interference] The paragraph on cross-dimensional interference states that 'when token sparsity reaches 60%, the compound sparse model only maintains 83% baseline performance' after parameter sparsity of 10%. No experiment with token sparsity = 60% appears in Table 1 or Table 2, and no other data table is provided for this setting. This claim is central to the 'cross-dimensional interference' and 'sparsity wall' narrative, so it needs direct supporting data — either a full table of the interference study or an explicit reference to a figure that is actually readable.
  4. [Table 2] Table 2 provides the only quantitative support for the balanced-allocation conclusion, but its presentation is unclear. The row label 'Acc.' does not specify whether the numbers are absolute accuracy, relative performance, or an average across the understanding tasks. The value at sp=30, st=28.6 (74.79) appears higher than the uncompressed baseline accuracy of 72.81 reported in Table 1 at 0% sparsity, which would imply a compression-induced improvement; this is never explained. Please clarify the metric, add task-level breakdowns, report standard deviations/seeds, and ensure the baseline is the correct comparison point.
minor comments (5)
  1. [Abstract] Typo: 'askswhether' should be 'asks whether.'
  2. [§4/Conclusion] The phrase 'Two Dimensions, One Wallprinciple' is missing a space; it should read 'Two Dimensions, One Wall principle.'
  3. [§3.1/Experimental setup] The experimental setup does not specify several details needed for reproducibility: which RedPajama subset was used, the number of training steps/tokens for the router post-training, the SkipGPT router architecture and training objective, and the exact LoRAP hyperparameters (e.g., rank and channel-pruning ratio). Please add these details or cite the exact configurations.
  4. [References] Several references are formatted inconsistently: 'JAISWAL et al.' in all caps, and 'and 1 others' appears in multiple author lists. Please clean up the reference list.
  5. [§3.3, Table 1] Table 1 is hard to interpret: the row labels 'Original' and 'Compressed' under the 'Base Sparsity (%)' heading are not self-explanatory, and the values in the 'Original' row decrease with increasing sparsity, which is confusing. Please rewrite the table with clear columns for the compression method, sparsity level, and performance.

Circularity Check

0 steps flagged

No significant circularity: the compound-sparsity comparison is an empirical study, and the metric convention in Eq. (4) is an assumption about compute equivalence, not a derivation that reduces the claims to their inputs.

full rationale

The central claim — compound sparsity outperforms single-mechanism compression at the same total sparsity — is established by benchmark measurements (Figure 2, Tables 1-2) on a controlled grid where Scomp = 1 - (1-sp)(1-st) is held at approximately 50%. The balanced-allocation result is derived as the minimax solution of that product formula but is then independently verified empirically (Table 2), so it does not reduce to the formula by construction. The citations to SkipGPT, LoRAP, and same-group works (Zhao et al. 2025; Ding et al. 2026; Han et al. 2025/2026; Wu et al. 2026; He et al. 2026) supply components and motivation rather than a load-bearing uniqueness theorem; no fitted parameter is renamed as a prediction. The only substantive caveat is that Eq. (4) treats parameter and token sparsity as interchangeable reductions of inference compute without FLOP/latency validation; that is a measurement-validity concern, not a circularity, because the empirical comparisons could in principle fail under a corrected cost metric. No circular step is therefore identified.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No invented entities are introduced; 'compound sparsity' and the 'cross-dimensional sparsity wall' are labels for empirical regularities, not independently falsifiable postulates. The main ledger items are the hand-set decay threshold, two unstated normalizations, the cost-equivalence convention of Eq. (4), and the imported essential-sparsity premise.

free parameters (3)
  • relative-performance decay threshold = 90% relative performance
    Hand-set in §3.2 to define the 'sharp decay point' used for the central 20%→30% delay claim; no sensitivity analysis over the threshold value is reported.
  • PPL relative-performance transform = unstated
    The modeling result '~70% relative performance at 30% sparsity' must be some function of WikiText-2 PPL (e.g., inverse-PPL ratio); the transform is never defined, so the number is not reproducible and is effectively a convention.
  • understanding-task averaging rule = unweighted average over 8 tasks (implied)
    Headline accuracy numbers are reported only as averaged relative performance; per-task numbers, weighting, and variance are omitted, so the aggregation is a hidden choice.
axioms (3)
  • domain assumption Scomp = 1 − (1−sp)(1−st) is a valid common scale equating parameter and token sparsity
    §2.2 Eq. (4): all 'same total sparsity' comparisons, the decay-delay claim, and the balanced-allocation optimum are defined on this multiplicative convention; it ignores router overhead and the memory-vs-FLOP asymmetry of the two mechanisms.
  • domain assumption Essential sparsity boundary exists (imported from JAISWAL et al. 2023 and same-group Ding et al. 2026)
    §1: the motivating premise that compression degrades sharply past a boundary is taken from cited work, including a same-group preprint (Ding et al. 2026, arXiv:2601.18091); the concluding 'universal wall' inherits this premise.
  • domain assumption LoRAP and SkipGPT transfer to the compound pipeline without loss of their published properties
    §2.1: the framework composes two off-the-shelf methods and assumes their retraining behavior is unchanged on a pruned backbone; no stage-isolation ablation is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 86 in / 23477 out tokens · 211603 ms · 2026-08-02T09:39:17.367631+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation, yet aggressive sparsification can trigger rapid performance degradation beyond an essential sparsity boundary. This work asks \emph{whether combining these two mechanisms can delay such degradation by distributing the compression burden}. We study a minimalist compound sparsity framework that first applies low-rank approximation and channel pruning to obtain a statically compressed backbone, and then introduces lightweight routers for per-token dynamic layer skipping. This design enables independent control of parameter sparsity and token-level computation sparsity. Experiments across language understanding and modeling benchmarks show that compound sparsity consistently outperforms single-mechanism compression under the same total sparsity, delaying the decay point on understanding tasks and preserving stronger modeling performance. Further analysis reveals cross-dimensional interference between parameter pruning and token skipping, and shows that near-balanced allocation is most effective under a fixed sparsity budget. These results demonstrate that compound compression provides a practical way to improve LLM compression, while revealing a broader cross-dimensional sparsity boundary that ultimately limits further compression. Code will be available at https://github.com/EIT-NLP/LLM-Pruning.

Figures

Figures reproduced from arXiv: 2607.18280 by Chao Han, Haozhe Hu, Xiaoyu Shen.

Figure 1
Figure 1. Figure 1: Overview of LLMs compression paradigms, compound sparse distributes sparsity burdens into orthogonal [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sparsity-performance landscape of different [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 9 linked inside Pith

  1. [1]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Explore what llm does not know in complex question answering , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  2. [2]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  3. [3]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  4. [4]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  5. [5]

    arXiv preprint arXiv:2403.06563 , year=

    Unraveling the mystery of scaling laws: Part i , author=. arXiv preprint arXiv:2403.06563 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2209.10372 , year=

    Welm: A well-read pre-trained language model for chinese , author=. arXiv preprint arXiv:2209.10372 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2602.23699 , year=

    Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit , author=. arXiv preprint arXiv:2602.23699 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2603.14785 , year=

    SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation , author=. arXiv preprint arXiv:2603.14785 , year=

    Pith/arXiv arXiv

  9. [9]

    2026 , eprint=

    UniRank: Unified Rank Allocation for Low-Rank LLM Compression , author=. 2026 , eprint=

    2026

  10. [10]

    arXiv preprint arXiv:2510.13831 , year=

    Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference , author=. arXiv preprint arXiv:2510.13831 , year=

    arXiv

  11. [11]

    arXiv preprint arXiv:2601.18091 , year=

    From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models , author=. arXiv preprint arXiv:2601.18091 , year=

    arXiv

  12. [12]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  13. [13]

    ICLR , year=

    Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , author=. ICLR , year=

  14. [14]

    Proceedings of the British Machine Vision Conference , pages=

    Speeding up Convolutional Neural Networks with Low Rank Expansions , author=. Proceedings of the British Machine Vision Conference , pages=. 2014 , organization=

    2014

  15. [15]

    International Conference on Learning Representations , year=

    Language model compression with weighted low-rank factorization , author=. International Conference on Learning Representations , year=

  16. [16]

    International Conference on Machine Learning , year=

    LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models , author=. International Conference on Machine Learning , year=

  17. [17]

    Wang Qinsi and Jinghan Ke and Masayoshi Tomizuka and Kurt Keutzer and Chenfeng Xu , booktitle=. Dobi-

  18. [18]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  19. [19]

    Proceedings of the 40th International Conference on Machine Learning , articleno =

    Frantar, Elias and Alistarh, Dan , title =. Proceedings of the 40th International Conference on Machine Learning , articleno =. 2023 , publisher =

    2023

  20. [20]

    Forty-second International Conference on Machine Learning , year=

    Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective , author=. Forty-second International Conference on Machine Learning , year=

  21. [21]

    Croci and Marcelo Gennari do Nascimento and Torsten Hoefler and James Hensman , booktitle=

    Saleh Ashkboos and Maximilian L. Croci and Marcelo Gennari do Nascimento and Torsten Hoefler and James Hensman , booktitle=. Slice

  22. [22]

    2024 , eprint=

    ShortGPT: Layers in Large Language Models are More Redundant Than You Expect , author=. 2024 , eprint=

    2024

  23. [23]

    Proceedings of the AAAI Conference on Artificial Intelligence, , year=

    Compressing Transformers: Features Are Low-Rank, but Weights Are Not! , author=. Proceedings of the AAAI Conference on Artificial Intelligence, , year=

  24. [24]

    2023 , eprint=

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. 2023 , eprint=

    2023

  25. [25]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  26. [26]

    Anhao Zhao and Fanghua Ye and Yingqi Fan and Junlong Tong and Jing Xiong and Zhiwei Fei and Hui Su and Xiaoyu Shen , booktitle=. Skip

  27. [27]

    Advances in neural information processing systems , volume=

    Redpajama: an open dataset for training large language models , author=. Advances in neural information processing systems , volume=

  28. [28]

    Shortened

    Bo-Kyeong Kim and Geonmin Kim and Tae-Ho Kim and Thibault Castells and Shinkook Choi and Junho Shin and Hyoung-Kyu Song , booktitle=. Shortened

  29. [29]

    arXiv preprint arXiv:2404.02258 , year=

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models , author=. arXiv preprint arXiv:2404.02258 , year=

    Pith/arXiv arXiv

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    D-llm: A token adaptive computing resource allocation strategy for large language models , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    Streamlining Redundant Layers to Compress Large Language Models , volume =

    Chen, Xiaodong and Hu, Yuxuan and Zhang, Jing and Wang, Yanling and Li, Cuiping and Chen, Hong , booktitle =. Streamlining Redundant Layers to Compress Large Language Models , volume =

  32. [32]

    arXiv preprint arXiv:1905.10044 , year=

    Boolq: Exploring the surprising difficulty of natural yes/no questions , author=. arXiv preprint arXiv:1905.10044 , year=

    Pith/arXiv arXiv 1905

  33. [33]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  34. [34]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    Pith/arXiv arXiv 1905

  35. [35]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  36. [36]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  37. [37]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

    Mihaylov, Todor and Clark, Peter and Khot, Tushar and Sabharwal, Ashish. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. doi:10.18653/v1/D18-1260

    doi:10.18653/v1/d18-1260 2018

  38. [38]

    2024 , eprint=

    What Matters in Transformers? Not All Attention is Needed , author=. 2024 , eprint=

    2024

  39. [39]

    ICLR , year=

    Pointer Sentinel Mixture Models , author=. ICLR , year=

  40. [40]

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

  41. [41]

    High-Dimensional Probability: An Introduction with Applications in Data Science

    Vershynin, Roman , year=. High-Dimensional Probability: An Introduction with Applications in Data Science. , booktitle=

  42. [42]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Right answer, wrong score: Uncovering the inconsistencies of LLM evaluation in multiple-choice question answering , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  43. [43]

    IEEE Transactions on Software Engineering , year=

    On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization , author=. IEEE Transactions on Software Engineering , year=

  44. [44]

    Proceedings of the ACM on Software Engineering , volume=

    Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation , author=. Proceedings of the ACM on Software Engineering , volume=. 2025 , publisher=

    2025

  45. [45]

    Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop , pages=

    Large language models for mathematical reasoning: Progresses and challenges , author=. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop , pages=

  46. [46]

    Advances in Neural Information Processing Systems , volume=

    Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold , author=. Advances in Neural Information Processing Systems , volume=

  47. [47]

    International Conference on Machine Learning , pages=

    From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications , author=. International Conference on Machine Learning , pages=. 2025 , organization=

    2025

  48. [48]

    Advances in Neural Information Processing Systems , volume=

    Compressing large language models using low rank and low precision decomposition , author=. Advances in Neural Information Processing Systems , volume=

  49. [49]

    Flexigpt: Pruning and extending large language models with low-rank weight sharing , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  50. [50]

    International Conference on Learning Representations , volume=

    OATS: Outlier-aware pruning through sparse and low rank decomposition , author=. International Conference on Learning Representations , volume=

  51. [51]

    International Conference on Learning Representations , volume=

    Svd-llm: Truncation-aware singular value decomposition for large language model compression , author=. International Conference on Learning Representations , volume=

  52. [52]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Adaptive feature-based low-rank compression of large language models via bayesian optimization , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  53. [53]

    International Conference on Learning Representations , volume=

    Lolcats: On low-rank linearizing of large language models , author=. International Conference on Learning Representations , volume=

  54. [54]

    International Conference on Learning Representations , volume=

    Modegpt: Modular decomposition for large language model compression , author=. International Conference on Learning Representations , volume=

  55. [55]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  56. [56]

    Adaptive Rank Selections for Low-Rank Approximation of Language Models

    Gao, Shangqian and Hua, Ting and Hsu, Yen-Chang and Shen, Yilin and Jin, Hongxia. Adaptive Rank Selections for Low-Rank Approximation of Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024

    2024

  57. [57]

    arXiv preprint arXiv:2509.25622 , year=

    Layer-wise dynamic rank for compressing large language models , author=. arXiv preprint arXiv:2509.25622 , year=

    arXiv

  58. [58]

    Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    Flat-llm: Fine-grained low-rank activation space transformation for large language model compression , author=. Findings of the Association for Computational Linguistics: EACL 2026 , pages=

    2026

  59. [59]

    The Fourteenth International Conference on Learning Representations , year=

    LeSTD: LLM Compression via Learning-based Sparse Tensor Decomposition , author=. The Fourteenth International Conference on Learning Representations , year=

  60. [60]

    The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter , volume =

    JAISWAL, AJAY and Liu, Shiwei and Chen, Tianlong and Wang, Zhangyang "Atlas" , booktitle =. The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter , volume =

  61. [61]

    International Conference on Learning Representations , volume=

    Compressing llms: The truth is rarely pure and never simple , author=. International Conference on Learning Representations , volume=

  62. [62]

    Workshop on Efficient Systems for Foundation Models II@ ICML2024 , year=

    LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference , author=. Workshop on Efficient Systems for Foundation Models II@ ICML2024 , year=

  63. [63]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Change is the only constant: Dynamic llm slicing based on layer redundancy , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  64. [64]

    NeurIPS 2025 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , year=

    From Entropy Rate to Redundancy: Information Dynamics in Large Language Models , author=. NeurIPS 2025 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , year=

    2025