Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Semantic Retention and Extreme Compression in LLMs: Can We Have Both?

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that pruning 25% of a model's weights before 4-bit quantization preserves about 20% more benchmark performance than pure 3-bit quantization at the same theoretical compression rate.

desk verdict A practical configuration recommendation with a clean empirical core, wrapped in a theory and metric that are more heuristic than principled; the reported 20% advantage holds on the data. read the letter →

arxiv 2505.07289 v1 pith:CK6OYEQ5 submitted 2025-05-12 cs.CL cs.AIcs.LG

Stanislas Laborde , Martin Cousseau , Antoun Yaacoub , Lionel Prevost This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords jointcompressionpruningquantizationsemanticretentiontheoreticalratelargelanguagemodelsSparseGPT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether pruning and quantization can be combined to compress large language models beyond what either method achieves alone. It argues that applying SparseGPT pruning first and GPTQ quantization second can approximate true joint compression, and uses a new metric, the Semantic Retention Compression Rate (SrCr), to compare configurations at equal theoretical compression. On LLaMA-3.1-8B and Mistral-7B-v0.3, the paper reports that 25% pruning followed by 4-bit quantization retains about 20% more benchmark performance than pure 3-bit quantization at the same theoretical compression rate. The finding matters because it offers a practical recipe for shrinking models without giving up as much reasoning ability, and because it suggests future work should optimize pruning and quantization jointly rather than pushing one method alone.

What carries the argument

Three pieces carry the argument. First, the Theoretical Compression Rate, $TCr = (1 - q/16)(100 - s)/100$ for joint configurations, with pruning-only and quantization-only as special cases, puts different recipes on one comparable scale. Second, the Semantic Retention Compression Rate, $SrCr_j = (-\log_2(q/16)/4) \cdot \sqrt{p} \cdot Sr_j$, combines sparsity, bit-width, and measured retention into a bounded $[0,1]$ score used to pick optimal configurations. Third, a sequential approximation argument: by decomposing GPTQ's column-wise error into immediate quantization error plus accumulated Hessian-based updates, the paper argues that quantizing all weights after pruning (Case A) approximates quantizing only non-pruned weights (Case B) at low sparsity, and validates this empirically by comparing GPTQ with simpler quantizers (NF4 and LLM.int8()) before and after pruning.

What would settle it

Compare 25% pruning plus 4-bit quantization against pure 3-bit quantization on held-out benchmarks the paper excluded, such as instruction-following, multistep soft reasoning, or graduate-level QA tasks, using the same two models and compression pipeline. If the roughly 20% retention advantage is absent or reversed at the same TCr, the claim is limited to the original three-task evaluation rather than semantic retention in general.

Watch

Extended reading notes

Core claim

The central claim is that a balanced sequential combination of pruning and quantization outperforms quantization alone at the same Theoretical Compression Rate (TCr). Concretely, SparseGPT pruning at 25% sparsity followed by GPTQ 4-bit quantization consistently beats GPTQ-only 3-bit quantization on MMLU-Pro, BBH, and MATH for both tested 7-8B models, with approximately 20% higher semantic retention. The paper also claims this advantage extends qualitatively to semi-structured patterns: 2:8 and 1:4 patterns combined with 4-bit quantization outperform pure 3-bit quantization, and at higher rates 33.333% pruning with 3-bit quantization retains meaningful capability where pure 2-bit quantization collapses. Quantization alone is the stronger single method, so the paper treats it as the baseline; joint compression is presented as superior to pushing quantization to lower bit-widths.

Load-bearing premise

The central claim stands on treating MMLU-Pro, BBH, and MATH as a representative sample of semantic retention; the paper's own benchmark screen excluded tasks where the baseline models were weak, so the 20% figure has only been measured on tasks the models could already handle.

Editorial extensions

If this is right

  • At 81.25% TCr, 25% pruning plus 4-bit quantization is the recommended configuration for both models, delivering roughly 20% higher semantic retention than pure 3-bit quantization.
  • Aggressive pruning with light quantization (50% pruning plus 8-bit) underperforms pure 4-bit quantization at the same 75% TCr, so balanced joint configurations matter rather than any combination.
  • Semi-structured patterns such as 2:8 retain most of the joint-compression benefit and are candidates for hardware acceleration.
  • Mistral-7B-v0.3 tolerates aggressive compression better than LLaMA-3.1-8B, and MATH is consistently the most compression-sensitive benchmark.
  • The SrCr metric gives future compression studies a bounded, interpretable way to compare pruning-quantization configurations on a single scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 20% advantage reproduces on other task suites, it implies that compression recipes should spend a small pruning budget before dropping bit-width instead of pushing quantization alone.
  • The sequential approximation argument leaves a concrete open prediction: a true joint optimizer that coordinates pruning masks with quantization error could beat the sequential recipe by an amount the paper estimates only indirectly.
  • Because only 7-8B models are tested, the optimal joint configuration for larger or smaller models is unknown; the paper itself says the SrCr formula may need adjustment as information density changes.
  • The per-task tables suggest the 20% headline is an average, not uniform behavior; MATH is consistently the most fragile task, so deployment on math-heavy workloads should expect smaller joint-compression gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates joint pruning and quantization for LLMs. It defines Theoretical Compression Rate (TCr) to align different compression configurations at the same nominal size and introduces Semantic Retention Compression Rate (SrCr) as a trade-off metric. Using SparseGPT (pruning) and GPTQ (quantization) on LLaMA-3.1-8B and Mistral-7B-v0.3, with MMLU-Pro, BBH, and MATH benchmarks, the authors report that 25% unstructured pruning with 4-bit quantization outperforms pure 3-bit quantization by approximately 20% in semantic retention at the same 81.25% TCr, and that semi-structured patterns such as 2:8 show similar promise. The central claim is that a balanced joint configuration beats aggressive single-method compression at equal nominal model size.

Significance. The empirical comparison is a useful addition: the results in Table VII directly support the qualitative claim that 25% pruning plus 4-bit quantization retains more benchmark performance than 3-bit quantization alone, and full result tables are provided for both models. However, the theoretical support is not yet sound. Equation (1) is inconsistent with Table I, the numerical claim of 'approximately 20%' is not what the tables show (observed average gain is roughly 30%), the validation of the error approximation compares the wrong quantities, and the SrCr metric embeds the preferences it is used to discover. These issues do not necessarily invalidate the empirical finding, but they must be repaired before the paper's stated claims can be accepted.

major comments (4)
  1. [III-A-1, Eq. (1); Table I] Equation (1) as printed gives TCr = (1 − q/16)·((100−s)/100) for joint compression. For q=4 and s=25 this evaluates to 56.25%, whereas Table I lists 81.25%. The table values are consistent with TCr_joint = 1 − (q/16)·((100−s)/100). Because the headline comparison in Section V requires TCr(25% pruning + 4-bit) = TCr(3-bit) = 81.25%, Eq. (1) must be corrected to match the table; otherwise the two configurations are not matched at the stated TCr.
  2. [Section V; Table VII] The abstract and Section V claim that 25% pruning + 4-bit quantization outperforms 3-bit quantization by approximately 20% in semantic retention at the same TCr. Reading directly from Table VII, the mean score of LLaMA-3.1-8B increases from 23.1 (3-bit) to 31.5 (25%+4-bit), a relative gain of 36.4%, and for Mistral-7B-v0.3 from 24.2 to 30.2, a gain of 24.8%. The average of the two models is about 30.6%, not roughly 20%. The 'approximately 20%' claim should be corrected or the computation behind it should be shown.
  3. [III-B-3, Fig. 4] The validation of the error decomposition in Eqs. (7)-(9) is not carried out on the quantities those equations describe. Equation (9) relates the difference in column-wise quantization error norms (E_GPTQ − E_simple) to the accumulated update norm ||δ_A,j||^2, but Fig. 4 and the accompanying text compare benchmark accuracy scores (MMLU-Pro, BBH, MATH) between GPTQ and NF4/LLM.int8(). Accuracy differences do not directly measure these error norms, so the conclusion that '||δ_A,j||^2 remains well-bounded' is not supported by the plotted evidence. The authors should either plot the actual error norms or supply a formal argument connecting the performance gaps to the error-norm bound.
  4. [III-C-2, Eqs. (13)-(15)] The SrCr metric is constructed with explicitly chosen functional forms—square-root in p, logarithmic in q, and normalization denominator 4—whose stated purpose is to match empirical observations and provide resolution in the practically-viable range. Using this metric in Section IV-B-3 to select 25% pruning + 4-bit quantization as the optimum therefore partly encodes the answer in the metric. The paper should include a sensitivity analysis over the metric's free parameters (exponent on p, normalization constant) and show that the ranking of configurations is robust, or else moderate the claim that SrCr is a principled basis for identifying optimal configurations.
minor comments (4)
  1. [IV-B-3] Point 2 states that the 25% + 4-bit configuration achieves 'around 20% higher semantic retention across all evaluation metrics,' but Table VII shows per-task relative gains ranging from about 15% (Mistral MMLU-Pro) to 150% (LLaMA MATH); please report the per-task numbers or use a range.
  2. [III-C-2, Eq. (15)] Because SrCr_j multiplies sqrt(p) by -log2(q/16)/4, it is exactly zero for any configuration with p=0 or q=16, so the metric cannot rank joint configurations against pruning-only or quantization-only baselines; this limitation should be stated explicitly.
  3. [III-A-1, Eq. (1)] The three-line piecewise definition of TCr in Eq. (1) is typeset without explicit multiplication signs; adding parentheses and an explicit product would prevent the ambiguity that contributes to the inconsistency noted in Major Comment 1.
  4. [IV-A-1] The exclusion of IFEval, MuSR, and GPQA because the baseline models perform poorly on them narrows the meaning of 'semantic retention' to the three selected benchmarks; the abstract and conclusion should qualify the claim accordingly.

Circularity Check

1 steps flagged · score 4.0 of 10

SrCr's functional form is calibrated on the retention curves it is then used to declare optimal; the headline 20% comparison is independent empirical evidence.

  1. fitted input called prediction [Section III-C-2 (Eqs. 13-15) and Section IV-B-1 / IV-B-3]
    "Based on empirical observations of retention rates across setups, we developed the SrCr metrics (Figs. 3(a) and 3(b)) to capture the balance between compression and performance retention in (13)–(15). These metrics indicate that relatively low sparsity levels (25–33.333%) and moderate quantization levels (4–3-bit) tend to maintain better compression-performance trade-offs compared to other approaches."

    SrCr is not derived from an independent first-principles argument; its functional form is chosen to reproduce the same empirical retention observations it is then cited to 'indicate'. The pruning component replaces the natural linear p by sqrt(p) expressly because 'the linear term... would incorrectly favor high sparsity levels' and to give 'higher resolution in the practically-viable pruning range [0%,50%]'. Since which joint configuration maximizes SrCr depends on this hand-chosen nonlinearity, the subsequent conclusion that 25% pruning + 4-bit is 'the optimal compression-performance trade-off' (Section IV-B-3) is partly a restatement of the metric's fitted shape plus the same measured Sr values, not an independent prediction.

full rationale

The strongest claim—25% pruning + 4-bit quantization beats 3-bit-only quantization by approximately 20% at TCr=81.25%—rests on directly measured retention scores in Table VII; this comparison is self-contained and does not use SrCr, so it is not circular. The paper contains no load-bearing self-citations: all cited compression algorithms and benchmarks are external. The only partly circular element is the SrCr optimality story: Eqs. (13)-(15) are explicitly designed to match the authors' empirical observations (sqrt(p) to avoid favoring high sparsity, log(q/16) to match sharp drops), and the same metric is then used to 'show' that 25%+4-bit is optimal. That conclusion is partially an artifact of the chosen nonlinear weighting rather than an independent prediction. Because this affects the auxiliary 'SrCr identifies optimum' framing rather than the headline empirical comparison, the overall circularity score is moderate-low. The TCr table values are consistent with the joint formula TCr = 1 - (q/16)*(1 - s/100) once the implicit multiplication is restored, so no circularity arises there; any ambiguity is a notation issue, not a derivation loop.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical comparison only needs the raw benchmark scores; the SrCr metric, however, rests on two hand-picked functional forms (sqrt(p) and -log2(q/16)/4) and a normalization constant, which are fitted to the authors' observations. The sequential approximation relies on two domain assumptions about information loss and error ordering. No new physical or architectural entities are introduced.

free parameters (4)
  • SrCr_p exponent (square root) = 0.5
    Chosen by hand in Eq. (13) to provide higher resolution in the 0-50% pruning range and to match empirical retention curves; this directly shapes the pruning rate identified as optimal.
  • SrCr_q normalization denominator = 4
    Chosen in Eq. (14) as log2(16/1) to normalize against 1-bit quantization; the resulting -log2(q/16)/4 weighting is an ad hoc fit to observed retention drops below critical bit-widths.
  • epsilon_q (statistical average quantization error)
    Introduced in Eq. (7) as an averaged quantization error term; no value or estimation procedure is given, yet it is central to the error-ratio comparison between Case A and Case B.
  • TCr reference bit-width = 16
    Set as the baseline precision for defining Theoretical Compression Rate; a reasonable default for modern LLM distribution, but a modeling choice that determines all TCr numbers.
assumptions (4)
  • domain assumption Information loss from pruning and quantization is fundamentally irreversible, and the order of operations changes only error distribution, not total information content.
    Invoked in Section III-B-1 to justify sequential compression as an approximation of joint compression. Not proven; the paper supports it only by plausibility.
  • domain assumption Case A (pruning then quantizing all weights) approximates Case B (pruning then quantizing only non-pruned weights) for small sparsity, and therefore approximates joint compression.
    Used in Section III-B-2 to justify the experimental pipeline. The error ratio in Eq. (7) is asserted with an approximation symbol, not derived.
  • domain assumption MMLU-Pro, BBH, and MATH form an approximately independent and representative sample of the model's capability space for computing semantic retention.
    Section IV-A-1. Tasks were chosen partly by excluding benchmarks where the baseline models underperform, so the coverage and independence assumptions are questionable.
  • domain assumption The GPTQ column-wise Hessian error model (Eqs. 2-6) applies to the pruned setting, including the new accumulated-update terms delta_A and delta_B.
    Borrowed from GPTQ [4] but extended here to masked weight matrices; the extension is not formally derived and the delta terms are new.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantic Retention and Extreme Compression in LLMs: Can We Have Both?." pith.science (2026). https://pith.science/paper/CK6OYEQ5

@misc{pith2026250507289,
  author       = {Pith},
  title        = {Pith review of: Semantic Retention and Extreme Compression in LLMs: Can We Have Both?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CK6OYEQ5}},
  note         = {Machine review of arXiv:2505.07289}
}
read the original abstract

The exponential growth in Large Language Model (LLM) deployment has intensified the need for efficient model compression techniques to reduce computational and memory costs. While pruning and quantization have shown promise, their combined potential remains largely unexplored. In this paper, we examine joint compression and how strategically combining pruning and quantization could yield superior performance-to-compression ratios compared to single-method approaches. Recognizing the challenges in accurately assessing LLM performance, we address key limitations of previous evaluation frameworks and introduce the Semantic Retention Compression Rate (SrCr), a novel metric that quantifies the trade-off between model compression and semantic preservation, facilitating the optimization of pruning-quantization configurations. Experiments demonstrate that our recommended combination achieves, on average, a 20% performance increase compared to an equivalent quantization-only model at the same theoretical compression rate.

Figures

Figures reproduced from arXiv: 2505.07289 by the authors.

Figure 1
Figure 1. Retention rates (Rt across tasks and Sr) with pruning-only at different sparsity levels (25%, 33.333%, 50%, and 75%). (a) LLaMA-3.1-8B (b) Mistral-7B-v0.3 [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Retention rates (Rt across tasks and Sr) with quantization-only at different bit-widths (8, 4, 3, and 2). Quantization consistently achieves greater performance than pruning at the same T Cr values. For instance, 8-bit quanti￾zation outperforms 50% pruning despite both achieving the same 50% T Cr, while 4-bit quantization maintains better performance than 75% pruning at 75% T Cr. This observation motivates our use o… view at source ↗
Figure 4
Figure 4. Error ∥δA,j∥ 2 2 analysis by comparison with NF4 and LLM.int8(). 3) Joint Compression Analysis: We focus on three critical compression points highlighted in Table I: 75% T Cr, 81.25% T Cr, and 87.5% T Cr. These represent key thresholds at which compression-performance trade-offs are most notable. TABLE I THEORETICAL COMPRESSION RATE TABLE Quant. Pruning 0% 25% 33.333% 50% 16 bit 0% 25% 33.333% 50% 8 bit 50% 62.5% 66… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Semantic retention compression rates for single compression methods. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 6
Figure 6. Figure 6: Semantic retention compression rate in joint compression ( [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 5
Figure 5. Figure 5: Retention rates (Rt across tasks and Sr) for joint compression vs. quantization-only at different T Cr. The SrCrj metric ( [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Retention rates (Rt across tasks and Sr) for pruning-only with and without semi-structured patterns. (a) LLaMA-3.1-8B (b) Mistral-7B-v0.3 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Retention rates (Rt across tasks and Sr) for joint compression with and without semi-structured patterns. 5) Key Observations: Our comprehensive analysis reveals several fundamental patterns across compression methods: • Model Architecture Impact: Mistral consistently …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

Reference graph

Works this paper leans on

35 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    LoRA: Low-rank adaptation of large language models

    Edward J. Hu et al., “LoRA: Low-rank adaptation of large language models”, Proc. ICLR, 2022

    work page 2022

  2. [2]

    DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter”, Proc. NeurIPS EMC2 Workshop, 2019

    work page 2019

  3. [3]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter, “A simple and effective pruning approach for large language models”, Proc. ICLR, 2024

    work page 2024

  4. [4]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers”, Proc. ICLR, 2023

    work page 2023

  5. [5]

    The era of 1-bit LLMs: All large language models are in 1.58 bits

    Shuming Ma et al., “The era of 1-bit LLMs: All large language models are in 1.58 bits”, arXiv:2402.17764, 2024

    arXiv 2024

  6. [6]

    Compressing LLMs: The truth is rarely pure and never simple

    Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang, “Compressing LLMs: The truth is rarely pure and never simple”, Proc. ICLR, 2024

    work page 2024

  7. [7]

    Densing law of LLMs

    Chaojun Xiao et al., “Densing law of LLMs”, arXiv:2412.04315, 2024

    arXiv 2024

  8. [8]

    SparseGPT: Massive language models can be accurately pruned in one-shot

    Elias Frantar, and Dan Alistarh, “SparseGPT: Massive language models can be accurately pruned in one-shot”, Proc. ICML, 2023

    work page 2023

Show all 35 references
  1. [9]

    Compressing large language models by joint sparsification and quantization

    Jinyang Guo et al., “Compressing large language models by joint sparsification and quantization”, Proc. ICML, 2024

    2024

  2. [10]

    Learning n:m fine-grained structured sparse neural networks from scratch

    Aojun Zhou et al., “Learning n:m fine-grained structured sparse neural networks from scratch”, Proc. ICLR, 2021

    2021

  3. [11]

    LLM-Pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang, “LLM-Pruner: On the structural pruning of large language models”, Proc. NeurIPS, 2023

    2023

  4. [12]

    What matters in transformers? Not all attention is needed

    Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li, “What matters in transformers? Not all attention is needed”, arXiv:2406.15786, 2024

    2024 arXiv

  5. [13]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale”, Proc. NeurIPS, 2022

    2022

  6. [14]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin et al., “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration”, Proc. MLSys, 2024

    2024

  7. [15]

    SmoothQuant: Accurate and efficient post- training quantization for large language models

    Guangxuan Xiao et al., “SmoothQuant: Accurate and efficient post- training quantization for large language models”, Proc. ICML, 2023

    2023

  8. [16]

    QuantEase: Optimization-based quantization for language models

    Kayhan Behdin et al., “QuantEase: Optimization-based quantization for language models”, arXiv:2309.01885, 2023

    2023 arXiv

  9. [17]

    OmniQuant: Omnidirectionally calibrated quantiza- tion for large language models

    Wenqi Shao et al., “OmniQuant: Omnidirectionally calibrated quantiza- tion for large language models”, Proc. ICLR, 2024

    2024

  10. [18]

    Channel- wise mixed-precision quantization for large language models

    Zihan Chen, Bike Xie, Jundong Li, and Cong Shen, “Channel- wise mixed-precision quantization for large language models”, arXiv:2410.13056, 2025

    2025 arXiv

  11. [19]

    Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models

    Jung Hwan Heo et al., “Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models”, Proc. ICLR, 2024

    2024

  12. [20]

    OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models

    Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park, “OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models”, Proc. AAAI, 2024

    2024

  13. [21]

    SpQR: A sparse-quantized representation for near- lossless LLM weight compression

    Tim Dettmers et al., “SpQR: A sparse-quantized representation for near- lossless LLM weight compression”, Proc. ICLR, 2024

    2024

  14. [22]

    SqueezeLLM: Dense-and-sparse quantization

    Sehoon Kim et al., “SqueezeLLM: Dense-and-sparse quantization”, Proc. ICML, 2024

    2024

  15. [23]

    SDQ: Sparse decomposed quantization for LLM inference

    Geonhwa Jeong, Po-An Tsai, Stephen W. Keckler, and Tushar Kr- ishna, “SDQ: Sparse decomposed quantization for LLM inference”, arXiv:2406.13868, 2024

    2024 arXiv

  16. [24]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs”, Proc. NeurIPS, 2023

    2023

  17. [25]

    The Llama 3 herd of models

    Aaron Grattafiori et al., “The Llama 3 herd of models”, arXiv:2407.21783, 2024

    2024 arXiv

  18. [26]

    Mistral 7B

    Albert Q. Jiang et al., “Mistral 7B”, arXiv:2310.06825, 2023

    2023 arXiv

  19. [27]

    A survey on evaluation of large language models

    Yupeng Chang et al., “A survey on evaluation of large language models”, ACM Trans. Intell. Syst. Technol., 2024

    2024

  20. [28]

    Evaluating large language models: A comprehensive survey

    Zishan Guo et al., “Evaluating large language models: A comprehensive survey”, arXiv:2310.19736, 2023

    2023 arXiv

  21. [29]

    Performances are plateauing, let’s make the leader- board steep again

    Clementine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf, “Performances are plateauing, let’s make the leader- board steep again”, HuggingFace Blog, 2024

    2024

  22. [30]

    MMLU-Pro: A more robust and challenging multi- task language understanding benchmark

    Yubo Wang et al., “MMLU-Pro: A more robust and challenging multi- task language understanding benchmark”, Proc. NeurIPS D&B Track, 2024

    2024

  23. [31]

    Challenging BIG-Bench tasks and whether chain- of-thought can solve them

    Mirac Suzgun et al., “Challenging BIG-Bench tasks and whether chain- of-thought can solve them”, Findings of ACL, 2023

    2023

  24. [32]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks et al., “Measuring mathematical problem solving with the MATH dataset”, Proc. NeurIPS D&B Track, 2021

    2021

  25. [33]

    Instruction-following evaluation for large language models

    Jeffrey Zhou et al., “Instruction-following evaluation for large language models”, arXiv:2311.07911, 2023

    2023 arXiv

  26. [34]

    MuSR: Testing the limits of chain-of-thought with multistep soft reasoning

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett, “MuSR: Testing the limits of chain-of-thought with multistep soft reasoning”, Proc. ICLR, 2024

    2024

  27. [35]

    GPQA: A graduate-level Google-proof Q&A bench- mark

    David Rein et al., “GPQA: A graduate-level Google-proof Q&A bench- mark”, Proc. COLM, 2024. APPENDIX TABLE II SPARSEGPT, UNSTRUCTUREDPRUNING-ONLYRESULTS Model Sparsity MMLU-Pro BBH MATH Mean Score Std Score Std Score Std Score Std LLaMA 8B 0% 35.4 0.4 62.3 0.5 17.7 0.5 38.5 0....

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.