Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey frames LLM compression for edge deployment as three families: distillation, quantization, and pruning.

desk verdict A decent but sloppy survey: useful taxonomy, overstated novelty, and citation errors that need fixing before I'd trust it as a reference. read the letter →

arxiv 2505.02309 v2 pith:QOL5LLM6 submitted 2025-05-05 cs.LG cs.AIcs.CL

Sanjay Surendranath Girija , Shashank Kapoor , Lakshit Arora , Dipen Pradhan , Aman Raj , Ankit Shetgaonkar This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords ModelcompressionLargelanguagemodelsKnowledgedistillationQuantizationPruningEarlyexitEdgedeploymentResource-constrainedenvironments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a survey rather than a new algorithm. Its aim is to establish that the scattered work on making large language models small enough for phones, edge devices, and IoT hardware can be organized into three primary compression families—knowledge distillation, quantization, and pruning—and that a survey focused specifically on resource-constrained deployment is a first in the literature. The paper catalogs the variants inside each family, gives worked examples of each method applied to LLMs, and compares the families on five axes: memory reduction, computation reduction, accuracy degradation, hardware adaptability, and edge suitability. A reader should care because the usefulness of LLMs in everyday devices depends on exactly these trade-offs, and this survey is intended as the reference point for choosing among them.

What carries the argument

The organizing device is a three-way taxonomy of compression families—distillation, quantization, pruning—each with named variants: soft-target, feature-based, relation-based, self-distillation, and multi-teacher distillation; post-training quantization, quantization-aware training, mixed-precision, and binary or ternary quantization; and structured and unstructured pruning. Wrapped around the taxonomy are two tables that collect published LLM compression examples and a comparison table that scores the families on memory reduction, computation reduction, accuracy degradation, hardware adaptability, and edge suitability. The tables and the five trade-off axes do the argumentative work: they turn a list of techniques into a decision-oriented map for someone deploying LLMs under resource constraints.

What would settle it

Locate a peer-reviewed survey published before this paper that explicitly frames LLM compression techniques around resource-constrained or edge deployment; the existence of such a survey would falsify the paper's claim to be the first focused treatment.

Watch

Extended reading notes

Core claim

The paper's central claim is that the field of LLM compression is best understood through three primary techniques with distinct trade-off profiles. Knowledge distillation transfers behavior from a large teacher to a small student; quantization cuts the bit-width of weights and activations; pruning removes low-importance parameters, heads, or layers. The survey then asserts that looking at these techniques from the explicit standpoint of mobile, edge, and IoT deployment—rather than from general model-efficiency or single-technique viewpoints—yields a new and practically oriented organizing frame. It supports that frame with tables of representative LLM compression results and a qualitative comparison of the three families on memory, computation, accuracy, hardware adaptability, and edge suitability. The discovery, if accepted, is not a new compression method but a structured map of the existing ones from the perspective of deployment constraints.

Load-bearing premise

The survey's value as a reference depends on the accuracy of its summaries, tables, and citation mapping; the rendered text already shows citation markers pointing to the wrong references, so the reliability of the map cannot be taken on faith.

Editorial extensions

If this is right

  • A practitioner choosing a compression method can use the survey's comparison to narrow options: quantization offers the largest memory and compute reductions with strong hardware adaptability, while distillation is the option most likely to improve a small student model's accuracy.
  • Post-training quantization is the fastest route because it needs no retraining, but quantization-aware training is the route to choose when the accuracy budget is tight, especially at INT8 and INT4.
  • Structured pruning is more hardware-friendly than unstructured pruning; unstructured sparsity may shrink the model on paper without speeding up inference on dense accelerators.
  • Compression techniques compound: distillation and quantization can be combined, as in layer-by-layer distillation with mixed-precision quantization, to get gains neither method gives alone.
  • Complementary strategies—mixture-of-experts and early exit—can reduce compute per request without shrinking parameter count, so they belong in the same deployment toolkit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the five qualitative comparison axes could be turned into a quantitative decision tool by measuring latency, energy, and accuracy for each family on representative mobile and edge hardware; the survey identifies the axes but leaves the measurement to future work.
  • Editorial inference: if the edge-suitability ranking is correct, the strongest practical deployments will combine structured pruning with low-bit quantization, because that pairing avoids the sparse-matrix penalty on dense accelerators; the survey lists these ingredients without making the combination explicit.
  • Editorial inference: the same three compression families should transfer to vision-language and multimodal models headed to edge devices, where the teacher-student, bit-width, and sparsity trade-offs are likely to reappear; this is a natural extension the survey does not develop.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript is a survey of techniques for adapting large language models to resource-constrained (mobile and edge) environments. It reviews three primary compression families—knowledge distillation, model quantization, and model pruning—together with their variants and example applications, and briefly discusses complementary efficiency strategies such as mixture-of-experts and early-exit mechanisms. The paper also outlines future directions including newer distillation forms, FP8/NF4 representations, neural architecture search, outlier-aware quantization, and quantized finetuning. The authors state in Section I that this is, to their knowledge, the first focused survey of LLM compression from the perspective of resource-constrained environments. The survey contains three tables: examples of KD on LLMs, examples of model quantization, and a qualitative comparison of the three compression techniques.

Significance. If the novelty claim and the reference apparatus were reliable, this survey would be a useful organizational reference for practitioners and researchers entering the area. The paper is clearly structured, covers the major technique families, and usefully includes complementary strategies (MoE, early exit) and recent topics (FP8, NF4, quantized finetuning). The absence of machine-checked proofs or new technical results is expected for a survey; the contribution is expository. However, the paper's value as a trustworthy reference depends on the accuracy of its technical descriptions and citation pointers, and on the validity of its first-survey claim. Both currently need substantive attention before the paper can be recommended for publication.

major comments (3)
  1. [Section III.A.1] The technical core of the knowledge-distillation subsection is incomplete. The text states 'The loss function becomes:' and then jumps to a list of symbol definitions, but the actual equation is absent. Because this equation defines the distillation loss and its temperature scaling, its omission leaves readers without a precise statement of one of the three main techniques. Please insert the full expression (e.g., L = α T² KL(softmax(z_T/T) ‖ softmax(z_S/T)) + (1−α) CE(y, softmax(z_S))) and verify the surrounding notation.
  2. [Table II] The ZeroQuant V2 row cites '[43]', but reference [43] is the Lottery Ticket Hypothesis paper, not the ZeroQuant-V2 paper; the correct bibliography entry appears to be [41]. The same row cites 'BLOOM Family (560M to 176B) [60]', but [60] is a paper on learned N:M sparsity, not the BLOOM model. These mismatches are not cosmetic: in a survey, the tables are the reference apparatus readers rely on, and incorrect pointers undermine the paper's reliability. Please correct every citation in Tables I–III and cross-check each one against the bibliography.
  3. [Section I] The claim that this is 'the first paper that provides a focused survey of LLM compression techniques from the lens of resource-constrained environments' is not currently supported. The manuscript does not define what the resource-constrained lens is operationally, does not report a search or screening protocol, and does not compare its scope against prior surveys. In addition, the sentence that lists prior work [9], [10], [13], [32], [55] as focusing on 'one or two compression techniques' mischaracterizes [9], which is itself a survey of model compression for LLMs covering knowledge distillation, quantization, and pruning. Please either substantiate the first-survey claim with a concrete lens definition and a comparison against existing surveys (including [9] and [10]), or revise the claim to state the paper's scope without asserting novelty.
minor comments (6)
  1. [Section III.B.1] The passage 'However, reducing the precision even further requires more … framework to decompose the quantization of weights into ternary optimization problems' contains a broken sentence and no citation; the described method appears to be from [26] (Bit-Split), which is in the bibliography but never cited in the text.
  2. [Section I] The statement that LLaMa 3 405B does not fit on '8 Nvidia H100 GPUs (800 GB of combined memory)' is numerically incorrect for standard 80 GB H100 parts, which total 640 GB; please correct the memory figure or specify the exact GPU variant.
  3. [References] The DeepSeek-V3 memory requirement in Section I is cited to [62], which is a HuggingFace GGUF quantization page rather than the DeepSeek-V3 technical report; please cite a primary source for this figure.
  4. [References] The SmoothQuant entry [40] lists the first author as 'G. Xizao'; the correct name is Guangxuan Xiao, so the citation should read 'G. Xiao'.
  5. [Table III] The direction of the 'High' values is ambiguous: high is favorable for Memory Reduction and Computation Cost Reduction but unfavorable for Accuracy Degradation; add a note or a visual cue indicating the favorable direction in each column.
  6. [Section II] The phrase '2 FLOPS per active parameter per token' should read '2 FLOPs per active parameter per token', and the unit notation should be made consistent throughout the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's content is compiled from external, independently published work and contains no derived predictions or fitted parameters.

full rationale

This is a survey paper. It does not claim to derive new results, fit parameters, or validate a method against data. The body consists of summaries of external publications (KD, quantization, pruning, MoE, early-exit) and tables referencing those publications. There are no equations in which an output is defined in terms of an input, no fitted quantity renamed as a prediction, and no load-bearing self-citation: the reference list contains no works by the present authors, and no uniqueness or impossibility claim is imported from prior work by the same group. The only normative assertion is the novelty claim in Section I that this is 'the first paper that provides a focused survey of LLM compression techniques from the lens of resource-constrained environments.' That claim is a factual assertion about the literature, not a derivation; it may be unsupported or even false, but an unsupported novelty claim is not circularity unless the proof of novelty explicitly assumes the conclusion, which it does not. The citation mismatches noted in the tables and the missing KD loss equation are quality or correctness issues, not circularity. Accordingly, the correct circularity score is 0.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

As a survey, the paper introduces no free parameters and no invented entities. It relies on the domain assumption that the secondary sources accurately reflect primary results; the citation errors diminish confidence in this assumption.

assumptions (1)
  • domain assumption The cited papers' reported results are accurate and their methods are correctly summarized.
    The survey's usefulness depends on accurate representation of cited work; this is invoked throughout, e.g., in Tables I and II.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques." pith.science (2026). https://pith.science/paper/QOL5LLM6

@misc{pith2026250502309,
  author       = {Pith},
  title        = {Pith review of: Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOL5LLM6}},
  note         = {Machine review of arXiv:2505.02309}
}
read the original abstract

Large Language Models (LLMs) have revolutionized many areas of artificial intelligence (AI), but their substantial resource requirements limit their deployment on mobile and edge devices. This survey paper provides a comprehensive overview of techniques for compressing LLMs to enable efficient inference in resource-constrained environments. We examine three primary approaches: Knowledge Distillation, Model Quantization, and Model Pruning. For each technique, we discuss the underlying principles, present different variants, and provide examples of successful applications. We also briefly discuss complementary techniques such as mixture-of-experts and early-exit strategies. Finally, we highlight promising future directions, aiming to provide a valuable resource for both researchers and practitioners seeking to optimize LLMs for edge deployment.

Figures

Figures reproduced from arXiv: 2505.02309 by the authors.

Figure 1
Figure 1. Different forms of Knowledge Distillation. The student model is represented by the green box and the teacher model is [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Quantizing LLMs to 4 or 8 bits cuts GPU memory by up to 75% with generally small performance changes across eight biomedical NLP benchmarks.

  2. From Construction to Injection: Edit-Based Fingerprints for Large Language Models

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A three-stage LLM fingerprinting pipeline (multilingual low-perplexity triggers, edit-based injection with adversarial suppression, and subspace-aware fine-tuning regularization) reports robust and persistent ownershi...

  3. Opportunities and Applications of GenAI in Smart Cities: A User-Centric Survey

    cs.OH 2025-05 conditional novelty 4.0 of 10

    A user-centric survey of generative AI applications for smart cities, covering conversational interfaces for citizens, operators, and planners.

Reference graph

Works this paper leans on

62 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [9]

    A Survey on Model Compression for Large Language Models,

    X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “A Survey on Model Compression for Large Language Models,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 1556–1577, 2024, doi: 10.1162/tacl_a_00704

    doi:10.1162/tacl_a_00704 2024

  2. [10]

    A Survey on Transformer Compression,

    Y. Tang et al., “A Survey on Transformer Compression,” Apr. 07, 2024, arXiv: arXiv:2402.05964. doi: 10.48550/arXiv.2402.05964

    arXiv doi:10.48550/arxiv.2402.05964 2024

  3. [43]

    The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,

    J. Frankle and M. Carbin, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,” Mar. 04, 2019, arXiv: arXiv:1803.03635. doi: 10.48550/arXiv.1803.03635

    arXiv doi:10.48550/arxiv.1803.03635 2019

  4. [60]

    Learning best combination for efficient n: M sparsity

    Y. Zhang et al., "Learning best combination for efficient n: M sparsity." In Advances in Neural Information Processing Systems 35 (2022)

    work page 2022

  5. [41]

    ZeroQuant -V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation,

    Z. Yao, X. Wu, C. Li, S. Youn, and Y. He, “ZeroQuant -V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation,” May 26, 2023, arXiv: arXiv:2303.08302. doi: 10.48550/arXiv.2303.08302

    arXiv doi:10.48550/arxiv.2303.08302 2023

  6. [2]

    Language models are few -shot learners

    T. Brown et al. "Language models are few -shot learners." Advances in neural information processing systems 33 (2020): 1877-1901

    work page 2020

  7. [13]

    Knowledge Distillation: A Survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge Distillation: A Survey,” Int J Comput Vis, vol. 129, no. 6, pp. 1789 –1819, Jun. 2021, doi: 10.1007/s11263-021-01453-z

    doi:10.1007/s11263-021-01453-z 2021

  8. [32]

    A Survey of Quantization Methods for Efficient Neural Network Inference,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A Survey of Quantization Methods for Efficient Neural Network Inference,” In Low-power computer vision (pp. 291-326)

    work page

  9. [55]

    Pruning and quantization for deep neural network acceleration: A survey,

    T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang, “Pruning and quantization for deep neural network acceleration: A survey,” Neurocomputing, vol. 461, pp. 370 –403, Oct. 2021, doi: 10.1016/j.neucom.2021.07.045

    doi:10.1016/j.neucom.2021.07.045 2021

Show all 62 references
  1. [1]

    Gemini: A Family of Highly Capable Multimodal Models,

    G. Team et al., “Gemini: A Family of Highly Capable Multimodal Models,” Jun. 17, 2024, arXiv: arXiv:2312.11805. doi: 10.48550/arXiv.2312.11805

    2024 doi arXiv

  2. [3]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron et al., “LLaMA: Open and Efficient Foundation Language Models,” Feb. 27, 2023, arXiv: arXiv:2302.13971. doi: 10.48550/arXiv.2302.13971

    2023 doi arXiv

  3. [4]

    The Llama 3 Herd of Models,

    A. Grattafiori et al., “The Llama 3 Herd of Models,” Nov. 23, 2024, arXiv: arXiv:2407.21783. doi: 10.48550/arXiv.2407.21783

    2024 doi arXiv

  4. [5]

    Hierarchical Neural Story Generation,

    A. Fan, M. Lewis, and Y. Dauphin, “Hierarchical Neural Story Generation,” May 13, 2018, arXiv: arXiv:1805.04833. doi: 10.48550/arXiv.1805.04833

    2018 doi arXiv

  5. [6]

    LaMDA: Language Models for Dialog Applications,

    R. Thoppilan et al., “LaMDA: Language Models for Dialog Applications,” Feb. 10, 2022, arXiv: arXiv:2201.08239. doi: 10.48550/arXiv.2201.08239

    2022 doi arXiv

  6. [7]

    A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage,

    M. U. Hadi et al., “A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage,” Jul. 10, 2023. doi: 10.36227/techrxiv.23589741.v1

    2023 doi

  7. [8]

    Highly accurate protein structure prediction with AlphaFold,

    J. Jumper et al., “Highly accurate protein structure prediction with AlphaFold,” Nature, vol. 596, no. 7873, pp. 583 –589, Aug. 2021, doi: 10.1038/s41586-021-03819-2

    2021 doi

  8. [11]

    Distilling the Knowledge in a Neural Network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” Mar. 09, 2015, arXiv: arXiv:1503.02531. doi: 10.48550/arXiv.1503.02531

    2015 doi arXiv

  9. [12]

    Model compression,

    C. Buciluǎ, R. Caruana, and A. Niculescu -Mizil,“Model compression, ” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, Philadelphia PA USA: ACM, Aug. 2006, pp. 535–541. doi: 10.1145/1150402.1150464

    2006

  10. [14]

    OPT: Open Pre-trained Transformer Language Models,

    S. Zhang et al., “OPT: Open Pre-trained Transformer Language Models,” Jun. 21, 2022, arXiv: arXiv:2205.01068. doi: 10.48550/arXiv.2205.01068

    2022 doi arXiv

  11. [15]

    FitNets: Hints for Thin Deep Nets,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “FitNets: Hints for Thin Deep Nets,” Mar. 27, 2015, arXiv: arXiv:1412.6550. doi: 10.48550/arXiv.1412.6550

    2015 doi arXiv

  12. [16]

    Relational knowledge distillation

    W. Park, D. Kim, Y. Lu and M. Cho, "Relational knowledge distillation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019

    2019

  13. [17]

    Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation,

    L. Zhang, J. Song, A. Gao, J. Chen, C. Bao and K. Ma, "Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3712-3721, doi: 10.1109/ICCV.2019.00381

    2019

  14. [18]

    Born again neural networks

    T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, A. Anandkumar, “Born again neural networks”, InInternational conference on machine learning 2018 Jul 3 (pp. 1607-1616). PMLR

    2018

  15. [19]

    Learning from Multiple Teacher Networks,

    S. You, C. Xu, C. Xu, and D. Tao, “Learning from Multiple Teacher Networks,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , Aug. 2017, pp. 1285–1294. doi: 10.1145/3097983.3098135

    2017

  16. [20]

    On-policy distillation of language models: Learning from self -generated mistakes,

    R. Agarwal et al., “On-policy distillation of language models: Learning from self -generated mistakes,” in the Twelfth International Conference on Learning Representations 2024 May

    2024

  17. [21]

    MiniLLM: Knowledge Distillation of Large Language Models,

    Y. Gu, L. Dong, F. Wei, and M. Huang, “MiniLLM: Knowledge Distillation of Large Language Models,” Apr. 10, 2024, arXiv: arXiv:2306.08543. doi: 10.48550/arXiv.2306.08543

    2024 doi arXiv

  18. [22]

    DistiLLM: Towards Streamlined Distillation for Large Language Models,

    J. Ko, S. Kim, T. Chen, and S.-Y. Yun, “DistiLLM: Towards Streamlined Distillation for Large Language Models,” Jul. 03, 2024, arXiv: arXiv:2402.03898. doi: 10.48550/arXiv.2402.03898

    2024 doi arXiv

  19. [23]

    GPT3.int8(): 8- bit Matrix Multiplication for Transformers at Scale,

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “GPT3.int8(): 8- bit Matrix Multiplication for Transformers at Scale,” Advances in Neural Information Processing Systems, vol. 35, pp. 30318–30332, Dec. 2022

    2022

  20. [24]

    Bitnet: Scaling 1 -bit transformers for large language models

    H. Wang, et al. "Bitnet: Scaling 1 -bit transformers for large language models." arXiv preprint arXiv:2310.11453 (2023)

    2023 arXiv

  21. [25]

    Quantizing deep convolutional networks for efficient inference: A whitepaper,

    R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” Jun. 21, 2018, arXiv: arXiv:1806.08342. doi: 10.48550/arXiv.1806.08342

    2018 doi arXiv

  22. [26]

    Towards Accurate Post-training Network Quantization via Bit-Split and Stitching,

    P. Wang, Q. Chen, X. He, and J. Cheng, “Towards Accurate Post-training Network Quantization via Bit-Split and Stitching,” in Proceedings of the 37th International Conference on Machine Learning, PMLR, Nov. 2020, pp. 9847–9856

    2020

  23. [27]

    Up or down? Adaptive rounding for post -training quantization

    M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos and T. Blankevoort, "Up or down? Adaptive rounding for post -training quantization", Proceedings of the 37th Int. Conference on Machine Learning, pp. 7197- 7206, 2020

    2020

  24. [28]

    BRECQ: Pushing the Limit of Post -Training Quantization by Block Reconstruction,

    Y. Li et al., “BRECQ: Pushing the Limit of Post -Training Quantization by Block Reconstruction,” Jul. 25, 2021, arXiv: arXiv:2102.05426. doi: 10.48550/arXiv.2102.05426

    2021 doi arXiv

  25. [29]

    Quantization and Training of Neural Networks for Efficient Integer -Arithmetic-Only Inference,

    B. Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer -Arithmetic-Only Inference,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 2704–2713. doi: 10.1109/CVPR.2018.00286

    2018

  26. [30]

    GPTQ: Accurate Post-Training Quantization for Generative Pre -trained Transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre -trained Transformers,” Mar. 22, 2023, arXiv: arXiv:2210.17323. doi: 10.48550/arXiv.2210.17323

    2023 doi arXiv

  27. [31]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation,

    Y. Bengio, N. Léonard, and A. Courville, “Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation,” Aug. 15, 2013, arXiv: arXiv:1308.3432. doi: 10.48550/arXiv.1308.3432

    2013 doi arXiv

  28. [33]

    Understanding and Overcoming the Challenges of Efficient Transformer Quantization,

    Y. Bondarenko, M. Nagel, and T. Blankevoort, “Understanding and Overcoming the Challenges of Efficient Transformer Quantization,” Sep. 27, 2021, arXiv: arXiv:2109.12948. doi: 10.48550/arXiv.2109.12948

    2021 doi arXiv

  29. [34]

    ZeroQuant: Efficient and Affordable Post -Training Quantization for Large-Scale Transformers,

    Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “ZeroQuant: Efficient and Affordable Post -Training Quantization for Large-Scale Transformers,” Advances in Neural Information Processing Systems, vol. 35, pp. 27168–27183, Dec. 2022

    2022

  30. [35]

    HAQ: Hardware -Aware Automated Quantization With Mixed Precision,

    K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, “HAQ: Hardware -Aware Automated Quantization With Mixed Precision,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA: IEEE, Jun. 2019, pp. 8604 –8612. doi: 10.1109/CVPR.2019.00881

    2019

  31. [36]

    Binaryconnect: Training deep neural networks with binary weights during propagations ,

    M. Courbariaux, Y. Bengio, and J. P. David, “Binaryconnect: Training deep neural networks with binary weights during propagations ,” in J. P. in Advances in Neural Information Processing Systems 2015, 28

    2015

  32. [37]

    Ternary Weight Networks,

    B. Liu, F. Li, X. Wang, B. Zhang, and J. Yan, “Ternary Weight Networks,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Jun. 2023, pp. 1 –5. doi: 10.1109/ICASSP49357.2023.10094626

    2023

  33. [38]

    Do Deep Nets Really Need to be Deep?,

    L. J. Ba and R. Caruana, “Do Deep Nets Really Need to be Deep?,” in Advances in Neural Information Processing Systems 2014, 27

    2014

  34. [39]

    Model compression via distillation and quantization,

    A. Polino, R. Pascanu, and D. Alistarh, “Model compression via distillation and quantization,” Feb. 15, 2018, arXiv: arXiv:1802.05668. doi: 10.48550/arXiv.1802.05668

    2018 doi arXiv

  35. [40]

    SmoothQuant: Accurate and Efficient Post -Training Quantization for Large Language Models,

    G. Xi zao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and Efficient Post -Training Quantization for Large Language Models,” In International Conference on Machine Learning (pp. 38087-38099). PMLR

  36. [42]

    AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han, “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,” GetMobile: Mobile Comp. and Comm., vol. 28, no. 4, pp. 12–17, Jan. 2025, doi: 10.1145/3714983.3714987

    2025

  37. [44]

    Optimal brain damage. Advances

    Y. LeCun, J. Denker, and S. Solla, “Optimal brain damage. Advances” In Neural Information Processing Systems, 1989

    1989

  38. [45]

    Optimal Brain Surgeon and general network pruning,

    B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal Brain Surgeon and general network pruning,” in IEEE International Conference on Neural Networks, San Francisco, CA, USA: IEEE, 1993, pp. 293 –299. doi: 10.1109/ICNN.1993.298572

    1993

  39. [46]

    Learning both weights and connections for efficient neural network

    S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network ” In the Advances In Neural Information Processing Systems, 2015

    2015

  40. [47]

    Movement Pruning: Adaptive Sparsity by Fine-Tuning,

    V. Sanh, T. Wolf, and A. Rush, “Movement Pruning: Adaptive Sparsity by Fine-Tuning,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 20378–20389

    2020

  41. [48]

    AMC: AutoML for Model Compression and Acceleration on Mobile Devices,

    Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “AMC: AutoML for Model Compression and Acceleration on Mobile Devices,” in Computer Vision – ECCV 2018, vol. 11211

    2018

  42. [49]

    Adaptive Mixtures of Local Experts,

    R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive Mixtures of Local Experts,” Neural Computation, vol. 3, no. 1, pp. 79 – 87, Feb. 1991, doi: 10.1162/neco.1991.3.1.79

    1991 doi

  43. [50]

    Outrageously Large Neural Networks: The Sparsely - Gated Mixture -of-Experts Layer,

    N. Shazeer et al., “Outrageously Large Neural Networks: The Sparsely - Gated Mixture -of-Experts Layer,” Jan. 23, 2017, arXiv: arXiv:1701.06538. doi: 10.48550/arXiv.1701.06538

    2017 doi arXiv

  44. [51]

    Glam: Efficient scaling of language models with mixture-of- experts

    N. Du et al. "Glam: Efficient scaling of language models with mixture-of- experts." In International conference on machine learning. PMLR, 2022

    2022

  45. [52]

    BranchyNet: Fast inference via early exiting from deep neural networks,

    S. Teerapittayanon, B. McDanel, and H. T. Kung, “BranchyNet: Fast inference via early exiting from deep neural networks,” in 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun: IEEE, Dec. 2016, pp. 2464–2469. doi: 10.1109/ICPR.2016.7900006

    2016

  46. [53]

    Retrieval-augmented generation for knowledge-intensive nlp tasks ,

    P. Lewis et al., "Retrieval-augmented generation for knowledge-intensive nlp tasks ," In Advances in neural information processing systems 33 (2020): 9459-9474

    2020

  47. [54]

    Structured Pruning of Deep Convolutional Neural Networks,

    S. Anwar, K. Hwang, and W. Sung, “Structured Pruning of Deep Convolutional Neural Networks,” J. Emerg. Technol. Comput. Syst., vol. 13, no. 3, pp. 1–18, Jul. 2017, doi: 10.1145/3005348

    2017 doi

  48. [56]

    Large Language Models Are Reasoning Teachers,

    N. Ho, L. Schmid, and S. -Y. Yun, “Large Language Models Are Reasoning Teachers,” Jun. 13, 2023, arXiv: arXiv:2212.10071. doi: 10.48550/arXiv.2212.10071

    2023 doi arXiv

  49. [57]

    Distilling Step -by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes,

    C.-Y. Hsieh et al., “Distilling Step -by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes,” Jul. 05, 2023, arXiv: arXiv:2305.02301. doi: 10.48550/arXiv.2305.02301

    2023 doi arXiv

  50. [58]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” In Advances in neural information processing systems 36, 2023, 10088-10115

    2023

  51. [59]

    LLaMA -NAS: Efficient Neural Architecture Search for Large Language Models,

    A. Sarah, S. N. Sridhar, M. Szankin, and S. Sundaresan, “LLaMA -NAS: Efficient Neural Architecture Search for Large Language Models,” May 28, 2024, arXiv: arXiv:2405.18377. doi: 10.48550/arXiv.2405.18377

    2024 doi arXiv

  52. [61]

    FP8 Formats for Deep Learning,

    P. Micikevicius et al., “FP8 Formats for Deep Learning,” Sep. 29, 2022, arXiv: arXiv:2209.05433. doi: 10.48550/arXiv.2209.05433

    2022 doi arXiv

  53. [62]

    unsloth/DeepSeek-V3-0324-GGUF · Hugging Face

    “unsloth/DeepSeek-V3-0324-GGUF · Hugging Face.” Accessed: Apr. 19, 2025. [Online]. Available: https://huggingface.co/unsloth/DeepSeek- V3-0324-GG

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.