Pith. sign in

REVIEW 8 cited by

Model Compression and Efficient Inference for Large Language Models: A Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.09748 v1 pith:RP54NJOW submitted 2024-02-15 cs.CL cs.AIcs.LGcs.PF

Wenxiao Wang , Wei Chen , Yicong Luo , Yongliu Long , Zhengkai Lin , Liye Zhang , Binbin Lin , Deng Cai
show 1 more author
Xiaofei He
This is my paper
classification cs.CLcs.AIcs.LGcs.PF
keywords modelslargecompressionalgorithmslanguageinferencemodelefficient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Analysis of 15 calibration sources shows opposite-sign Spearman correlations between perplexity and retention across General vs. Math/Code dimensions in LLM pruning, and multi-source mixing via IGSP raises total reten...

  2. Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning

    cs.LG 2026-06 reject novelty 6.0 of 10

    At 60% SparseGPT sparsity, mixing four capability-matched calibration corpora preserves 58.8% of unpruned performance versus 40.0% for C4 — but the mix is partially calibrated on the same GSM8K and MBPP benchmarks use...

  3. Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models

    cs.SD 2026-03 conditional novelty 5.0 of 10

    Using CMA-ES to jointly optimize activation quantization scales keeps speech-model accuracy near full precision under full INT8 and INT4 quantization.

  4. Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

    cs.CL 2025-09 conditional novelty 5.0 of 10

    At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.

  5. Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code

    cs.SE 2025-08 unverdicted novelty 5.0 of 10

    Empirical tests show compressed code language models retain task performance but suffer markedly lower robustness under four standard adversarial attacks.

  6. Less LLM, More Documents: Searching for Improved RAG

    cs.IR 2025-10 unverdicted novelty 4.0 of 10

    Corpus scaling in RAG frequently matches the accuracy gains from larger LLMs on open-domain QA tasks, with mid-sized models benefiting most due to better passage coverage.

  7. A Survey on Efficient Inference for Large Language Models

    cs.CL 2024-04 accept novelty 3.0 of 10

    The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.

  8. A Survey on the Memory Mechanism of Large Language Model based Agents

    cs.AI 2024-04 accept novelty 3.0 of 10

    A systematic review of memory designs, evaluation methods, applications, limitations, and future directions for LLM-based agents.

Pith tools