REVIEW 17 cited by
Model Compression and Efficient Inference for Large Language Models: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.
Forward citations
Cited by 17 Pith papers
-
LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
An empirical GitHub mining study finds vLLM is the most adopted LLM serving framework, parallel and memory optimizations dominate, and multi-framework use is rare.
-
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.
-
Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission
An uncertainty-aware hybrid language model skips and compresses uplink token transmissions, achieving up to 206 times higher token throughput with 97.4% accuracy in simulation.
-
Accelerating Large Language Models through Partially Linear Feed-Forward Network
TARDIS folds two feed-forward weight matrices into one by linearly approximating activations in common input ranges, then recomputes outliers with a small predictor, claiming 80 percent FFN parameter reduction and up ...
-
FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
A token-wise pruner with a learned router and a genetic-algorithm sparsity scheduler claims near-lossless LLM inference at 22-40% token sparsity.
-
Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models
Using CMA-ES to jointly optimize activation quantization scales keeps speech-model accuracy near full precision under full INT8 and INT4 quantization.
-
Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT
OptiKIT combines automated quantization and SLO-driven serving-parameter tuning to more than double per-GPU throughput on some production workloads while preserving near full-precision accuracy.
-
Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.
-
EfficientLLM: Efficiency in Large Language Models
A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.
-
Less is More: Towards Green Code Large Language Models via Unified Structural Pruning
Flab-Pruner applies unified vocabulary, layer, and FFN pruning with a KL objective to Code LLMs, cutting 22% of parameters while retaining about 97% of code-generation performance and recovering the rest via LoRA post...
-
Insights into resource utilization of code small language models serving with runtime engines and execution providers
Serving configuration choice, especially runtime engine and execution provider, significantly changes energy, time, and resource use of code SLMs, with TORCH+CUDA most efficient on the tested hardware.
-
Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics
STA improves task arithmetic by masking task vectors with a first-order Taylor expansion importance metric, raising average fused accuracy to 82.84% on six vision tasks.
-
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Combining SmoothQuant-style channel scaling with QuaRot's Hadamard rotation reduces 4-bit weight-activation quantization error in LLMs by 10-30% of the gap to FP16.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
-
SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors
An integer-only Softmax approximation from I-BERT, mapped onto associative processors, can cut Softmax energy by up to 1300x and latency by up to 12.58x versus GPUs, but with small perplexity loss at the advertised precision.
-
An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
Quantising StarCoder2 with bitsandbytes increases total inference energy by 19-75% because slower inference outweighs lower per-second power, while pruning layers saves little energy and sharply reduces accuracy.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
Discussion (0). Continue with ORCID to comment.