Pith. sign in

REVIEW 41 cited by

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.05736 v2 pith:FC4Y6H4I submitted 2023-10-09 cs.CL cs.LG

Huiqiang Jiang , Qianhui Wu , Chin-Yew Lin , Yuqing Yang , Lili Qiu This is my paper · ORCID
classification cs.CLcs.LG
keywords compressionlanguagellmlinguamodelsinferencelargellmsmethod
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. With advancements in technologies such as chain-of-thought (CoT) prompting and in-context learning (ICL), the prompts fed to LLMs are becoming increasingly lengthy, even exceeding tens of thousands of tokens. To accelerate model inference and reduce cost, this paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity under high compression ratios, a token-level iterative compression algorithm to better model the interdependence between compressed contents, and an instruction tuning based method for distribution alignment between language models. We conduct experiments and analysis over four datasets from different scenarios, i.e., GSM8K, BBH, ShareGPT, and Arxiv-March23; showing that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss. Our code is available at https://aka.ms/LLMLingua.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 41 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Reconstructing the exact model invocation state for each sampled action, instead of flattening the whole interaction, lets teacher distillation guide compact-memory agents without state mismatch.

  2. Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CAPC combines query-agnostic compression with prompt caching and is reported cheapest in 16/16 LongBench configurations, but the empirical support has unresolved specification and consistency gaps.

  3. Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SPIRAL aligns a model's rendered-image input behavior to its native-text input behavior using self-generated text-path supervision, nearly closing the vision-text compression gap on VTCBench.

  4. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  5. Mach-Mind-4-Flash Technical Report

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Post-training alone—parallel domain RL experts, Multi-Teacher On-Policy Distillation, and Hybrid Median-length Policy Optimization—lifts a 3B-activated MoE to roughly 100B-class agent and reasoning scores.

  6. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  7. Compression, structure, and executor capability: a controlled real-cost decomposition of language-model agent skill optimisation

    cs.SE 2026-07 accept novelty 6.0 of 10

    On 40 software-engineering agent tasks, executor model tier dominates skill optimisations: no shortening, structure, scoped loading, or compiler tier beats raw skills on quality or real cost.

  8. What Context Does a Coding Agent Actually Need to Act?

    cs.LG 2026-06 accept novelty 6.0 of 10

    At oracle localization on SWE-bench Verified, the edited source itself carries the act signal; structured surrounding context resolves no more issues than dropping it, and compressed context matches whole files at one...

  9. Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing

    cs.AI 2026-06 conditional novelty 6.0 of 10

    At compression ratios above 8x, grouping tokens via a Krylov-projected LSH of an implicit attention kernel preserves LLM output quality far better than block averaging, at a large preprocessing latency cost.

  10. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  11. Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

    cs.AI 2026-05 conditional novelty 6.0 of 10

    SCP ranks and prunes context sentences using Shapley values from a learned Deep Sets value function, matching or beating baselines on several multi-hop QA datasets at 50% compression.

  12. Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Extra-CoT trains a semantic compressor on math CoT data, applies mixed-ratio SFT, and uses CHRPO reinforcement learning to achieve over 73% token reduction on MATH-500 with 0.6% accuracy gain on Qwen3-1.7B.

  13. When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents

    cs.CR 2025-10 reject novelty 6.0 of 10

    The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.

  14. ARC-Encoder: learning compressed text representations for large language models

    cs.CL 2025-10 conditional novelty 6.0 of 10

    ARC-Encoder pools queries in an encoder's last attention layer to produce compressed continuous representations that a frozen decoder consumes as token embeddings.

  15. How can we assess human-agent interactions? Case studies in software agent design

    cs.AI 2025-10 conditional novelty 6.0 of 10

    PULSE combines sparse human ratings with prediction-powered inference to cut confidence intervals by ~40% and shows LLM choice matters more than scaffolding for user satisfaction.

  16. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.

  17. LoRA-Gen: Specializing Large Language Model via Online LoRA Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LoRA-Gen generates task-specific LoRA weights with a cloud-side LLM and reparameterizes them into a smaller edge model, enabling training-free specialization with compressed context.

  18. Brevity is the soul of sustainability: Characterizing LLM response lengths

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs produce longer-than-needed answers to factual questions, and simple prompt instructions such as 'provide only the minimal answer' cut response length and inference energy by about 25-60% without hurting automated...

  19. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  20. SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...

  21. QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    QwenLong-CPRS is a 7B instruction-guided compressor that shrinks long contexts to query-relevant spans, boosting downstream LLM accuracy and cutting prefill cost.

  22. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Cache-Craft reuses stored attention states for repeated RAG text chunks with selective recomputation, cutting redundant LLM prefill computation by 51% over prefix caching while retaining roughly 90% of answer quality.

  23. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A training-free method that selects a subset of prompt tokens with a small speculator model to accelerate LLM prefill, yielding up to 7.66x TTFT speedup on Llama-3.1-405B.

  24. EvoWiki: Evaluating LLMs on Evolving Knowledge

    cs.CL 2024-12 conditional novelty 6.0 of 10

    EvoWiki categorizes facts as stable, evolved, or uncharted and shows that LLMs perform much worse on evolved and uncharted knowledge, with RAG plus continual learning providing the best adaptation.

  25. C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

    cs.CL 2024-12 conditional novelty 6.0 of 10

    C3oT uses prompt-conditioned fine-tuning on both long and short chain-of-thought data to generate about 50 percent shorter reasoning traces with roughly unchanged accuracy.

  26. FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing

    cs.CL 2024-12 reject novelty 6.0 of 10

    A token-wise pruner with a learned router and a genetic-algorithm sparsity scheduler claims near-lossless LLM inference at 22-40% token sparsity.

  27. B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens

    cs.CV 2024-12 conditional novelty 6.0 of 10

    B-VLLM selects question-relevant frames and spatial tokens before feeding a video to a vision LLM, improving video QA accuracy under a fixed token budget.

  28. ContextSniper: AntTrail's Token-Efficient Code Memory for Repository-Level Program Repair

    cs.AI 2026-07 unverdicted novelty 5.0 of 10

    ContextSniper reduces token use by 38.9-51.5% in repository-level program repair agents on SWE-bench Lite with 2 percentage point drops in resolution rate.

  29. DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression

    cs.CL 2025-09 reject novelty 5.0 of 10

    DaMoC combines data filtering, token compression, and layer pruning to select the best LLM for domain fine-tuning, claiming ~20x faster training while preserving model rankings.

  30. Lossless Token Sequence Compression via Meta-Tokens

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.

  31. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  32. MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAG

    cs.CL 2025-05 conditional novelty 5.0 of 10

    MacRAG retrieves small slices of compressed summaries, scales up candidate chunks, and merges neighboring chunks, achieving higher average F1 than RAPTOR and LongRAG on three LongBench multi-hop QA datasets.

  33. LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.

  34. AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups

    cs.AI 2026-08 reject novelty 4.0 of 10

    AquiLLM presents an updated local RAG-LLM architecture with memory, skills, multimodal support, and local retrieval, aimed at preserving research group knowledge.

  35. Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Caching query-aware LLM document summaries and reusing them for similar questions cuts input token use by roughly half while keeping answer similarity close to full-document processing.

  36. Addressing the sustainable AI trilemma: a case study on LLM agents and RAG

    cs.CY 2025-01 conditional novelty 4.0 of 10

    LLM-dependent memory operations in agents and RAG consume orders of magnitude more energy than vector methods, and resource-constrained hardware pays higher energy for lower quality.

  37. EvoPat: A Multi-LLM-based Patents Summarization and Analysis Agent

    cs.DL 2024-12 reject novelty 4.0 of 10

    EvoPat uses five specialized LLM roles with retrieval and web search to summarize and compare patents, and the authors report it outperforms GPT-4o, though evaluative details are incomplete.

  38. JPPO++: Joint Power and Denoising-inspired Prompt Optimization for Mobile LLM Services

    cs.NI 2024-12 reject novelty 4.0 of 10

    JPPO++ jointly tunes prompt compression ratio and wireless transmit power with a DRL agent, reporting large latency savings in simulation but with an evaluation that partly reuses its own reward function.

  39. JPPO: Joint Power and Prompt Optimization for Accelerated Large Language Model Services

    eess.AS 2024-11 reject novelty 4.0 of 10

    A joint SLM prompt compression and DRL power allocation framework cuts simulated wireless LLM response time by about 17% while keeping reported fidelity near 0.9.

  40. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  41. PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression

    cs.CL 2025-04 reject novelty 3.0 of 10

    PIS prunes tokens with BERT attention and TF-IDF, chooses per-sentence compression ratios with a DDQN, and drops similar sentences via a Russian roulette rule, reporting modest QA and summarization gains on GPT-4o-mini.

Pith tools