REVIEW 19 cited by
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
read the original abstract
This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa-7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x. Our code is available at https://aka.ms/LLMLingua-2.
Forward citations
Cited by 19 Pith papers
-
Optical Reasoning: Rethinking Images as an Expressive Reasoning Medium Beyond Text
Optical reasoning encodes rationales in images rather than text, matching or exceeding text-based performance on math, science, and multimodal benchmarks while cutting tokens by 28.57% on language tasks and 16% on mul...
-
Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets
Tool schema compression by 44-50% enables agentic RAG at 8K context where uncompressed schemas fail, with +20.5 pp exact match lift across models and scaling to over 800 tools.
-
Mach-Mind-4-Flash Technical Report
Post-training alone—parallel domain RL experts, Multi-Teacher On-Policy Distillation, and Hybrid Median-length Policy Optimization—lifts a 3B-activated MoE to roughly 100B-class agent and reasoning scores.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs are consistent but miscalibrated when selecting verbal descriptors for likelihood and uncertainty of probabilistic predictions, with the bottleneck in verbalization itself.
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs produce consistent but miscalibrated natural-language descriptors of likelihood and uncertainty from probabilistic predictions and are not yet reliable zero-shot risk communicators.
-
HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning
HMARS introduces a hierarchical multi-agent memory system that outperforms standard retrieval and other baselines on long-document and multi-turn reasoning tasks through improved evidence coverage.
-
SOMA: Efficient Multi-turn LLM Serving via Small Language Model
SOMA estimates a local response manifold from early turns and adapts a small surrogate model via divergence-maximizing prompts and localized LoRA fine-tuning for efficient multi-turn serving.
-
Mapping Text to Multiplex Graph: Prompt Compression as L\'evy Walk-Guided Graph Pruning
RAGP models prompt compression as redundancy-aware pruning on a multiplex graph using Lévy walks, achieving 49.3 average on LongBench at 4x compression versus 48.8 for LongLLMLingua at 3x.
-
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
CRISP compresses chain-of-thought by 50-60% using intrinsic attention saliency from the termination token to prune redundancy while preserving accuracy on math tasks.
-
Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models
K-Token Merging compresses LLM inputs by merging blocks of K token embeddings in latent space, achieving up to 75% length reduction with minimal performance drop on reasoning, classification, and code tasks.
-
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
MemoSight unifies context compression and multi-token prediction via special tokens and tailored position layouts to reduce KV cache by up to 66% and accelerate inference by 1.56x while outperforming prior CoT compres...
-
On the Effectiveness of Context Compression for Repository-Level Tasks: An Empirical Investigation
Continuous latent-vector compression improves BLEU scores on repository-level code tasks by up to 28.3% at 4x compression while cutting inference latency.
-
Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models
A unified compressed-sensing framework enables dynamic, task- and token-adaptive structured reduction of LLMs with formal sample-complexity bounds.
-
HyMem: Hybrid Memory Architecture with Dynamic Retrieval Scheduling
HyMem introduces dual-granular memory storage with a lightweight summary module for fast responses and selective activation of a deep LLM module for complex queries, outperforming full-context baselines by 92.6% lower...
-
Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
Extra-CoT trains a semantic compressor on math CoT data, applies mixed-ratio SFT, and uses CHRPO reinforcement learning to achieve over 73% token reduction on MATH-500 with 0.6% accuracy gain on Qwen3-1.7B.
-
ONTO: A Token-Efficient Columnar Notation for LLM Input Optimization
ONTO notation reduces LLM input tokens by 46-51% versus JSON on synthetic operational datasets while preserving task accuracy through a schema-once columnar design with hierarchical support.
-
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
MemoSight unifies context compression and multi-token prediction via special tokens and positional layouts, cutting KV cache up to 66% and speeding inference 56% with under 3% accuracy loss.
-
Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
Byte-exact deduplication reduces RAG context size by 0.16% to 80.34% across three regimes with zero measurable quality regression per multi-vendor LLM evaluation.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.