REVIEW 14 cited by
Compressing Context to Enhance Inference Efficiency of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Compressing Context to Enhance Inference Efficiency of Large Language Models
read the original abstract
Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.
Forward citations
Cited by 14 Pith papers
-
TSCG: Deterministic Tool-Schema Compilation for Agentic LLM Deployments
TSCG compiles JSON tool schemas into token-efficient structured text, raising tool-use accuracy for small LLMs from 0% to 84.4% on benchmarks while cutting tokens by 52-57%.
-
Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning
SCP ranks and prunes context sentences using Shapley values from a learned Deep Sets value function, matching or beating baselines on several multi-hop QA datasets at 50% compression.
-
Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts
Reasoning language models extract answers from sparse, order-shuffled chain-of-thought traces with little accuracy loss.
-
When Compression Becomes an Attack Surface: Black-Box Attacks on Prompt-Compressed LLM Agents
The paper claims prompt compression is a new attack surface, but the abstract's COMA attack never appears in the body and the body's SoftCom requires white-box access.
-
DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation
DCMI infers RAG database membership by subtracting the system's yes-probability on a perturbed query from the original query, cancelling the interference of non-member retrieved documents.
-
MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
MemAgent uses multi-conversation RL to train a memory agent that reads text in segments and overwrites memory, extrapolating from 8K training to 3.5M token QA with under 5% loss and 95%+ on 512K RULER.
-
END: Early Noise Dropping for Efficient and Effective Context Denoising
LLMs implicitly identify informative context in early layers, enabling a linear prober to drop noisy chunks and improve both performance and efficiency across tasks and models.
-
SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration
A multi-agent web-search framework that stores progress in shared evidence, coverage, and failure state reports the best F1 scores among compared baselines on WideSearch (80.3 item F1) and GISA (76.5 set F1).
-
Information-Aware KV Cache Compression for Long Reasoning
InfoKV is an entropy-aware KV cache compression framework that integrates token-level predictive uncertainty and layer-wise representation evolution with attention scores to improve performance on long-context reasoni...
-
Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS)
SSAS improves LLM sentiment prediction consistency and data quality by up to 30% on three review datasets via syntactic and semantic context assessment summarization.
-
AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models
AdaComp trains a compression-rate predictor on annotated minimum top-k data to adaptively retain only the documents needed for each RAG query.
-
Hierarchical Reranking for Scalable Financial RAG System
A finance-specific RAG pipeline combining table-to-JSON conversion, two-stage reranking, and long-context split-fusion reports NDCG@20=0.7918 and second place in the ICAIF '24 FinanceRAG challenge.
-
Leveraging Weighted Syntactic and Semantic Context Assessment Summary (wSSAS) Towards Text Categorization Using LLMs
wSSAS is a two-phase deterministic framework that uses hierarchical text organization and SNR-based feature prioritization to improve clustering integrity, categorization accuracy, and reproducibility when applying LL...
-
Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines
Entropy Gate applies entropy quenching with adaptive temperature schedules and multi-factor token energies to achieve 40-60% compression in LLM prompts while keeping semantic similarity above 0.80.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.