REVIEW 4 cited by
Characterizing Prompt Compression Methods for Long Context Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Long context inference presents challenges at the system level with increased compute and memory requirements, as well as from an accuracy perspective in being able to reason over long contexts. Recently, several methods have been proposed to compress the prompt to reduce the context length. However, there has been little work on comparing the different proposed methods across different tasks through a standardized analysis. This has led to conflicting results. To address this, here we perform a comprehensive characterization and evaluation of different prompt compression methods. In particular, we analyze extractive compression, summarization-based abstractive compression, and token pruning methods. Surprisingly, we find that extractive compression often outperforms all the other approaches, and enables up to 10x compression with minimal accuracy degradation. Interestingly, we also find that despite several recent claims, token pruning methods often lag behind extractive compression. We only found marginal improvements on summarization tasks.
Forward citations
Cited by 4 Pith papers
-
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
Independent hard prompt compression frequently retains an answer while deleting the definition or bridge needed to interpret it, and fixed-budget reinsertion of that missing support substantially improves QA accuracy.
-
Learning What to Remember: Test-Time Training via Context Distillation
IP-TTCD distills the hidden-state gap between a long-window teacher and a short-window student into MLP fast weights, improving long-context language modeling and retrieval over DeltaNet, Gated DeltaNet, SWA, and IP-TTT.
-
SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...
-
Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
HyCo2 combines soft global compression with hard local token selection, reporting QA performance near uncompressed retrieval while cutting context tokens by about 88.8%.
Discussion (0). Sign in to comment.