REVIEW 5 cited by
FocusLLM: Precise Understanding of Long Context by Dynamic Condensing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Empowering LLMs with the ability to precisely understand long contexts is crucial for many downstream applications. However, handling long contexts with conventional transformer architecture requires substantial training and inference resources. Existing context condensing methods cannot accurately understand the full context, as there is a considerable amount of information loss in the condensing process. To address these issues, we present FocusLLM, a framework designed to extend the fixed context length of any decoder-only LLM, allowing the model to focus on relevant information from very long sequences. FocusLLM first divides long text input into chunks based on the model's original context length. It then employs the dynamic condensing process to distill crucial information from each chunk. Ultimately, through the novel parallel decoding mechanism, FocusLLM can integrate the extracted information into its local context. FocusLLM stands out for great training efficiency and versatility: trained with an 8K input length and with much less training cost than previous methods, FocusLLM exhibits superior performance across downstream tasks and maintains strong language modeling ability when handling extensive long texts, even up to 400K tokens. Our code is available at https://github.com/leezythu/FocusLLM.
Forward citations
Cited by 5 Pith papers
-
PEVLM: Parallel Encoding for Vision-Language Models
A fine-tuning-free parallel encoding scheme for VLMs that partitions video frames into context blocks with a shared sink block and sequential position embeddings, reducing prefill attention from quadratic to linear wh...
-
APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding
Adaptive Parallel Encoding restores sequential-style attention in parallel KV-cache RAG/ICL with three training-free tweaks, enabling fast pre-cached long-context generation.
-
Semantic Drift and the Stability of Operator Control in Reasoning-Class Decision Support Systems
Reasoning LLMs in ultra-long sessions exhibit latent semantic drift that inverts operator control; a fitted stability coefficient Ks detects the bifurcation and a latent-steering arbitrator is proposed to restore it.
-
HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
HCAttention combines key quantization, CPU value offloading, and cumulative-attention eviction to run long-context LLMs with 12.5% to 25% of the GPU KV cache while keeping LongBench accuracy close to full attention.
-
Context-Enhanced Contrastive Search for Improved LLM Text Generation
CECS extends Contrastive Search with adaptive temperature, candidate thresholds, and penalties, reporting better coherence and diversity, but the description is underspecified.
Discussion (0). Continue with ORCID to comment.