Pith. sign in

REVIEW 8 cited by

D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.13035 v3 pith:34SSKFYQ submitted 2024-06-18 cs.CL

Zhongwei Wan , Xinjian Wu , Yu Zhang , Yi Xin , Chaofan Tao , Zhihong Zhu , Xin Wang , Siqi Luo
show 3 more authors
Jing Xiong Longyue Wang Mi Zhang
This is my paper · ORCID
classification cs.CL
keywords cachedynamicinferenceattentioncontextdiscriminativedynamicallyeviction
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache, especially for longer sequences. Traditional KV cache eviction strategies, which discard less critical KV pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. In this work, we introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimizes KV cache size dynamically and discriminatively at two levels without fine-tuning, while preserving essential context. At layer level, D2O leverages the varying densities of attention weights between shallow and deep layers to dynamically determine which layers should avoid excessive eviction via a novel dynamic allocation strategy to minimize information loss. At token level, D2O incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of currently discarded tokens, determining whether they should be recalled and merged with similar tokens. We conduct experiments on various benchmarks and LLM architectures. Our results show that D2O not only achieves significant memory savings and enhances inference throughput by more than 3$\times$ but also maintains high-quality long-text generation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  2. Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    ReCo coordinates KV-cache compression, reflection-token logit penalties, and confidence-based early stopping under one per-step process reward, reducing tokens and latency while largely preserving accuracy.

  3. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  4. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  5. LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    LaCache keeps a layer-dependent diagonal slice of the KV cache and iteratively compacts old entries, improving long-context perplexity and retrieval accuracy versus StreamingLLM at fixed cache sizes.

  6. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...

  7. Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.

  8. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

Pith tools