Pith. sign in

REVIEW 2 major objections 3 minor 52 cited by

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

T0 review · 2 major / 3 minor · reviewed 2026-05-17 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A theoretical upper bound on attention loss from KV cache eviction enables adaptive per-head budget allocation.

desk verdict Ada-KV derives a loss upper bound on pre- and post-eviction attention outputs to motivate head-wise adaptive KV budgets instead of uniform ones, with broad experiments showing gains but the bound's direct support for adaptivity still needing verification. read the letter →

arxiv 2407.11550 v5 pith:6S2KKZLV submitted 2024-07-16 cs.CL cs.AI

Yuan Feng , Junlin Lv , Yukun Cao , Xike Xie , S. Kevin Zhou This is my paper · ORCID
classification cs.CLcs.AI
keywords KVcacheevictionadaptivebudgetallocationLLMinferenceattentionheadslong-contextmodelingcompressionefficiencyoptimization
checked against Cost.FunctionalEquationFoundation.HierarchyEmergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper derives a theoretical upper bound on the loss between attention outputs before and after KV cache eviction. This bound accounts for the optimization goal of earlier eviction techniques and indicates how to assign different cache budgets to individual attention heads. The authors use the bound to create Ada-KV, the first head-wise adaptive allocation method that works as a plug-in with existing eviction approaches. Tests on 13 Ruler and 16 LongBench datasets under both question-aware and question-agnostic conditions show clear quality gains over uniform allocation baselines.

What carries the argument

Theoretical loss upper bound between pre- and post-eviction attention outputs, which guides the choice of non-uniform cache budgets across heads.

What would settle it

Applying the head-wise adaptive allocation derived from the bound to new models or datasets and measuring no quality improvement or a drop relative to uniform allocation.

Watch

Extended reading notes

Core claim

A derived loss upper bound between pre- and post-eviction attention outputs explains the target of prior cache eviction work and supports optimizing budget allocation separately for each attention head, resulting in the Ada-KV strategy that yields measurable quality improvements while reducing cache size.

Load-bearing premise

The loss upper bound accurately captures the quality impact of eviction and attention heads show distinct enough patterns to benefit from unequal budgets.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper claims to establish a theoretical loss upper bound between pre- and post-eviction attention outputs that explains the optimization target of prior uniform KV cache eviction methods and guides the design of head-wise adaptive budget allocation. Building on this, it proposes Ada-KV as the first such adaptive strategy, which integrates plug-and-play with existing eviction methods. Extensive experiments on 13 Ruler and 16 LongBench datasets under question-aware and question-agnostic settings report consistent quality gains over baselines.

Significance. If the bound holds and is reasonably tight, the work supplies a principled, mechanics-derived target for cache eviction that moves beyond uniform allocation, with the adaptive strategy offering practical efficiency gains. The broad dataset coverage and open-sourced code strengthen the empirical case and reproducibility. This could influence future KV cache designs by providing an independent theoretical grounding for non-uniform budgets.

major comments (2)
  1. [Theoretical Analysis] Theoretical bound section: the derivation of the loss upper bound from attention output mechanics is independent of final quality metrics, but the manuscript should explicitly show (e.g., via the per-head score distribution properties used in the proof) that adaptive budget allocation reduces the bound more than uniform allocation without additional unstated assumptions on tail behavior across heads. If the bound treats heads symmetrically inside the derivation, the theoretical motivation for Ada-KV would require further justification beyond the observed empirical gains.
  2. [§4] §4 (budget allocation algorithm): the post-hoc choices in computing per-head budgets are not fully detailed in how they interact with the bound; an ablation confirming that the adaptive rule minimizes the derived bound (rather than a proxy) would make the link between theory and method load-bearing.
minor comments (3)
  1. [Abstract] Abstract: 'Base on this' should read 'Based on this'.
  2. [Experiments] The manuscript would benefit from a short paragraph or table entry quantifying bound tightness (actual vs. upper-bound loss) on a representative subset of the evaluated datasets.
  3. [Method] Notation for the per-head budget variables should be introduced once and used consistently; current usage mixes symbols in the method description.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive evaluation and recommendation for minor revision. The comments raise valid points about strengthening the explicit connection between the theoretical bound and the adaptive allocation. We respond to each major comment below and indicate the revisions we will incorporate.

read point-by-point responses

  1. Referee: [Theoretical Analysis] Theoretical bound section: the derivation of the loss upper bound from attention output mechanics is independent of final quality metrics, but the manuscript should explicitly show (e.g., via the per-head score distribution properties used in the proof) that adaptive budget allocation reduces the bound more than uniform allocation without additional unstated assumptions on tail behavior across heads. If the bound treats heads symmetrically inside the derivation, the theoretical motivation for Ada-KV would require further justification beyond the observed empirical gains.

    Authors: We thank the referee for this observation. The upper bound is derived separately for each attention head from the mechanics of its attention scores and is additive across heads because the final output is assembled from head-wise results. The per-head bound is a monotonic function of the dispersion and tail properties of that head's score distribution for a given retained budget. Heads exhibiting heavier tails or greater dispersion therefore contribute larger terms to the total bound. Allocating the global budget proportionally to these head-specific quantities therefore yields a strictly smaller total bound than uniform allocation for the same aggregate budget. No cross-head assumptions on tail behavior are required; the argument relies only on the observed heterogeneity in per-head score distributions, which is already used in the proof. In the revised manuscript we will insert a short paragraph immediately after the bound derivation that explicitly compares the total bound value under adaptive versus uniform allocation, using the per-head distribution properties to demonstrate the reduction. revision: yes

  2. Referee: [§4] §4 (budget allocation algorithm): the post-hoc choices in computing per-head budgets are not fully detailed in how they interact with the bound; an ablation confirming that the adaptive rule minimizes the derived bound (rather than a proxy) would make the link between theory and method load-bearing.

    Authors: We agree that the interaction between the allocation rule and the bound can be made more transparent. The per-head budgets in §4 are obtained by partitioning the total budget in proportion to the per-head upper-bound values estimated directly from each head's attention-score distribution; this is a closed-form rule that targets minimization of the sum of the per-head bounds. To make the link load-bearing, we will add a new ablation subsection that evaluates the numerical value of the derived loss upper bound itself (not a downstream quality metric) for Ada-KV, uniform allocation, and two alternative heuristics. The results will confirm that the adaptive rule produces the lowest bound value among the compared strategies under identical total budgets. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Theoretical loss upper bound derived from attention mechanics provides independent grounding

full rationale

The paper's central derivation establishes a loss upper bound between pre- and post-eviction attention outputs directly from the mechanics of attention computation. This bound is used to reinterpret prior uniform-eviction methods and to motivate head-wise adaptive budget allocation as an optimization step. No equations or claims reduce the bound or the adaptive strategy to fitted parameters, self-citations, or definitional equivalences; the bound is presented as a first-principles result that remains sensitive to per-head score distributions. Empirical results on Ruler and LongBench are reported separately as validation rather than as the source of the bound itself. The derivation chain is therefore self-contained against external attention mathematics.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on standard transformer attention mechanics and a derived bound; no new free parameters, invented entities, or ad-hoc axioms are introduced in the abstract.

assumptions (1)
  • domain assumption Attention output difference after eviction admits an upper bound that can guide per-head budget decisions
    Invoked to justify the adaptive allocation over uniform budgets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference." pith.science (2026). https://pith.science/paper/6S2KKZLV

@misc{pith2026240711550,
  author       = {Pith},
  title        = {Pith review of: Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6S2KKZLV}},
  note         = {Machine review of arXiv:2407.11550}
}
read the original abstract

Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache required for long-sequence inference. Recent efforts aim to reduce KV cache size by evicting vast non-critical cache elements during runtime while preserving generation quality. However, these methods typically allocate compression budgets uniformly across all attention heads, ignoring the unique attention patterns of each head. In this paper, we establish a theoretical loss upper bound between pre- and post-eviction attention output, explaining the optimization target of prior cache eviction methods, while guiding the optimization of adaptive budget allocation. Base on this, we propose {\it Ada-KV}, the first head-wise adaptive budget allocation strategy. It offers plug-and-play benefits, enabling seamless integration with prior cache eviction methods. Extensive evaluations on 13 datasets from Ruler and 16 datasets from LongBench, all conducted under both question-aware and question-agnostic scenarios, demonstrate substantial quality improvements over existing methods. Our code is available at https://github.com/FFY0/AdaKV.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 52 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Error Certificates for KV-Cache Eviction via Randomized Design

    cs.LG 2026-07 conditional novelty 8.0 of 10

    Deterministic KV-cache eviction provably cannot self-diagnose its own error; Poisson-sampled eviction with a Hajek logit offset yields an empirically valid error certificate (0.97 coverage) that attributes, but does n...

  2. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  3. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  4. RoPE-Aware Bit Allocation for KV-Cache Quantization

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Block-GTQ performs RoPE-aware greedy bit allocation on KV caches using per-block energy scores, cutting logit MAE 32-80% versus uniform TQ-MSE and lifting long-context task scores substantially at 2-3 bits per dimension.

  5. Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Tangram makes non-uniform KV cache compression practical for LLM serving with deterministic budget allocation, head group paging, and ahead-of-time load balancing, achieving up to 2.6x throughput gains.

  6. QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    QCFuse achieves full-prefill quality in RAG with 1.7x average prefill speedup over full prefill and 1.5x over ProphetKV via compressed query-aware cache fusion.

  7. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    AURA-Mem uses an action-gated recurrent memory trained on closed-loop action error to deliver constant 4,224-byte state and 5-9x fewer writes than baselines while matching base policy success on LIBERO-Long.

  8. Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression

    cs.IT 2026-05 unverdicted novelty 7.0 of 10

    Under a polynomial context-truncation sensitivity assumption, suffix-only KV cache policies require per-token memory scaling as Θ(ε^{-1/α}) to achieve distortion ε.

  9. Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation

    cs.CV 2026-05 conditional novelty 7.0 of 10

    HeadKV compresses KV cache for autoregressive image generation via head-aware budget allocation, early head-type identification from consistent patterns, and stratified token eviction.

  10. Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

    cs.CL 2025-02 unverdicted novelty 7.0 of 10

    KV cache compression causes task-dependent degradation in high-density reasoning due to disrupted CoT links; ShotKV mitigates this by preserving few-shot examples as indivisible semantic units through phase separation...

  11. FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration

    cs.LG 2025-02 unverdicted novelty 7.0 of 10

    FastKV decouples prefill context reduction via Token-Selective Propagation from independent KV cache selection, delivering up to 1.82x prefill and 2.87x decoding speedups while matching decoding-only accuracy.

  12. PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.

  13. MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Region-labeled tokens in LLM agent traces have order-of-magnitude different attention lifetimes, and a decay-plus-pinning eviction policy preserves system facts under fixed KV budgets while recency collapses as context grows.

  14. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  15. CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    CompressKV uses Semantic Retrieval Heads to guide KV-cache token selection and layer-wise budget allocation, retaining over 97% performance with 3% cache on LongBench QA tasks.

  16. TGV-KV: Text-Grounded KV Eviction for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    TGV-KV uses text-vision budgeting, weighted ranking, and prioritised retention to evict KV cache in VLMs while retaining 99.2% accuracy at 5% budget on VizWiz-VQA.

  17. MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    MomentKV maintains count, key mean, value mean, and value-key covariance over evicted tokens to guide selective eviction and provide a first-order approximation of their attention contribution, outperforming baselines...

  18. Adaptive Mass-Segmented KV Compression for Long-Context Reasoning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    AMS KV compression adaptively partitions the cache by attention mass regions and assigns quotas to protect contiguous reasoning blocks during long-context LLM inference.

  19. ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    ArborKV uses search-structure awareness to evict low-reuse KV states in Tree-of-Thoughts inference, delivering up to 4x memory savings with near-full accuracy retention.

  20. Runtime-Certified Bounded-Error Quantized Attention

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    A tiered KV cache architecture computes per-head per-step error bounds on quantized attention and uses adaptive fallback to guarantee bounded or exact outputs relative to FP16 reference.

  21. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

    cs.AR 2026-05 unverdicted novelty 6.0 of 10

    VeriCache turns lossy KV cache compression into lossless LLM inference by drafting with compressed cache and verifying drafts with full cache, achieving up to 4x throughput with identical outputs.

  22. RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    RAT+ pretrains a single dense recurrent-augmented attention model that supports flexible dilated sparse inference after short adaptation, matching dense accuracy at moderate dilation and losing only 1-3 points at high...

  23. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

  24. ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs

    cs.LG 2026-02 conditional novelty 6.0 of 10

    ParisKV retrieves top-K KV pairs on GPU using analytic sign-pattern centroids and calibrated 4-bit reranking, matching full attention accuracy in most settings while cutting million-token decode latency up to 44x.

  25. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    Token Sparse Attention uses dynamic per-head token compression and decompression during attention to achieve up to 3.23x speedup at 128K context with under 1% accuracy loss.

  26. EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

    cs.CL 2025-09 unverdicted novelty 6.0 of 10

    EpiCache clusters long conversation history into coherent episodes for per-episode KV cache eviction, delivering up to 30% accuracy gains and 3.7x peak memory reduction on LongConvQA tasks under fixed budgets.

  27. Adaptive KV-Cache Compression without Manually Setting Budget

    cs.DB 2025-09 conditional novelty 6.0 of 10

    GVote removes hand-set KV-cache budgets by synthesizing future queries from the hidden state distribution and keeping the union of keys they vote for, yielding around 2x memory savings at comparable accuracy.

  28. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  29. RCStat: A Statistical Framework for using Relative Contextualization in Transformers

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RCStat uses pre-softmax attention logits to define a Relative Contextualization score that improves adaptive KV-cache eviction and attention-head selection for attribution on LLaMA models.

  30. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.

  31. TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    TailorKV combines 1-bit quantization in shallow attention layers with dynamic Top-K token retrieval in deeper layers to serve 128k-context Llama-3.1-8B on a single 24GB GPU with a small accuracy loss.

  32. CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new selection metric that combines attention weights with projected value-state norms reduces output perturbation in LLM key-value cache eviction.

  33. Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Task-KV identifies 'heterogeneous' attention heads by distance from a per-task semantic center and allocates differentiated KV cache budgets, achieving modest average gains on long-context benchmarks at reduced memory.

  34. Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    K-VEC is a coverage-aware KV-cache eviction strategy using cross-head and cross-layer modules that improves performance by up to 10.35 points over prior methods on LongBench subsets at fixed memory budget.

  35. HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    HARD-KV bridges dynamic head-adaptive KV cache compression with static inference engine constraints via Cascade Cache and Logits Calibration, reporting up to 2x throughput gains on long-context math benchmarks.

  36. Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    AdaQ is a training-free adaptive quasi-Gaussian sampling method for keyframe selection that improves long-video understanding in MLLMs and can outperform GPT-4o with 64 frames.

  37. Recency/Frequency Adaptive KV Caching for Large Language Model Serving

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Presents a recency/frequency adaptive KV caching approach that achieves up to 10.8% higher hit rate and 12.6% lower TTFT compared to vLLM on synthetic workloads.

  38. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

  39. cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations

    astro-ph.GA 2026-04 conditional novelty 5.0 of 10

    Recursive k-section domain decomposition, Morton-key hashing, and GPU dispatch cut communication and memory bottlenecks in RAMSES while preserving conservation to ~0.5%.

  40. Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

    cs.LG 2026-02 conditional novelty 5.0 of 10

    LU-KV allocates per-head KV cache budgets by optimizing learned marginal-utility curves, achieving 80% cache reduction with near-full LongBench accuracy but larger drops on RULER retrieval tasks.

  41. Lag-Relative Sparse Attention In Long Context Training

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Training a language model with LagKV's static sparse attention mask improves its robustness to KV-cache compression at 2x and 4x rates, while claims of overall average gains are not consistently supported.

  42. Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.

  43. SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation

    cs.CL 2024-12 conditional novelty 5.0 of 10

    SCOPE keeps a fixed compressed prefill cache and applies sliding, adaptive, and discontinuous eviction only to decoding-stage KV tokens, improving long-output reasoning accuracy at low memory budgets.

  44. THRIVE: Therapeutic Humanoid Robot In Virtual Environment

    cs.RO 2026-08 conditional novelty 4.0 of 10

    THRIVE is a modular at-home VR rehabilitation system for children with cerebral palsy that combines four customizable reaching games, camera tracking, and an interchangeable physical, remote, or virtual robot therapist.

  45. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  46. Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction

    cs.LG 2026-05 unverdicted novelty 4.0 of 10

    Structural protection of boundary tokens in globally capped KV cache eviction recovers 69-90% of full-cache quality at 13% retention and dominates differences among scoring policies.

  47. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  48. CE-LSLM: Efficient Large-Small Language Model Inference and Communication via Cloud-Edge Collaboration

    cs.NI 2025-05 reject novelty 4.0 of 10

    CE-LSLM reports lower edge inference latency by reusing cloud LLM KV caches in edge SLMs, but the cross-model reuse mechanism is asserted, not demonstrated.

  49. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  50. DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs

    cs.CL 2024-12 conditional novelty 4.0 of 10

    DynamicKV adaptively allocates per-layer KV cache budgets using attention scores, reaching about 90% of full-cache accuracy on LongBench with about 1.7% of the cache.

  51. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.

  52. XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

    cs.LG 2024-12 conditional novelty 4.0 of 10

    XKV gives each network layer a personalized KV cache budget based on attention retention, cutting memory 61.6% with comparable accuracy.

Reference graph

Works this paper leans on

69 extracted references · 69 canonical work pages · cited by 52 Pith papers

  1. [1]

    A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013, 2024

    work page arXiv 2024

  2. [2]

    Summedits: measuring llm ability at factual reasoning through the lens of summarization

    Philippe Laban, Wojciech Kry´sci´nski, Divyansh Agarwal, Alexander Richard Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Summedits: measuring llm ability at factual reasoning through the lens of summarization. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9662–9676, 2023

    work page 2023

  3. [3]

    Llm-based code generation method for golang compiler testing

    Qiuhan Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 2201–2203, 2023

    work page 2023

  4. [4]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    work page Pith review arXiv 2023

  5. [5]

    The claude 3 model family: Opus, sonnet, haiku, March 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, March 2024. Accessed: 2024-07- 09

    work page 2024

  6. [6]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    work page Pith review arXiv 2024

  7. [7]

    Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    work page Pith review arXiv 2023

  8. [8]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

Show all 69 references
  1. [9]

    PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics ACL 2024, ...

    2024

  2. [10]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024

    2024 arXiv

  3. [11]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  4. [12]

    Llm kv cache compression made easy, 2024

    Maximilian Jeblick Simon Jegou. Llm kv cache compression made easy, 2024

    2024

  5. [13]

    Catalyst: Optimizing cache management for large in-memory key-value systems

    Kefei Wang and Feng Chen. Catalyst: Optimizing cache management for large in-memory key-value systems. Proceedings of the VLDB Endowment, 16(13):4339–4352, 2023. 11

    2023

  6. [14]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  7. [15]

    Lm-infinite: Zero-shot extreme length generalization for large language models, 2024

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models, 2024

    2024

  8. [16]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  9. [17]

    Scissorhands: Exploiting the persistence of impor- tance hypothesis for llm kv cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of impor- tance hypothesis for llm kv cache compression at test time. Advances in Neural Information Proce...

    2024

  10. [18]

    On the efficacy of eviction policy for key-value constrained generative language model inference

    Siyu Ren and Kenny Q Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference. arXiv preprint arXiv:2402.06262, 2024

    2024

  11. [19]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processin...

    2024

  12. [20]

    Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention.arXiv preprint arXiv:2504.16083, 2025

    Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention.arXiv preprint arXiv:2504...

    2025

  13. [21]

    Retrievalattention: Accelerating long- context llm inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long- context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024

    2024

  14. [22]

    Arkvale: Efficient generative llm inference with recallable key-value eviction

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative llm inference with recallable key-value eviction. Advances in Neural Information Processing Systems, 37:113134– 113155, 2024

    2024

  15. [23]

    Pqcache: Product quantization-based kvcache for long context llm inference

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization-based kvcache for long context llm inference. Proceedings of the ACM on Management of Data, 3(3):1–30, 2025

    2025

  16. [24]

    Breaking the boundaries of long- context llm inference: Adaptive kv management on a single commodity gpu

    He Sun, Li Li, Mingjun Xiao, and Chengzhong Xu. Breaking the boundaries of long- context llm inference: Adaptive kv management on a single commodity gpu. arXiv preprint arXiv:2506.20187, 2025

    2025

  17. [25]

    Deja vu: Contextual sparsity for efficient llms at inference time, 2023

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient llms at inference time, 2023

    2023

  18. [26]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    2022

  19. [27]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  20. [28]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  21. [29]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri .et al. The llama 3 herd of models, 2024. 12

    2024

  22. [30]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  23. [31]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    2023

  24. [32]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  25. [33]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    2023 arXiv

  26. [34]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325–338, 2024

    2024

  27. [35]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024

    2024 arXiv

  28. [36]

    Kvzip: Query-agnostic kv cache compression with context reconstruction

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. Kvzip: Query-agnostic kv cache compression with context reconstruction. arXiv preprint arXiv:2505.23416, 2025

    2025

  29. [37]

    Expected attention: Kv cache compres- sion by estimating attention from future queries distribution

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: Kv cache compres- sion by estimating attention from future queries distribution. arXiv preprint arXiv:2510.00636, 2025

    2025

  30. [38]

    Draft-based approximate inference for llms

    Kevin Galim, Ethan Ewer, Wonjun Kang, Minjae Lee, Hyung Il Koo, and Kangwook Lee. Draft-based approximate inference for llms. arXiv preprint arXiv:2506.08373, 2025

    2025

  31. [39]

    Identify critical kv cache in llm inference from an output perturbation perspective

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Identify critical kv cache in llm inference from an output perturbation perspective. arXiv preprint arXiv:2502.03805, 2025

    2025

  32. [40]

    Kevin Zhou, and Xike Xie

    Yuan Feng, Haoyu Guo, JunLin Lv, S. Kevin Zhou, and Xike Xie. Taming the fragility of kv cache eviction in llm inference, 2025

    2025

  33. [41]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024

    2024

  34. [42]

    Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410.19258, 2024

    2024

  35. [43]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022

    2022

  36. [44]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  37. [45]

    Qaq: Quality adaptive quantization for llm kv cache

    Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643, 2024

    2024

  38. [46]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024. 13

    2024

  39. [47]

    Longspec: Long-context lossless speculative decoding with efficient drafting and verification

    Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. Longspec: Long-context lossless speculative decoding with efficient drafting and verification. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models

  40. [48]

    Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning

    Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen, and Huan Li. Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning. arXiv preprint arXiv:2508.16201, 2025

    2025

  41. [49]

    Needle In A Haystack - pressure testing LLMs

    Gregory Kamradt. Needle In A Haystack - pressure testing LLMs. Github, 2023

    2023

  42. [50]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. In ICLR, 2024

    2024

  43. [51]

    The narrativeqa reading comprehension challenge

    Tomáš Koˇcisk`y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transac- tions of the Association for Computational Linguistics, 6:317–328, 2018

    2018

  44. [52]

    A dataset of information-seeking questions and answers anchored in research papers

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021

    2021

  45. [53]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  46. [54]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Lingu...

    2020

  47. [55]

    Musique: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  48. [56]

    Efficient attentions for long document summarization

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112, 2021

    2021

  49. [57]

    Qmsum: A new benchmark for query-based multi-domain meeting summarization

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021

    2021

  50. [58]

    Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model

    Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.01749, 2019

    1906 arXiv

  51. [59]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017

    2017

  52. [60]

    Samsum cor- pus: A human-annotated dialogue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum cor- pus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237, 2019

    1911

  53. [61]

    Learning question classifiers

    Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002

    2002

  54. [62]

    Longcoder: A long-range pre-trained language model for code completion, 2023

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion, 2023

    2023

  55. [63]

    Repobench: Benchmarking repository-level code auto-completion systems, 2023

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023. 14

    2023

  56. [64]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023

    2023 arXiv

  57. [65]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, 2020

    2020

  58. [66]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages ...

    2017

  59. [67]

    Longcoder: A long-range pre-trained language model for code completion

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion. arXiv preprint arXiv:2306.14893, 2023

    2023

  60. [68]

    needle," is embedded within a lengthy text, called the

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023. 15 A Appendix A.1 Additional Related Works Additional works also mitigate the challenges posed by massive KV Caches during...

    2023 arXiv

  61. [69]

    unanswerable

    word-a 2. word-b 3. word-c 4. word-a 5. word-d 6. word-a 7. word-e 8. word-f ...... Question: What are the 10 most common words in the above list? Answer: The top 10 words that appear most often in the list are: Frequent Words Extraction (FWE) Task Template: Read the following...

Pith tools

Reviewed May 17, 2026 · model on record in the stance chip above.