Pith. sign in

REVIEW 4 major objections 7 minor 64 references

LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a standardized long-context evaluation framework that cuts an 8B model's six-capability assessment to roughly 50 GPU hours.

desk verdict Solid evaluation-framework engineering, but the headline efficiency rests on an unvalidated benchmark subset and a batch-size confound. read the letter →

arxiv 2507.04723 v1 pith:L6GK5TS4 submitted 2025-07-07 cs.CL

Zecheng Tang , Haitian Wang , Quantong Qiu , Baibei Ji , Ruoxi Sun , Keyan Zhou , Juntao Li , Min Zhang This is my paper
classification cs.CL
keywords long-contextevaluationlargelanguagemodelsframeworkbenchmarksuiteinferenceaccelerationretrieval-augmentedgenerationLOOMBenchstandardization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make long-context evaluation of large language models both fair and cheap. It claims that evaluation results across benchmarks are inconsistent because each benchmark uses different prompts, inference settings, and scoring, and that LOOM-Scope removes those confounders by standardizing the evaluation pipeline. It also claims that a lightweight composite benchmark, LOOMBench, built by up-sampling twelve existing benchmarks, covers six long-context capabilities and can assess an 8B model in roughly 50 H20 GPU hours. If true, researchers could compare long-context models on a common footing without spending thousands of GPU hours.

What carries the argument

The load-bearing mechanism is the standardized evaluation pipeline coupled with the lightweight composite benchmark LOOMBench. LOOMBench is constructed by up-sampling from twelve existing long-context benchmarks, spanning all six capability areas, so that a single pass through the pipeline produces a six-capability profile rather than scores scattered across incompatible benchmarks. The standardization of instruction templates and inference hyperparameters inside the BENCHMARK and DEPLOYMENT modules is what makes the scores comparable, while the up-sampling is what makes the whole assessment cheap.

What would settle it

Run the same set of models on the full twelve source benchmarks and on LOOMBench, then compute rank correlation per capability; if LOOMBench's ranking diverges substantially from the full-benchmark ranking, the up-sampled subset is not representative and the efficiency claim reduces to a speed claim.

Watch

Extended reading notes

Core claim

The central claim is that LOOM-Scope is an evaluation platform that combines broad long-context benchmark coverage with built-in inference acceleration, making comprehensive evaluation practical. The BENCHMARK module ingests 22 benchmarks and more than 140 tasks spanning the 8K-to-2M token range; the DEPLOYMENT module runs models on multiple architectures and inference engines and can attach augmentation methods such as retrieval-augmented generation, KV-cache optimization, and sparse attention; and the EVALUATOR module applies task-appropriate discriminative and generative metrics. On top of this, LOOMBench up-samples tasks from twelve of the supported benchmarks so that one run yields scores for six capabilities: General, Faithfulness, Reasoning, Retrieval, Generation, and Specialization. The authors report that a full LOOMBench assessment of an 8B model takes about 50 H20 GPU hours, and that acceleration methods can reach up to 12x speedup on 128K-length RULER tasks.

Load-bearing premise

The load-bearing premise is that the samples LOOMBench keeps from the twelve source benchmarks are representative of the full benchmarks, so a score on LOOMBench reflects true long-context ability; the paper does not specify how many samples per task or how they were chosen.

Editorial extensions

If this is right

  • A standard, fair comparison of long-context models becomes possible because prompts and inference settings are unified across the 22 supported benchmarks.
  • Comprehensive six-capability evaluation becomes accessible to labs with modest GPU budgets; an 8B model can be assessed in about 50 H20 GPU hours.
  • Practitioners can directly measure the accuracy-latency trade-off of acceleration methods such as KV-cache compression, sparse attention, and retrieval augmentation under the same evaluation conditions.
  • The framework's support for non-Transformer architectures extends long-context evaluation beyond standard attention-based checkpoints.
  • Because LOOMBench is built from existing benchmarks, its scores can be traced back to the source tasks, preserving comparability with prior work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LOOMBench's up-sampling is representative, it could become a de facto lightweight standard for long-context evaluation, but that representativeness needs to be demonstrated by rank-correlation analysis against the full benchmarks.
  • The platform's integration of acceleration methods points toward a useful service: measuring accuracy-latency Pareto frontiers for long-context models, which the paper only begins with RULER.
  • The observed results suggest that simple lexical retrieval can hurt long-context model performance while model-based retrieval can help, so augmentation choices should be treated as part of the evaluated system rather than a separate axis.
  • Extending the same standardized pipeline to multimodal long-context inputs is the natural next step the paper names as future work, and the modular design suggests it could absorb such tasks without re-architecting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. LOOM-Scope is a framework for evaluating long-context LLMs that standardizes evaluation settings, supports 22 benchmarks and more than 140 tasks, integrates multiple inference engines (Hugging Face, vLLM, SGLang, API), and provides plug-in support for RAG, KV-cache compression, and sparse-attention acceleration methods. The paper introduces LOOMBench, a composite benchmark up-sampled from 12 existing benchmarks, and claims that a full six-capability evaluation of an 8B-scale model on LOOMBench costs only about 50 H20 GPU hours. It reports evaluations of 14 models across six capability dimensions, compares RAG and acceleration methods, and argues that LOOM-Scope is the most comprehensive and the only acceleration-aware long-context evaluation platform.

Significance. If the framework and LOOMBench are validated, this is a useful community resource: it is open-source, covers a broad set of benchmarks, supports heterogeneous model architectures and inference backends, and provides a unified environment for comparing long-context models and acceleration techniques. The explicit release of code, evaluation scripts, and demo inputs is a strength, as is the breadth of the empirical comparison across 14 models. The central efficiency claim, however, depends on LOOMBench being a representative and statistically reliable proxy for long-context capability, and this is not yet demonstrated. The acceleration speedup comparisons also contain a batch-size confound. These issues are load-bearing for the paper's headline claims and require revision.

major comments (4)
  1. [Appendix B.2, Table 4, Table 6] The representativeness of LOOMBench is not established. Appendix B.2 states only that LOOMBench is 'up-sampled and reorganized from 12 different benchmarks' and that the selected datasets 'ensure a balanced evaluation across six core long-context capabilities,' but it provides no per-task sample sizes, no sampling algorithm, no length distributions, and no inclusion criteria for the 12 source benchmarks. Table 6 shows only latency comparisons between LOOMBench and the native benchmarks, not an accuracy or ranking correlation. Without a validation that model rankings on LOOMBench agree with rankings on the full source benchmarks, the headline efficiency figure (about 50 H20 GPU hours) only shows that a small, unvalidated subset can be evaluated quickly. This is a load-bearing omission because the central 'comprehensive and efficient' claim depends on LOOMBench being a valid proxy; the Limitation section does not acknowledge this missing validation.
  2. [Section 3.2, Table 7] The acceleration latency comparison is not controlled. The paper states that the Native Transformer (FlashAttention) ran with batch size 1 while all acceleration methods ran with batch size 8, and Table 7's caption says 'The batch size per GPU is 8.' This mismatch confounds the effect of the acceleration method with the effect of larger batch processing, so the reported up-to-12x speedup and the per-method latency numbers in Figure 7 and Figure 11 do not isolate the acceleration contribution. The authors should measure all methods at the same batch size, or report throughput per GPU at matched batch sizes, and state the batch size for every row in Table 7 including the Transformer/vLLM/SGLang baselines.
  3. [Table 5, Figure 5, Section 3.2] All reported accuracy and latency results appear to be single runs with no variance estimates. This is particularly problematic because several LOOMBench components are small: Counting-Stars has 128 instances (Table 3), and the RULER acceleration evaluation uses 15 instances per subtask (Section 3.2). With sample sizes this small, differences of a few points in Table 5 (e.g., the 44.71 vs. 44.89 average scores separating Qwen3-8B and GLM-4-9B-chat) are within likely sampling noise. The paper should provide multiple seeds or bootstrap confidence intervals, or at least report per-task sample sizes so readers can judge the reliability of the rankings.
  4. [Section 3.3, Table 1] The claim that LOOM-Scope is 'the only existing platform that incorporates long-context inference acceleration methods' is stronger than the evidence in Table 1. The comparison lists only four other frameworks (OpenCompass, EvalHardness, UltraEval, TAIL) with no stated inclusion criteria or systematic survey, so the 'only' claim is not established. The authors should either qualify the claim to the frameworks surveyed or provide a more systematic comparison.
minor comments (7)
  1. [Throughout] The benchmark name is inconsistently typeset as 'LOOMB ENCH' in many places (e.g., Abstract, Section 3, Table 6); this should be corrected to 'LOOMBench'.
  2. [Section 2.4] There is a typo: 'specified sever' should read 'specified server'.
  3. [Figure 2] The figure caption says 'the workflow of three modules can be refereed to Figure 1'; 'refereed' should be 'referred'.
  4. [Table 3] The table caption says 'The hyphen ("-") represents infinite length,' but hyphens also appear in the Data Volume column (e.g., RULER, NIAH, NoLiMa) and in the Data Length column, which is ambiguous; please distinguish 'not reported' from 'infinite length.'
  5. [Figure 7 and Figure 11] The figures would benefit from explicit axis labels, units, and a legend that clearly identifies which bars correspond to which GPU and which method; as printed, the reader must infer this from the caption and Section 3.2.
  6. [Section 3.2, Table 7 caption] The caption states 'The batch size per GPU is 8' but the text says the Native Transformer baseline used batch size 1; the caption should state the batch size used for every configuration, including the non-accelerated baselines.
  7. [References] The reference list contains duplicate entries for Hooper et al. (KVQuant appears as both 2024a and 2024b with identical content); this should be consolidated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LOOM-Scope is a benchmark-integration and measurement framework; its efficiency and capability claims rest on direct runs of external benchmarks and native inference baselines, not on fitted parameters or self-referential derivation.

full rationale

The paper's central claims are platform support breadth (22 benchmarks, 140+ tasks), standardized evaluation settings, acceleration-method integration, and the lightweight LOOMBench suite. None of these involve fitting a parameter and then predicting a closely related quantity from that same fit. LOOMBench is explicitly constructed by up-sampling and reorganizing 12 existing external benchmarks, and the scores in Table 5 are measured on that subset; they are not derived from LOOM-Scope's own design assumptions. The headline efficiency figures (about 50 H20 GPU hours for an 8B model) are direct wall-clock measurements reported in Table 6, comparing LOOMBench against native full-benchmark runs on the same hardware, so the efficiency claim is not imposed by construction. Acceleration results in Table 7 likewise compare native Transformers, vLLM, SGLang, and various acceleration methods on RULER under fixed sampling and batch-size settings; no fitted parameter is renamed as a prediction. The self-citations that appear (e.g., Su et al. 2025 on KV-cache quantization) are background references to external techniques and are not load-bearing for the framework's own conclusions. The strongest available concern is that LOOMBench's representativeness is not validated in Appendix B.2: per-task sample sizes, the sampling algorithm, and a correlation check against full-benchmark rankings are absent, so the 'comprehensive' characterization is only asserted. That is an external-validity or correctness concern, not circularity, because the benchmark data are externally sourced and the paper does not use LOOMBench scores as inputs to derive its own efficiency or comprehensiveness claims. Accordingly, no circular step can be exhibited from the paper's own equations or citation chain, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The central claims rest on assumptions about the representativeness of a curated benchmark subset, the fairness of a unified prompt/hyperparameter setting, and the comparability of the acceleration experiments. LOOMBench is an introduced artifact with an external handle, not an unverifiable entity.

assumptions (3)
  • domain assumption LOOMBench's up-sampled subset of 12 benchmarks is representative of long-context model capability and preserves relative model ranking.
    The paper does not validate the subset against full-benchmark rankings or report sampling details (Appendix B.2), yet uses LOOMBench scores to rank models in Figure 5 and Table 5.
  • domain assumption A single or user-defined instruction template and unified hyperparameter settings do not systematically disadvantage any benchmark or model.
    Section 2.1 states the BENCHMARK module lets users define templates to 'eliminate performance discrepancies caused by prompt variations,' but no experiment checks whether the chosen templates preserve each benchmark's intended difficulty.
  • domain assumption Native Transformer with FlashAttention at batch size 1 is a fair baseline for comparing the latency of acceleration methods run at batch size 8.
    Section 3.2 explicitly states this batch size difference; it is a modeling choice in the experiment design that can inflate the reported speedups.
invented entities (1)
  • LOOMBench independent evidence
    purpose: A lightweight composite benchmark suite for evaluating six long-context capabilities in a few hours.
    LOOMBench is a new constructed dataset assembled from 12 existing benchmarks; its falsifiable handle is that it produces reproducible model scores that can be compared against full-benchmark scores. It is an artifact rather than a theoretical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework." pith.science (2026). https://pith.science/paper/L6GK5TS4

@misc{pith2026250704723,
  author       = {Pith},
  title        = {Pith review of: LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6GK5TS4}},
  note         = {Machine review of arXiv:2507.04723}
}
read the original abstract

Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmarks have been proposed. However, variations in evaluation settings across these benchmarks lead to inconsistent results, making it difficult to draw reliable comparisons. Besides, the high computational cost of long-context evaluation poses a significant barrier for the community to conduct comprehensive assessments of long-context models. In this paper, we propose LOOM-Scope, a comprehensive and efficient framework for long-context evaluation. LOOM-Scope standardizes evaluation settings across diverse benchmarks, supports deployment of efficient long-context inference acceleration methods, and introduces a holistic yet lightweight benchmark suite to evaluate models comprehensively. Homepage: https://loomscope.github.io

Figures

Figures reproduced from arXiv: 2507.04723 by the authors.

Figure 1
Figure 1. Workflow of LOOM-Scope framework. unlock unbounded external knowledge through the long context input (Ray, 2025). Alongside advancements in long-context language mod￾els (LCLMs), recent research in the long-context processing field has increasingly focused on two directions: (1) establishing various bench￾marks to evaluate model performance across diverse long-context tasks (Liu et al., 2025) and (2) improving effic… view at source ↗
Figure 2
Figure 2. Overview of LOOM-Scope, where the workflow of three modules can be refereed to Figure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of two low-code ways to start [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Partial snapshot of the local deployment WebUI of LOOM-Scope. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: RAG results for partial models. Full evalu [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Time cost of acceleration methods. compared to directly using LCLMs. In con￾trast, prediction based on the model-based method (Self-Route) can improve the perfor￾mance. More RAG evaluation results are shown in Appendix C ( [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Task distribution of LOOMBENCH. B.2 Statistic of LOOMBENCH LOOMBENCH is a composite benchmark con￾structed from 12 existing datasets, as indicated by the orange-highlighted entries in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 10
Figure 10. Figure 10: Complete RAG test results for mainstream models. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Complete augmentation methods test results for Llama-3.1-8B-Instruct models. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 28 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  5. [5]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024 a . Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

    arXiv 2024

  6. [6]

    Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024 b . Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055

    arXiv 2024

  7. [7]

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2024. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200

    arXiv 2024

  8. [8]

    OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

Show all 64 references
  1. [9]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get more with less: synthesizing recurrence with kv cache compression for efficient llm inference. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  2. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  3. [11]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv e-prints, pages arXiv--2312

    2023

  4. [12]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. https://api.semanticscholar.org/CorpusID:263609075 Model tells you what to discard: Adaptive kv cache compression for llms . In The Twelfth International Conference on Learning Representations,...

    2023 arXiv

  5. [13]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  6. [14]

    Raghavv Goel, Junyoung Park, Mukul Gagrani, Dalton Jones, Matthew Morse, Harper Langston, Mingu Lee, and Chris Lott. 2025. Caote: Kv caching through attention output error based token eviction. arXiv preprint arXiv:2504.14051

    2025

  7. [15]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752

    2023 arXiv

  8. [16]

    Gefei Gu, Yilun Zhao, Ruoxi Ning, Yanan Zheng, and Arman Cohan. 2024. https://aclanthology.org/2024.emnlp-demo.21 TAIL : A toolkit for automatic and realistic long-context large language model evaluation . In Proceedings of the 2024 Conference on Empirical Methods in Natural L...

    2024

  9. [17]

    Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. https://arxiv.org/abs/2404.07584 Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms . Preprint, arXiv:2404.07584

    2024 arXiv

  10. [18]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024 a . Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270--1303

    2024

  11. [19]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/028fcbcf85435d39a40c4d61b42c99a4-Paper-Conference.pdf Kvquant: Towards 10 million context ...

    2024

  12. [20]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  13. [21]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. https://openreview.net/forum?id=fPBACAbqSN MI nference 1.0: Accelerating pre-filling for long-context LLM ...

    2024

  14. [22]

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. https://proceedings.mlr.press/v262/kang24a.html GEAR : An efficient error reduction framework for KV cache compression in LLM inference . In Proceedings of The 4th NeurIPS Eff...

    2024

  15. [23]

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149

    2024 arXiv

  16. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  17. [25]

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. 2024. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations

    2024

  18. [26]

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, a...

    2024 arXiv

  19. [27]

    Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. 2024. Long context rag performance of large language models. arXiv preprint arXiv:2411.03538

    2024 arXiv

  20. [28]

    Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. 2024 a . A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.19442

    2024 arXiv

  21. [29]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/28ab418242603e0f7323e54185d19bde-Paper-Conference.pdf Snapkv: Llm knows what you...

    2024

  22. [30]

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024 c . Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. arXiv preprint arXiv:2407.16833

    2024 arXiv

  23. [31]

    Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089

    2025

  24. [32]

    Jerry Liu. 2022. https://doi.org/10.5281/zenodo.1234 LlamaIndex

    2022 doi

  25. [33]

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, and 1 others. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407

    2025

  26. [34]

    Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024 a . Longgenbench: Long-context generation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 865--883

    2024

  27. [35]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  28. [36]

    Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. 2024. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747

    2024 arXiv

  29. [37]

    AI Meta. 2024. Introducing llama 3.1: Our most capable models to date, 2024. URL https://ai. meta. com/blog/meta-llama-3-1/. New models including flagship 405B parameter model, along with upgraded 8B and 70B models featuring 128K context length and multilingual capabilities

    2024

  30. [38]

    AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025

    2025

  31. [39]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMN...

    2023

  32. [40]

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. https://openreview.net/forum?id=EQgEMAD4kv CAKE : Cascading and adaptive KV cache eviction with layer preferences . In The Thirteenth International Conference on Learning Repre...

    2025

  33. [41]

    Partha Pratim Ray. 2025. A survey on model context protocol: Architecture, state-of-the-art, challenges and future directions. Authorea Preprints

    2025

  34. [42]

    Jonathan Roberts, Kai Han, and Samuel Albanie. 2025. Needle threading: Can llms follow threads through near-million-scale haystacks? In The Thirteenth International Conference on Learning Representations

    2025

  35. [43]

    Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  36. [44]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. Preprint

    2024

  37. [45]

    Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2025. https://api.semanticscholar.org/CorpusID:278714866 Accurate kv cache quantization with outlier tokens tracing . In The 63rd Annual Meeting of the Association for ...

    2025

  38. [46]

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, and 1 others. 2025. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology

    2025

  39. [47]

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, and 1 others. 2024 a . Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. arXiv preprint arXiv:2406.17419

    2024 arXiv

  40. [48]

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, and 1 others. 2024 b . Needle in a multimodal haystack. Advances in Neural Information Processing Systems, 37:20540--20565

    2024

  41. [49]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024 c . https://api.semanticscholar.org/CorpusID:271097687 Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks . ArXiv, abs/2407.08454

    2024 arXiv

  42. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020

  43. [51]

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503.16428

    2025 arXiv

  44. [52]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv prepr...

    2025 arXiv

  45. [53]

    Songlin Yang and Yu Zhang. 2024. https://github.com/fla-org/flash-linear-attention Fla: A triton-based library for hardware-efficient implementations of linear attention mechanism

    2024

  46. [54]

    Dingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, and Weiping Wang. 2025. https://arxiv.org/abs/2505.19586 Tailorkv: A hybrid framework for long-context inference via tailored kv cache optimization . Preprint, arXiv:2505.19586

    2025 arXiv

  47. [55]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694

    2024 arXiv

  48. [56]

    Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao, Di Yin, Fang Li, Jie Wang, Zengxi Chen, Suncong Zheng, Xiaolong Liang, and Xing Sun. 2025. Sequential-niah: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts. arXiv preprint arXiv:2504.04713

    2025

  49. [57]

    Yijiong Yu, Ma Xiufa, Fang Jianwei, Zhi Xu, Su Guangyao, Wang Jiancheng, Yongfeng Huang, Zhixiao Qi, Wei Wang, Weifeng Liu, and 1 others. 2024. Hyper-multi-step: The truth behind difficult long-context tasks. arXiv preprint arXiv:2410.04422

    2024 arXiv

  50. [58]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, and 1 others. 2024. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136

    2024

  51. [59]

    Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. 2025 a . Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR)

    2025

  52. [60]

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. 2025 b . Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML)

    2025

  53. [61]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and 1 others. 2024. Infinite-bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Comp...

    2024

  54. [62]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: heavy-hitter oracle for efficient generative inference of large language models. In Proceedin...

    2023

  55. [63]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others. 2024. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processin...

    2024

  56. [64]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, and 1 others. 2024. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.