Pith. sign in

REVIEW 1 major objections 5 minor 1 cited by

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

T0 review · 1 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard long-context benchmarks mostly measure a model's baseline ability, not its long-context capability; LongScore isolates the length-specific drop.

desk verdict A useful length-controllable benchmark, but LongScore is relative degradation, not the disentangled metric the paper claims. read the letter →

arxiv 2505.19293 v2 pith:NLMF3BJF submitted 2025-05-25 cs.CL cs.AIcs.LG

Wang Yang , Hongye Jin , Shaochen Zhong , Song Jiang , Qifan Wang , Vipin Chaudhary , Xiaotian Han This is my paper
classification cs.CLcs.AIcs.LG
keywords long-contextevaluationLLMbenchmarklength-controllablecontextLongScoremetricbaseabilityBenchcontext-windowrelativeperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard long-context benchmarks, the paper argues, mostly certify a model's ordinary short-context strength rather than its ability to handle longer inputs. A model that is simply better at Q&A or summarization tends to look better on LongBench-style tasks regardless of how it copes as context grows. To separate the two, the paper builds 100-LongBench, a length-controllable set of eight real and synthetic tasks, and proposes LongScore, the relative drop of a model's score at long lengths against its own average score at 2k, 4k, and 6k. Under that metric the rankings reverse: a model with a modest base score but a shallow drop, such as Llama 3.1-8B-Instruct, ranks above models with stronger short-context scores but steeper declines. If this is right, evaluations should report both numbers — how good a model is, and how much of that it keeps when the context grows.

What carries the argument

The object carrying the argument is a length-controllable benchmark paired with a relative metric. 100-LongBench generates contexts of a specified token budget by taking one task-relevant ground-truth article and padding it with shuffled distractor articles from the same domain, so the task stays realistic rather than becoming a bare synthetic needle-in-a-haystack. It covers four task families — key retrieval, information retrieval, information comprehension, and information summarization — with eight tasks total, and it adds a QA filtering step: any question a model can answer correctly without the context is removed, so prior knowledge cannot inflate long-context scores. LongScore is the metric that does the separation, normalizing the long-length score by the model's own short-context average and thereby measuring how much performance the model keeps, not how high its absolute level sits.

What would settle it

Recompute LongScore for the eight open models on 100-LongBench with Base Ability defined as the average over 1k, 2k, and 4k contexts instead of 2k, 4k, and 6k; if the model rankings change materially, the metric is measuring where the baseline is drawn rather than long-context retention alone.

Watch

Extended reading notes

Core claim

The paper's central claim is that raw accuracy on long-context benchmarks conflates two quantities: Base Ability, the model's score on the same task at short lengths (2k, 4k, and 6k tokens), and true long-context capability, the ability to retain performance as context grows. It defines LongScore at length $l$ as $\mathrm{LC}_l = (S_l - \mathrm{BaseAbility}) / \mathrm{BaseAbility}$, where Base Ability is the average score over the three short lengths. Measured this way, the relative decline from a model's own baseline replaces the absolute score as the quantity of interest. The paper shows that rankings under LongScore differ sharply from raw-score rankings: on 100-LongBench, Llama3.1-8B-Instruct moves to rank 1 while Qwen2.5-14B-Instruct, which has the highest Base Ability, drops to rank 4, and on Ruler the ordering of Llama3.1-70B and Yi-34B flips at the longest lengths. The metric also amplifies differences between context-extension methods, such as NTK versus PI, that raw accuracy leaves nearly indistinguishable.

Load-bearing premise

The method assumes that a model's average score at 2k, 4k, and 6k contexts is a clean, length-free measure of its base ability, so that the relative drop from that number isolates long-context skill; if short contexts already contain length effects, or if a task has a low base score that fluctuates, the LongScore ranking can be driven by the denominator rather than by genuine context handling.

Editorial extensions

If this is right

  • Under LongScore, model rankings on 100-LongBench differ sharply from raw-score rankings: Llama3.1-8B-Instruct moves to rank 1, while Qwen2.5-14B-Instruct, which has the highest Base Ability, falls to rank 4.
  • LongScore amplifies method differences that raw accuracy nearly hides, such as NTK versus PI and different RoPE theta ratios, while keeping the expected win direction.
  • The length-controllable design reveals that some models collapse only beyond a threshold: Qwen2.5 models score well at 8k–16k but drop steeply at 128k–256k.
  • The QA filtering step excludes samples answerable from memorized knowledge, so reported scores reflect use of the provided context rather than pretraining memory.
  • Length-controllable construction lets one benchmark serve models with different context windows, avoiding the problem of fixed-length datasets becoming outdated as windows grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the paper's logic, any fixed-length benchmark without a length sweep cannot locate a model's breakdown point; evaluation should report a score-length curve rather than a single average.
  • The denominator-normalization idea could transfer to other comparisons where a baseline skill confounds the quantity of interest — for example, dividing out a model's non-context QA ability when comparing retrieval-augmented pipelines — with the same caveat that the baseline must be unaffected by the manipulated variable.
  • A natural extension is per-task LongScore rather than an average across tasks, since a model may retain retrieval ability while losing synthesis ability as context grows, and averaging could hide task-specific breakdowns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes 100-LongBench, a length-controllable long-context benchmark built by concatenating real articles from existing datasets, with eight tasks spanning key retrieval, passage counting, QA, and summarization, plus a QA filtering mechanism intended to remove questions answerable from prior knowledge. The paper also introduces LongScore, defined as LC_l = (S_l - BaseAbility) / BaseAbility, where BaseAbility is the average score at 2k, 4k, and 6k tokens, and claims that this metric separates a model's baseline ability from its true long-context capability. Experiments report reliability checks across model families, comparisons of NTK vs. PI and other known method pairs, and rankings of open-source models on 100-LongBench and RULER.

Significance. If the claims were fully supported, the benchmark and metric would be a useful contribution to long-context evaluation: the length-controllable construction addresses a real weakness of fixed-length benchmarks, the QA filtering mechanism targets a known source of inflated QA scores, and the code is publicly released. The raw ranking results, such as Llama3.1-8B rising above Qwen2.5 models under LongScore, are interesting and potentially actionable. However, the central conceptual claim that LongScore disentangles baseline ability from long-context capability is an overstatement, and the current evidence does not establish that the metric is more than a relative-degradation score with a data-dependent denominator.

major comments (1)
  1. [Section 4.3, Tables 4 and 7] No error bars, confidence intervals, or statistical tests are provided for the average scores and LongScore values, even though N=100 samples per length is stated. Many of the ranking differences are small (e.g., Llama3.1-8B at -17.4 vs. Llama3.2-1B at -28.8, or Qwen2.5-7B at -30.6 vs. Qwen2.5-14B at -31.1), and with 100 samples per length these differences may be within sampling noise. The reliability validation in Section 4.1 and the effectiveness validation in Section 4.2 confirm only that the data and metric align with already-known trends (larger models do better; NTK beats PI); they do not provide evidence that LongScore itself is a reliable estimator of an independently defined long-context capability. The paper should add bootstrap confidence intervals or repeated-run variance, and should strengthen the validation against a ground-truth notion of long-context ability.
minor comments (5)
  1. [Table 4 caption] The caption says 'five models' but the table lists four models; the appendix Table 7 lists eight. Please correct the caption or unify the tables.
  2. [Table 4 and Figures 7-8] There are typos: 'Avg socre' should be 'Avg score', and the axis label 'Long-T ext Capacity' should be 'Long-Text Capacity'.
  3. [Section A.1] The sentence 'All results are in Section A.1' is self-referential and should instead point to Figures 9-12, which contain the actual results.
  4. [Abstract and Figure 8] The abstract uses 'v.s.'; it should be 'vs.' Also, Figure 8 includes a 256k column for some models, but Section 4.3 states that contexts are 2k, 4k, 6k, 8k, 16k, 32k, 64k, and 128k; please clarify whether 256k is part of the benchmark or only for selected models.
  5. [Related Work / References] The NIAH benchmark is mentioned multiple times but no citation is given; please add the original reference. Several other URLs are cited only as arXiv preprints without version numbers, which makes reproducibility harder.

Circularity Check

1 steps flagged · score 6.0 of 10

LongScore's claimed disentanglement of Base Ability is contradicted by its own definition: LC_l = (S_l - B)/B makes the metric a decreasing function of B for fixed S_l, so rankings are partially forced by the denominator.

  1. self definitional [Section 3.2, Eq. (2) and the paragraph after Eq. (2)]
    "LongScore separates the model's Base Ability from Long-context Capability. Our metric focuses on the relative improvement or decline at longer lengths and provides a more precise assessment of long-context capabilities without being influenced by the model's Base Ability. ... LCl = Sl − Base Ability / Base Ability (2)"

    The metric is defined as LC_l = (S_l - B)/B = S_l/B - 1, so Base Ability B appears explicitly in the denominator. For any fixed long-context score S_l, d(LC_l)/dB = -S_l/B^2 < 0: two models with identical raw long-context scores are ranked differently solely because their short-context averages differ. The central claim that LongScore is 'without being influenced by the model's Base Ability' is therefore false by construction, not derived from any validated decomposition. Table 4 makes the effect concrete: Qwen2.5-14B has B=59.1 and average long-context score 40.7, giving LC -31.1, while Llama3.1-8B has B=44.0 and average long-context score 36.3, giving LC -17.4.

full rationale

The benchmark construction itself is not circular: the length-controllable data generation, QA filtering, and comparisons against external benchmarks (RULER, HELMET) provide independent content, and the paper does not fit parameters to data. The circularity is concentrated in the metric's central claim. LongScore is defined as relative degradation from a Base Ability estimate, and the paper asserts that this 'disentangles' baseline ability from long-context capability without offering a model of the form S_l = B * C_l with C_l independent of B, nor any test of that independence. The definition directly contradicts the claim: LC_l = S_l/B - 1 is monotone decreasing in B for fixed S_l, so Base Ability influences the metric by construction. The Limitations passage concedes instability for low-base-ability models but does not acknowledge this denominator dependence. The effectiveness validation in Section 4.2 confirms known expected winners (NTK > PI, Gemini-Pro > Flash, larger RoPE ratio > smaller), which is a reasonable sanity check but not independent evidence for the disentanglement claim. Because the paper's headline contribution — a metric that separates baseline ability from true long-context capability — is undermined by the metric's own defining equation, the core claim is partially circular and the ranking shifts in Table 4 are partly forced by the denominator rather than by genuine long-context skill.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to optimize results, but the metric's baseline window and the QA filtering threshold are model and design choices that materially affect rankings. The central claim rests on the assumption that a relative drop from a short-context baseline isolates long-context ability.

free parameters (3)
  • Base Ability length window = 2k, 4k, 6k
    Chosen by hand as representative of pretraining context lengths; changing this window changes Base Ability and thus LongScore rankings (Section 3.2, Eq. 1).
  • QA prior-knowledge filtering threshold = not reported
    Samples where a no-context response score exceeds an unspecified threshold are excluded from QA tasks; threshold and scoring details are omitted (Section 3.1, Appendix A.2).
  • Number of samples per length = 100
    Each task-length cell uses 100 samples; no confidence intervals are reported, so ranking stability is unknown (Section 4.3).
assumptions (3)
  • domain assumption Short-length scores at 2k/4k/6k on the same tasks constitute a valid, model-specific baseline for Base Ability.
    Underlies Eq. (1)-(2); if short-context scores are not a stable baseline, the normalization in LongScore is not interpretable.
  • domain assumption Larger models from the same family should outperform smaller models on every task and length, which is used to validate benchmark reliability.
    Section 4.1 treats this expected trend as evidence the benchmark is reliable, but it does not specifically validate long-context measurement.
  • domain assumption Concatenating a ground-truth article with same-domain distractor articles produces 'real-life reflective' long-context tasks.
    Section 3.1 claims this mimics RAG-style concatenation; the assumption is that in-domain padding does not distort task validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?." pith.science (2026). https://pith.science/paper/NLMF3BJF

@misc{pith2026250519293,
  author       = {Pith},
  title        = {Pith review of: 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLMF3BJF}},
  note         = {Machine review of arXiv:2505.19293}
}
read the original abstract

Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.g., digesting a long-form document to find answers vs. directly asking an LLM about it. However, existing real-task-based long-context evaluation benchmarks have two major shortcomings. First, benchmarks like LongBench often do not provide proper metrics to separate long-context performance from the model's baseline ability, making cross-model comparison unclear. Second, such benchmarks are usually constructed with fixed input lengths, which limits their applicability across different models and fails to reveal when a model begins to break down. To address these issues, we introduce a length-controllable long-context benchmark and a novel metric that disentangles baseline knowledge from true long-context capabilities. Experiments demonstrate the superiority of our approach in effectively evaluating LLMs.

Figures

Figures reproduced from arXiv: 2505.19293 by the authors.

Figure 1
Figure 1. Illustration of LM-Infinite (Han et al., 2024), a long-context enhancement method’s performances on three LongBench tasks. The colored dashed lines represent the average score of each model on the corresponding task. The size of the markers corresponds to the proportion of each text length within the entire dataset. The larger the marker, the higher the proportion. The results exhibit significant variation across ta… view at source ↗
Figure 3
Figure 3. Illustration of the Data Generation Process for [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. One sample in Question Answering where models provide accurate answers regardless of context Base Ability. It refers to the model’s score when conducting short-context tasks. To estimate Base Ability, we sample N instances from short text lengths (like 2k, 4k, 6k). For each length, N/3 samples are selected, and the model’s average score across these lengths is computed: Base Ability = S2k + S4k + S6k 3 (1) where S∗k… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Verification of the reliability of -LongBench: results of two models of different sizes from the same LM family tree, showcasing their average scores in different tasks. These findings confirm a well-established trend: within the same series, larger models generally ou…
Figure 6
Figure 6. Figure 6: Results of four open-source models on all [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Results of eight open-source models on eight tasks are presented, with their scores calculated using [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Results of eight models on -LongBench by using LongScore metric. The gray shading indicates either anomalous models’ scores or cases where the model is unable to generate outputs for 256k-long contexts. Qwen 2.5 and Phi 3) on -LongBench. The evalua￾tion uses LongScore …
Figure 9
Figure 9. Figure 9: Illustration of NTK’s performances on three LongBench tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Illustration of PI’s performances on three LongBench tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Illustration of YaRN’s performances on three LongBench tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Illustration of Longlora’s performances on three LongBench tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Verification the reliability of -LongBench: results of two models of different sizes from the same LM family tree, showcasing their scores in different tasks across various context lengths. One color represents a specific task, with solid lines indicating larger model…
Figure 14
Figure 14. Figure 14: Results of eight open-source models on all [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

    cs.CL 2026-07 conditional novelty 8.0 of 10

    WILDTRACE evaluates long-context models on 481 natural multi-hop evidence trails from 214 real documents, with top systems at 75.3% and geometry-specific weaknesses.

Reference graph

Works this paper leans on

45 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018

    arXiv 2024

  6. [6]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088

    arXiv 2023

  7. [7]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  8. [8]

    Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. 2024. Claude 2.0 large language model: Tackling a real-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336

    work page 2024

Show all 45 references
  1. [9]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 a . Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  2. [10]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023 b . Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307

    2023 arXiv

  3. [11]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Yongqi Fan, Hongli Sun, Kui Xue, Xiaofan Zhang, Shaoting Zhang, and Tong Ruan. 2024. https://arxiv.org/abs/2406.15019 Medodyssey: A medical domain benchmark for long context evaluation up to 200k tokens . Preprint, arXiv:2406.15019

    2024 arXiv

  6. [14]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660

    2024

  7. [15]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024

  8. [16]

    Mourad Heddaya, Kyle MacMillan, Anup Malani, Hongyuan Mei, and Chenhao Tan. 2024. https://arxiv.org/abs/2501.00097 Casesumm: A large-scale dataset for long-context summarization from u.s. supreme court opinions . Preprint, arXiv:2501.00097

    2024 arXiv

  9. [17]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  10. [18]

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for llm training. arXiv preprint arXiv:2410.10989

    2024 arXiv

  11. [19]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839

    2023 arXiv

  12. [20]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. https://arxiv.org/abs/2401.01325 Llm maybe longlm: Self-extend llm context window without tuning . Preprint, arXiv:2401.01325

    2024 arXiv

  13. [21]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023 a . Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939

    2023 arXiv

  14. [22]

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2021. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120

    2021 arXiv

  15. [23]

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023 b . Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201

    2023 arXiv

  16. [24]

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. 2024. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875

    2024 arXiv

  17. [25]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 a . World model on million-length video and language with blockwise ringattention. CoRR

    2024

  18. [26]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 b . Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  19. [27]

    Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. 2024. A controlled study on long context extension and generalization in llms. arXiv preprint arXiv:2409.12181

    2024 arXiv

  20. [28]

    Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300

    2023 arXiv

  21. [29]

    Bowen Peng and Jeffrey Quesnelle. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    2023

  22. [30]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  23. [31]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024. https://arxiv.org/abs/2403.11802 Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models . Preprint, arXiv:2403.11802

    2024 arXiv

  24. [32]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  25. [33]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762

    2024 arXiv

  26. [34]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  27. [35]

    Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. https://arxiv.org/abs/2404.06480 Ada-leval: Evaluating long-context llms with length-adaptable benchmarks . Preprint, arXiv:2404.06480

    2024 arXiv

  28. [36]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  29. [37]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 a . Efficient streaming language models with attention sinks. arXiv

    2023

  30. [38]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023 b . Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453

    2023 arXiv

  31. [39]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039

    2023 arXiv

  32. [40]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025

    2023 arXiv

  33. [41]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  34. [42]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694

    2024 arXiv

  35. [43]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  36. [44]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136

    2024

  37. [45]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024. Bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.