Pith. sign in

REVIEW 4 major objections 5 minor 39 references

NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When every sentence is relevant, LLMs fail to integrate even 200 tokens

desk verdict The NeedleChain benchmark is a genuinely useful new probe of all-relevant context integration, but the paper's headline failure numbers are inflated by a scoring rule that counts correct final answers as wrong when a name is omitted from a brief explanation. read the letter →

arxiv 2507.22411 v3 pith:RWQ27DDF submitted 2025-07-30 cs.CL cs.AI

Hyeonseok Moon , Heuiseok Lim This is my paper
classification cs.CLcs.AI
keywords long-contextcomprehensionintactcontextneedle-in-a-haystackchainreasoningROPEcontractionpositionbiasbenchmarkconstructionLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current long-context benchmarks overstate what LLMs understand, because they bury relevant facts in irrelevant text and therefore reward retrieval rather than integration. To test this, the authors build NeedleChain, a benchmark in which every sentence is needed: salary facts form a chain (e.g., 'A received $1600'; 'A earns twice as much as B'), and the only way to answer is to combine all of them. Across three orderings (forward, backward, mixed), models that score near 100% on a needle-in-a-haystack version of the same data fall sharply once the chain exceeds a few steps, with GPT-4o unreliable at roughly 200 tokens. The authors also propose ROPE contraction, increasing the rotary-position rotation angle at inference, which improves full-context integration. If right, the result means advertised context lengths say little about whether a model can use a short, information-dense passage as a whole.

What carries the argument

The load-bearing object is the NeedleChain benchmark, built from two kinds of 'needles': independent needles that fix one person's salary (e.g., 'A received $1600 last week') and dependent needles that relate salaries ('A earns twice/half/the same as B'). A chain of k−1 dependent needles plus one independent needle forms a single semantic unit, and the query asks for the salary at the end of the reasoning order, so omitting any needle makes the answer unreachable. The benchmark's three variants—forward, backward, and mixed—keep the same needles and only change the presented order, which isolates the effect of reasoning direction. A parallel NeedleStack benchmark concatenates independent needles, turning all but one into irrelevant 'haystack' content. The analysis tool that exposes positional weakness is a heatmap of which names appear in the model's output, used to separate presented position from reasoning position. The proposed intervention, ROPE contraction, increases the rotary position embedding rotation angle at inference (opposite of ROPE extension) to sharpen positional distinctions and reduce evidence omission.

What would settle it

Run the same backward-chain NeedleChain data rendered as a natural narrative (e.g., 'Cairo took home $1600, and Ramon makes twice Cairo's salary...') at k=20; if a model that fails the terse version scores high on the narrative version, the reported failure is an artifact of the benchmark's synthetic format rather than a general inability to integrate all-relevant context.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that LLMs possess an 'intact context-understanding length' far shorter than their nominal context window: when the context contains only query-relevant text, models such as GPT-4o, Llama3.3-70B, and the Qwen family fail to reliably integrate inputs as short as roughly 200 tokens (about 20 salary relations), even though the same models are essentially perfect at locating a single relevant fact in a haystack built from the same sentences. The failure is order-dependent: forward chains, where the needed reasoning follows the presented order, are handled well, but backward chains, requiring right-to-left integration, show the largest drop, larger than arbitrarily mixed order. Error analysis attributes the failures first to calculation errors and increasingly, as the chain lengthens, to omission of entire evidence units, with the omitted positions following the 'logical' middle of the reasoning path rather than the middle of the text. The authors further report that changing the question from a single target to the total salary collapses needle-in-a-haystack performance, that adding a code interpreter rescues that retrieval setting but not NeedleChain, and that ROPE contraction—using a larger rotation angle at inference than at training—significantly improves intact comprehension, while the standard YaRN extension strategy degrades it.

Load-bearing premise

The load-bearing premise is that performance on synthetic salary-ratio chains is a faithful measure of 'intact context comprehension': the paper assumes that when a model omits a name from its answer, the corresponding evidence was not used, and that these arithmetic chains stand in for realistic all-relevant contexts.

Editorial extensions

If this is right

  • Accuracy on needle-in-a-haystack style evaluations should not be read as evidence of context comprehension; a model can be perfect at retrieval while failing to integrate a 200-token all-relevant passage.
  • The order in which evidence is presented materially changes reasoning quality; structuring documents so that the logical dependencies run left-to-right should improve LLM performance at no training cost.
  • Context-length extension methods that weaken positional distinctions (e.g., RoPE extension) can actively hurt full-context integration, while sharper position signals (ROPE contraction) can help.
  • Benchmarks for 'long context' should include information-dense, all-relevant conditions, because retrieval-style tests overestimate practical ability.
  • Error growth with length is dominated by omission of entire evidence pieces, so improving integration may require attention or memory mechanisms that keep all items active, not better arithmetic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'logically lost in the middle' pattern suggests a testable extension: reordering evidence so that the logical chain is monotone in presented order should eliminate most of the backward-chain deficit, which could be checked on the same benchmark without retraining.
  • Because the benchmark uses only salary comparisons, a natural next test is whether the same collapse appears with other all-relevant relations (causal, temporal, spatial); if it does not, the finding may be narrower than 'context comprehension' and partly about numerical-chaining format.
  • The success of ROPE contraction hints that position-signal sharpening may trade off against retrieval of very long contexts; combining contraction for dense passages with extension for sparse long documents could be a productive research direction.
  • Tool incorporation failing to help NeedleChain suggests the bottleneck is not computation but evidence maintenance; probing hidden states during backward chains could test whether earlier needles are attended to but then overwritten.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NeedleChain, a synthetic benchmark for measuring whether LLMs integrate all query-relevant information in a context. Each instance is a chain of salary facts (one independent needle plus k-1 dependent needles), with three variants requiring forward, backward, or mixed reasoning order, and a parallel Needle-in-a-Haystack benchmark called NeedleStack built from the same needles. Experiments on Qwen, Llama, and GPT-4o show near-perfect accuracy on NeedleStack but substantial degradation on NeedleChain as k grows, especially for backward chains. The paper also proposes a training-free 'ROPE contraction' strategy, which increases the RoPE rotation angle at inference, and reports that it improves NeedleChain accuracy relative to YaRN extension. The central claim is that current LLMs fail to fully understand contexts even at short lengths when all context is relevant.

Significance. If the main result holds, it is an important qualification of long-context evaluation: retrieval-heavy benchmarks may overestimate holistic context comprehension. The controlled chain design, with the same name lists and needle compositions across the three variants and NeedleStack, is a useful contribution, and the authors state that they release the data generation code, which supports reproducibility. The comparison of forward, backward, and mixed orders gives a clean way to separate reasoning-direction effects from context-length effects. However, the benchmark's validity depends on two assumptions that are not adequately defended: that name-mention in the explanation is a faithful proxy for information use, and that synthetic salary-ratio chains represent 'intact context comprehension' in a way that supports the paper's broad conclusions. The reported ROPE contraction results are also under-specified. These issues are load-bearing for the abstract and Section 4 claims, so the manuscript needs substantial revision before the central claims can be accepted.

major comments (4)
  1. [§3.2, §3.3, Appendix B] The 'Needle Omission' error category conflates surface mention with information use. Appendix B explicitly states that a response is incorrect 'regardless of the correctness of the final answer' if a needle's name is absent from the generated text. However, the evaluation prompt in Table 3 only asks for 'a brief explanation' and does not require the model to restate every input name. A model that correctly derives the final answer while summarizing the chain without repeating each name would be scored as an omission error. Because needle omission is reported as the dominant error mode at large k and drives the position heatmap in §3.3 (which measures name-mention rates, not actual evidential dependence), the accuracy numbers in Table 1 and the conclusion that models 'fail to integrate' short contexts may be substantially overstated. Please re-score the data with final-answer correctness assessed independently of explanation completeness, or change the prompt to require an exhaustive derivation, and report both the original and re-scored results.
  2. [Abstract and Table 1] The abstract's claim that 'even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens' is not supported for all chain variants. At k=20 (approximately 0.2K tokens), Table 1 shows GPT-4o at 98.0% on the forward chain and 88.5% on the mixed chain, with the backward chain at 78.5%. The headline failure at 200 tokens is thus really a backward-chain phenomenon at that length. Please qualify the claim to state which chain direction fails, or identify the length at which all variants show reliable failure.
  3. [§4, §7] The construct validity of NeedleChain is assumed rather than demonstrated. Section 7 concedes that the benchmark uses only numeric salary needles, yet Section 4 generalizes to the broad conclusion that 'LLMs do not yet fully comprehend given contexts.' No external validation is provided, such as correlation with established long-context tasks (e.g., LongBench, RULER, or a naturalistic all-relevant document task) or a human baseline on the same chains. Without such evidence, the paper should either restrict its conclusions to the synthetic setting or add a validation study showing that NeedleChain performance tracks a broader notion of intact context comprehension.
  4. [§4, Figure 7] The ROPE contraction experiments are not sufficiently specified. The text says rotation angles of 2x and 4x were used and that contraction improves performance relative to YaRN, but it does not state which models and k values are shown in Figure 7, how the multipliers were selected, how many independent runs were performed, or whether the improvements are statistically significant. Since ROPE contraction is a central contribution, the experimental protocol and variability measures must be reported.
minor comments (5)
  1. [Section 2.1] Typo: 'This refers to a sentences' should be 'This refers to a sentence'.
  2. [Appendix B and Section 2.2] Terminology is inconsistent: the main text uses 'Mixed Chain' but the appendix table label says 'Chaotic Chain'. Please unify the terminology.
  3. [Section 4] The sentence 'The experimental results clearly support our argument' appears twice in Section 4; one occurrence should be removed.
  4. [Figure 2 caption] The caption 'Performance variation with respect to the domain composition of training data' appears to be mismatched with the figure content, which shows example chain compositions. Please correct the caption.
  5. [Throughout] 'ROPE' should be consistently spelled 'RoPE' (rotary position embedding), and 'Yarn' should be 'YaRN' when referring to the method of Peng et al.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation reduces to its inputs; the benchmark's name-mention scoring rule is a construct-validity assumption, not a circular step.

full rationale

NeedleChain is a new synthetic benchmark, and all central results are empirical measurements on that instrument rather than consequences of fitted equations. The ROPE contraction result is an inference-time intervention evaluated on the same benchmark; since the intervention is not tuned to the benchmark outputs and no parameter is fitted from the data, it does not reduce to its input. The only self-referential element is the benchmark's scoring convention (Appendix B): responses with a correct final answer but missing names are labeled needle-omission errors 'regardless of the correctness of the final answer,' and the position heatmap equates name absence with non-use. This is a substantive construct-validity threat to the generalization 'LLMs do not yet fully comprehend given contexts,' but it is an operational assumption, not a definitional equivalence or a fitted-input-called-prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation; consequently, the paper does not exhibit circular derivation.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the benchmark's construct validity and the targeted, hand-picked ROPE contraction multiplier. No external validation anchors these choices to broader capabilities.

free parameters (1)
  • ROPE contraction multiplier = 2x and 4x
    The rotation-angle scale factors are hand-selected in the toy experiment and tested only on NeedleChain; no tuning curve or selection criterion is given.
assumptions (3)
  • domain assumption Performance on synthetic salary-chain integration is a valid measure of intact context comprehension.
    The paper defines the construct via the benchmark and does not validate it against other long-context benchmarks or real-world tasks.
  • domain assumption A name missing from the model's output implies the corresponding needle was not used in reasoning.
    This underpins the needle-omission error category and the positional heatmap, but a model could use a fact without explicitly naming the person.
  • ad hoc to paper Increasing the RoPE rotation angle at inference sharpens positional distinctions without harming other capabilities.
    This is the hypothesized mechanism for ROPE contraction; the paper does not analyze attention maps or evaluate on standard benchmarks to check for side effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models." pith.science (2026). https://pith.science/paper/RWQ27DDF

@misc{pith2026250722411,
  author       = {Pith},
  title        = {Pith review of: NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWQ27DDF}},
  note         = {Machine review of arXiv:2507.22411}
}
read the original abstract

Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether models can faithfully incorporate all given evidence. NeedleChain includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-in-a-haystack(NIAH) paradigm. By comparing these variants, NeedleChain enables a more comprehensive assessment of context understanding. We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context.

Figures

Figures reproduced from arXiv: 2507.22411 by the authors.

Figure 1
Figure 1. Performance comparison between the NEEDLECHAIN (Backward chain) and its parallel NIAH paradigm benchmark (NeedleStack). Reported number of tokens were estimated with Qwen2.5 tok￾enizer. 2024). For example, Llama-2 (Touvron et al., 2023) supports a 4,096-token context window, whereas Llama-4 (Meta, 2025) reportedly scales to over one million tokens. These advances suggest markedly improved ability to model and utiliz… view at source ↗
Figure 2
Figure 2. Performance variation with respect to the domain composition of training data [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Error analysis on NEEDLECHAIN. We analyze errors in each category to determine which of the three predefined error types they fall into. • Instruction not Followed: This refers to in￾stances where the model fails to generate a response by not adhering to the given output format, or fails to determine the final answer. • Needle Omission This refers to cases when certain "needles" are omitted in generating final answe… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Heatmap to show the weaknesses for each position. Left-sided figures shows positional needle-missing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: We compare the accuracy of models for different types of questions: those requires understanding the tail [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: A simple diagram illustrating the concepts of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance variation derived by the ROPE contraction and extension methodologies [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 13 canonical work pages

  1. [1]

    L -eval: Instituting standardized evaluation for long context language models

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L -eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp....

    doi:10.18653/v1/2024.acl-long.776 2024

  2. [2]

    Why does the effective context length of llms fall short? arXiv preprint arXiv:2410.18745, 2024 b

    Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short? arXiv preprint arXiv:2410.18745, 2024 b

    arXiv 2024

  3. [3]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for...

    2024

  4. [4]

    Lost in the haystack: Smaller needles are more difficult for llms to find

    Owen Bianchi, Mathew J Koretsky, Maya Willey, Chelsea X Alvarado, Tanay Nayak, Adi Asija, Nicole Kuznetsov, Mike A Nalls, Faraz Faghri, and Daniel Khashabi. Lost in the haystack: Smaller needles are more difficult for llms to find. arXiv preprint arXiv:2505.18148, 2025

    arXiv 2025

  5. [5]

    Longrope: extending llm context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: extending llm context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 11091--11104, 2024

    work page 2024

  6. [6]

    Robinson, Keren Gu, Anna-Luisa Brakman, Pamela Mishkin, Meghan Shah, Johannes Heidecke, Lilian Weng, and Adam Tauman Kalai

    Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu, Anna-Luisa Brakman, Pamela Mishkin, Meghan Shah, Johannes Heidecke, Lilian Weng, and Adam Tauman Kalai. First-person fairness in chatbots. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TlAdgeoDTo

    work page 2025

  7. [7]

    A little goes a long way: Efficient long context training and inference with partial contexts

    Suyu Ge, Xihui Lin, Yunan Zhang, Jiawei Han, and Hao Peng. A little goes a long way: Efficient long context training and inference with partial contexts. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TrKRpaOk8y

    work page 2025

  8. [8]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 39 references
  1. [9]

    RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy

    2024

  2. [10]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  3. [11]

    Llm maybe longlm: Selfextend llm context window without tuning

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Selfextend llm context window without tuning. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 22099--22114, 2024

    2024

  4. [12]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (ed...

    2024

  5. [13]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p...

    2023

  6. [14]

    Summary of a haystack: A challenge to long-context llms and rag systems

    Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 9885--9903, 2024

    2024

  7. [15]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. L oo GLE : Can long-context language models understand long contexts? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024 doi

  8. [16]

    Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024 b

    Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024 b

    2024

  9. [17]

    MARIO : MA th reasoning with code interpreter output - a reproducible pipeline

    Minpeng Liao, Chengxi Li, Wei Luo, Wu Jing, and Kai Fan. MARIO : MA th reasoning with code interpreter output - a reproducible pipeline. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 905--924,...

    2024 doi

  10. [18]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    2024 arXiv

  11. [19]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024 b

    2024

  12. [20]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    AI Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4 0 (7): 0 2025, 2025

    2025

  13. [21]

    Ya RN : Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Ya RN : Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=wHBfxhZu1u

    2024

  14. [22]

    o ring, and Julius Tr \

    Thomas Schuster, Marian Lambert, Nico D \"o ring, and Julius Tr \"o gele. Needle-in-the-haystack testing llms with a complex reasoning task. In International Conference on Engineering Applications of Neural Networks, pp.\ 254--266. Springer, 2025

    2025

  15. [23]

    G eo C oder: Solving geometry problems by generating modular code through vision-language models

    Aditya Sharma, Aman Dalmia, Mehran Kazemi, Amal Zouaq, and Christopher Pal. G eo C oder: Solving geometry problems by generating modular code through vision-language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Lingui...

    2025 doi

  16. [24]

    Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models

    Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds....

    2025

  17. [25]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  18. [26]

    Qwq-32b: Embracing the power of reinforcement learning, 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, 2025

    2025

  19. [27]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  20. [28]

    Focused transformer: Contrastive training for context scaling

    Szymon Tworkowski, Konrad Staniszewski, Miko aj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mi o \'s . Focused transformer: Contrastive training for context scaling. Advances in neural information processing systems, 36: 0 42661--42688, 2023

    2023

  21. [29]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, ...

    2024

  22. [30]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  23. [31]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  24. [32]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    2025 arXiv

  25. [33]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383, 2025 b

    2025 arXiv

  26. [34]

    Sequential-niah: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts

    Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao, Di Yin, Fang Li, Jie Wang, Zengxi Chen, Suncong Zheng, Xiaolong Liang, and Xing Sun. Sequential-niah: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts. arXiv preprint arXiv:2504.04713, 2025

    2025

  27. [35]

    B ench: Extending long context evaluation beyond 100 K tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. B ench: Extending long context evaluation beyond 100 K tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62n...

    2024 doi

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  29. [37]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  30. [38]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  31. [39]

    ROPE Contraction

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.