Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

Lost in the Middle is a real effect, but only up to half a model's context window.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The relative-length framing cleanly resolves the LiM contradiction; the retrieval-prerequisite claim is correlation dressed as causation and should be softened. the 3 major comments →

arxiv 2508.07479 v1 pith:CGHB5RZ5 submitted 2025-08-10 cs.CL

Positional Biases Shift as Inputs Approach Context Window Limits

Blerta Veseli , Julian Chibane , Mariya Toneva , Alexander Koller This is my paper
classification cs.CL
keywords positional biasLost in the Middleprimacy biasrecency biasrelative input lengthcontext windowlong-context language modelsretrieval-reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the conflicting reports about the Lost-in-the-Middle effect come from a measurement choice: studies used absolute token counts instead of input length normalized by each model's context window. Measured relative to the window, the effect appears consistently in every tested model when inputs occupy up to 50% of available context; beyond that, the bias for information at the start collapses while the bias for the end survives, leaving a distance-based preference for information near the end. The paper further argues that successful retrieval is a prerequisite for reasoning on long inputs, and that the positional biases seen in reasoning are mostly inherited from retrieval. If correct, this gives researchers a simple normalization rule—compare positions at comparable fractions of the context window—and redirects long-context reasoning failures toward retrieval failures.

Core claim

The paper's central claim is that the inconsistent record on positional biases is an artifact of measuring input length in absolute token counts rather than relative to each model's context window. Across six instruction-tuned open-source LLMs with context windows from 8K to 128K tokens, on four controlled reasoning datasets, the Lost-in-the-Middle effect appears consistently whenever the input fills at most half the model's context window; beyond that threshold, primacy bias falls sharply, even dropping below middle-position accuracy for retrieval in some cases, while recency bias persists. The result is that the V-shaped LiM curve flattens and is replaced by a distance-based bias: the clos

What carries the argument

The central object is relative input length, $L_{\mathrm{rel}} = L_{\mathrm{input}} / L_{\mathrm{max}}$, the ratio of an input's token count to the model's declared context-window size, evaluated at seven levels from 6% to 100%. Around it the paper defines three intensity metrics: primacy bias intensity (first-position accuracy minus middle-position accuracy), recency bias intensity (last-position accuracy minus middle-position accuracy), and LiM intensity, the sum of the first two only when both are positive. The supporting object is a set of retrieval-reasoning minimal pairs: for each reasoning question a matched retrieval question targets exactly the single premise needed, so reasoning ac

Load-bearing premise

The causal reading of the retrieval-reasoning results rests on the assumption, stated only in the appendix, that every example in a dataset has the same logical structure and reasoning difficulty; if retrieval-easy instances are also reasoning-easy, the conditional-accuracy gap could be a composition effect rather than evidence that retrieval enables reasoning.

What would settle it

Run the same positional protocol on a model at $L_{\mathrm{rel}}$ values of 0.25, 0.5, and 0.9 while holding the absolute number of tokens between the relevant text and the end of the input constant: if accuracy tracks distance to the end, the distance-based bias is confirmed, and if it tracks $L_{\mathrm{rel}}$, the window-normalization claim is confirmed. For the retrieval-prerequisite claim, build difficulty-matched retrieval-reasoning pairs and test whether $P(RA=1|RT=0)$ stays measurably below $P(RA=1|RT=1)$ when reasoning complexity is held fixed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Positional-bias evaluations should report $L_{\mathrm{rel}}$ alongside token counts, because equal absolute lengths from different context windows are not comparable.
  • Prior conflicting results line up: studies below 50% of the window see LiM, while 100K-token studies mostly do not, because they are sampling different halves of the window.
  • Failures on long-context reasoning should be investigated as retrieval failures first; reasoning accuracy conditioned on successful retrieval is a cleaner measure of reasoning ability.
  • Long-context benchmarks need to include multiple relative lengths, since a single fixed length can conceal or fabricate a positional effect.
  • As inputs fill the window, distance from the end replaces first/middle/last as the organizing variable for accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the distance-based-bias result suggests that model internals track absolute distance to the end rather than ordinal position; a natural follow-up is probing attention or hidden-state dynamics as a function of remaining tokens.
  • Beyond the paper: if retrieval is truly a prerequisite, retrieve-then-think prompting should improve reasoning more at high $L_{\mathrm{rel}}$ than at low $L_{\mathrm{rel}}$—a direct, testable prediction of the inheritance claim.
  • Beyond the paper: the causal reading could be checked by constructing difficulty-matched pairs where retrieval fails for positional reasons rather than complexity reasons; if the conditional gap persists, the prerequisite claim strengthens.
  • Beyond the paper: the window-normalization rule may transfer to multimodal or recurrent models whose effective horizons differ from nominal context windows, but the paper only tests transformer-based LLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that positional biases in LLMs should be studied with input length normalized by each model's context window (Lrel), rather than with absolute token counts. Using four controlled synthetic datasets (MonoRel, PIR, RuleTaker, BoxTracker), six open-weight models with context windows from 8K to 128K, padding variants from prior work, and defined metrics PriMi, ReCi, and LiMi, the authors report that Lost in the Middle (LiM) appears consistently for Lrel ≤ 0.5 and fades beyond that point because primacy bias drops while recency bias persists, yielding a distance-based end preference. They additionally analyze retrieval and reasoning minimal pairs and claim that successful retrieval is a prerequisite for effective reasoning and that positional biases in reasoning are largely inherited from retrieval. The paper includes a real-world QA replication in Appendix A.2.1 and releases code.

Significance. If the relative-length finding holds, it would reconcile seemingly contradictory prior results (e.g., Liu et al. observing LiM at 6K tokens versus Zhang et al. not observing it at 100K tokens) and would provide a concrete methodological norm for future long-context evaluation. The study has notable strengths: controlled synthetic tasks that minimize reliance on in-weight knowledge, padding variants adopted from Levy et al. (2024), per-model and per-dataset results in appendices, a real-world QA check, and no fitted parameters in the core claims. The Lrel-based LiM pattern is credible and independently visible in the raw accuracy curves (Figures 3, 4, 13–16). However, the retrieval-reasoning 'prerequisite' claim is not established by the conditional-probability analysis; it requires either difficulty-matched instance controls or a substantial softening of the causal language. The paper is therefore promising but needs revision before the strongest claims can be accepted.

major comments (3)
  1. [Section 4.3, Table 4, Appendix A.4] The headline claim in Section 4.3 that 'successful retrieval is indeed a prerequisite for effective reasoning' is not supported by the conditional analysis. Tables 3–4 compare P(RA=1 | RT=1) with P(RA=1 | RT=0); this is observational conditioning, not an intervention, and the gap is equally consistent with retrieval-easy instances also being reasoning-easy instances. The defense offered in Appendix A.4 ('all examples within the same dataset follow the same logical structure and reasoning depth...') is asserted, not demonstrated, and the paper's own data strain it: Llama-3-70B on MonoRel has P(RA=1 | RT=0)=0.73, well above the chance floor of 0.5 that a strict prerequisite would imply, and the same quantity ranges from 0.15 to 0.73 across model-dataset cells. The paper should either match or stratify instance difficulty (e.g., by number of operations or by performance under retrieval-guar
  2. [Section 4.3, Figure 5] The claim that positional biases in reasoning are 'largely inherited from retrieval' is also not identified by conditioning on RT. Figure 5 plots P(RA=1 | RT=1) by position; if RT is a mediator of the position→RA path, conditioning on RT=1 blocks the very indirect effect the paper wants to measure, so flattened curves are expected even when reasoning itself has position effects. Conversely, if RT and RA share latent difficulty, conditioning on RT=1 selects a difficulty subset and can flatten any position curve. A mediation analysis, or a design where retrieval success is fixed by construction, is needed before inheritance can be concluded.
  3. [Eq. (2); Sections 4.1–4.2] LiMi is defined as a rectified sum that is set to 0 whenever either edge accuracy does not exceed the middle. Thus part of the message that 'the LiM effect disappears because primacy bias drops' is built into the metric: once PriMi goes non-positive, LiMi must be 0 even if recency is large. The raw curves in Figure 4 independently show the phenomenon, so the substantive claim is plausible, but the paper should separate the metric-defined quantity from the empirical accuracy pattern. Relatedly, the claim in Section 4.2 that primacy bias 'fades significantly' lacks significance tests; with 100 instances per dataset and six models, paired tests or confidence intervals across the seven Lrel levels would allow a quantitative check of the Lrel ≈ 0.5 crossover.
minor comments (6)
  1. [Introduction, Section 3] Typos: 'crutial' should be 'crucial' (fourth finding, Introduction); 'context widow' should be 'context window' (Introduction, third finding); 'adress' should be 'address' (start of Section 3).
  2. [Figure 5 caption] The caption says the right panel shows P(RA=1 | RT=1) but the surrounding text describes the right panel as P(RA=1 | RT=0). The right panel label is also inconsistent with the left-panel formula in the same sentence.
  3. [Table 4] The increase column is formatted inconsistently: most entries are percentages (e.g., 52, 42) but the Mistral-Small-24B MonoRel row reports 0.69 as if it were a proportion. The column header '∼Increase (%)' is also malformed.
  4. [Section 3.4] The sentence 'We use only the instruction-tuned versions of these models.' is repeated verbatim in the same paragraph.
  5. [Table 1 / Appendix A.1] In the BoxTracker false example, the name 'Hanna' appears in Table 1 but 'Hannah' appears in the full example in Appendix A.1; the name should be consistent.
  6. [Appendix A.2.1] Typo: 'ntheir' should be 'their'. Also, several reference titles contain spurious spaces (e.g., 'T ransactions', 'T ask'), likely from PDF extraction; these should be cleaned.

Circularity Check

2 steps flagged

Partial definitional coupling in the LiMi metric and the retrieval-reasoning minimal pairs, but the main Lrel finding is supported by raw accuracy curves.

specific steps
  1. self definitional [Section 3.3, Eq. (2); Section 4.1, paragraph 'Given LiMi’s definition']
    "LiMi = (accF − accM) + (accL − accM), if accF > accM and accL > accM, 0, otherwise. ... Since PriMi drops below zero, the fading LiM effect likely actually reflects a loss of primacy bias."

    By Eq. (2), LiMi is identically zero whenever accF ≤ accM (i.e., PriMi ≤ 0) or accL ≤ accM. Therefore the paper's explanation that LiM fades 'because primacy bias drops below zero' is a direct consequence of the metric's zeroing rule, not an independent empirical discovery: the moment PriMi becomes negative, the first condition in Eq. (2) fails and LiMi is set to 0 by construction. The raw accuracy curves in Fig. 4 do show first-position decline independently, so the effect is not fabricated; but the specific attribution that the LiM disappearance is 'driven by' the primacy drop is entailed by the definition of LiMi whenever the primacy term turns negative.

  2. self definitional [Section 3.1; Section 4.3, Evaluation paragraph]
    "Each retrieval question targets a specific piece of information required for the reasoning process. ... If successful retrieval is indeed a required precursor of successful reasoning, we expect a significant and constant performance increase of reasoning in case of sucessful retrieval, i.e. P(RA = 1 | RT = 1) > P(RA = 1 | RT = 0)."

    The retrieval question is, by construction, the atomic fact that the reasoning question requires ('targets the specific piece of information required'). Hence 'successful retrieval is a prerequisite for reasoning' is the dataset-design assumption restated as a conclusion: a reasoning answer that is faithful to the premises already entails having the targeted fact available. Because P(RA=1|RT=0) ranges up to 0.73 (Table 4), the entailment is empirically violated by guessing/other paths, so the conditional gap is not a mathematical identity; the causal reading additionally depends on the unverified equal-complexity assumption in Appendix A.4. This is definitional coupling plus confound, not a forced derivation.

full rationale

This is a measurement study with no fitted parameters, so the central quantitative claims are not predictions from a fitted model. The main Lrel finding (LiM strongest at Lrel ≤ 0.5, primacy fading beyond) is backed by raw position-specific accuracy curves (Figures 3–4) and by the real-world QA check (Appendix A.2.1), which are independent of the LiMi metric's zeroing rule. However, two components have definitional coupling. First, Eq. (2) hard-codes LiMi = 0 whenever either edge accuracy does not exceed the middle; the paper's statement that the LiM effect disappears 'because primacy bias drops below zero' is partly a consequence of this definition, as the paper itself acknowledges ('Given LiMi’s definition'). Second, the retrieval-reasoning minimal pairs define retrieval as the atomic sub-question of reasoning, so the 'prerequisite' conclusion is partly embedded in the construction; the causal interpretation additionally relies on the equal-complexity assumption stated in Appendix A.4 ('all examples within the same dataset follow the same logical structure and reasoning depth'), which is unverified and strained by Table 4's P(RA=1|RT=0)=0.73 for Llama-3-70B on MonoRel. This is a confound/limitation rather than a circular derivation. The only self-citation (Veseli et al. 2023, used for long-tail QA questions in Appendix A.2.1) is not load-bearing. No uniqueness theorems are imported from the authors. Overall, the central Lrel result has independent content, so the circularity score is moderate rather than high.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper is an empirical measurement study: it introduces no fitted parameters in the derivation sense, but the central claims rest on hand-chosen design values (Lrel grid, middle placement, 100 instances per dataset), on vendor-declared context sizes as the normalization denominator, on position-neutral padding, and on an untested equal-complexity assumption that carries the causal 'prerequisite' claim. The bias metrics PriMi, ReCi, and LiMi are definitions, not new entities; no new forces, mechanisms, or conserved quantities are postulated.

free parameters (4)
  • Lrel grid = {0.06, 0.12, 0.25, 0.38, 0.5, 0.75, 1.0}
    Hand-chosen relative lengths aligned to Levy et al. (2024) absolute lengths (Section 3.2); grid resolution determines how precisely the 0.5 crossover can be located.
  • Observed crossover threshold = Lrel ≈ 0.5 (by inspection of averaged curves, Fig. 3)
    The central quantitative claim that LiM is strongest up to 50% of the context window is read off averaged curves without a fitted breakpoint model, a confidence interval, or a significance test.
  • Middle position placement = not precisely specified
    The exact location of the relevant text when labeled 'middle' is not defined in the main text; exact placement directly determines acc_M and hence all three bias metrics (Eqs. 2-4).
  • Instances per dataset = 100 per dataset
    Each dataset contains 100 instances (Section 3.1), which puts the standard error of an accuracy cell near five points and limits the resolution of per-model, per-position differences shown in Figure 4.
axioms (5)
  • domain assumption The advertised context window Lmax is the correct normalization denominator (Eq. 1)
    All results are stated in units of vendor-declared context sizes (8K, 32K, 128K). If a model's effective capacity is smaller than advertised, Lrel is miscalibrated and the observed 0.5 crossover shifts or vanishes.
  • domain assumption Distractor padding text is position-neutral and content-neutral with respect to the query
    Inputs are extended to target Lrel using Levy et al. (2024) style narrative padding (Section 3.2); any correlation between distractor content and position would be attributed to position.
  • domain assumption Within-dataset base instances have equal complexity, making the conditioning analysis causally interpretable
    Appendix A.4 states 'all examples within the same dataset follow the same logical structure and reasoning depth'; this is the basis for reading P(RA=1|RT=1) > P(RA=1|RT=0) as retrieval supporting reasoning rather than instance difficulty.
  • domain assumption Instruction-tuned checkpoints represent LLM positional behavior
    Section 3.4 restricts evaluation to instruction-tuned models and selects models with above-chance base accuracy; results may not transfer to base or continued-pretraining variants.
  • domain assumption The metric definitions in Eqs. 2-4 operationalize the phenomena (LiM requires both primacy and recency to beat the middle)
    LiMi is forced to zero unless acc_F > acc_M and acc_L > acc_M (Section 3.3), so conclusions about 'LiM disappearing' are partly inherited from this definitional choice.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Positional Biases Shift as Inputs Approach Context Window Limits." pith.science (2026). https://pith.science/paper/CGHB5RZ5

@misc{pith2026250807479,
  author       = {Pith},
  title        = {Pith review of: Positional Biases Shift as Inputs Approach Context Window Limits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGHB5RZ5}},
  note         = {Machine review of arXiv:2508.07479}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) often struggle to use information across long inputs effectively. Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears at the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest. To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model's context window. Our findings reveal that the LiM effect is strongest when inputs occupy up to 50% of a model's context window. Beyond that, the primacy bias weakens, while recency bias remains relatively stable. This effectively eliminates the LiM effect; instead, we observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input. Furthermore, our results suggest that successful retrieval is a prerequisite for reasoning in LLMs, and that the observed positional biases in reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoning

    cs.AI 2026-03 unverdicted novelty 7.0

    LEAD lets LLMs solve checkers jumping puzzles up to size 13 by using lookahead to recover from irreversible errors on hard steps that break extreme decomposition.

  2. Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG

    cs.AI 2026-07 conditional novelty 6.0

    On a new sibling-table benchmark, dense retrievers find the right table group but identify the uniquely answerable table at near-chance rates; cross-encoder reranking lifts top-1 from 18.2% to 57.4%.

  3. An Evaluation of Chat Safety Moderations in Roblox

    cs.CY 2026-05 unverdicted novelty 5.0

    Roblox's chat moderation system allows many unsafe messages involving grooming, sexualizing minors, bullying, harassment, violence, self-harm, and sharing sensitive information to go undetected, with users using evasi...

  4. An Evaluation of Chat Safety Moderations in Roblox

    cs.CY 2026-05 unverdicted novelty 5.0

    Roblox's automated chat moderation fails to catch numerous unsafe messages involving grooming, sexualization of minors, bullying, violence, self-harm, and sensitive information sharing, with users evading detection th...

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages · cited by 3 Pith papers · 1 internal anchor

  1. [1]

    15 Published as a conference paper at COLM 2025 ���������������������������������������������������������������������������������������� ��������������������������

    Figure 9: BoxTracker. 15 Published as a conference paper at COLM 2025 ���������������������������������������������������������������������������������������� ��������������������������. Dave, being small, often finds himself having to navigate through a world that is not always accommodating to his size. Whether it's struggling to reach items on high she...

    work page 2025

  2. [2]

    URL https://aclanthology

    Association for Computational Linguistics. URL https://aclanthology. org/2024.acl-long.172/. Arie Cattan, Alon Jacovi, Alex Fabrikant, Jonathan Herzig, Roee Aharoni, Hannah Rashkin, Dror Marcus, Avinatan Hassidim, Yossi Matias, Idan Szpektor, and Avi Caciularu. Can few-shot work in long-context? recycling the context to generate demonstrations,

    work page 2024

  3. [3]

    DoubleDipper: Improving Long-Context LLMs via Context Recycling

    URL https://arxiv.org/abs/2406.13632. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The 12th International Conference on Learning Representations,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg

    URL https://arxiv.org/pdf/2310.00785.pdf. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? In Proceedings of the 1st Conference on Language Modeling (COLM) ,

    Pith/arXiv arXiv

  5. [5]

    Maor Ivgi, Uri Shaham, and Jonathan Berant

    URL https://arxiv.org/abs/2404.06654. Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short- text models. T ransactions of the Association for Computational Linguistics, 11:284–299,

    Pith/arXiv arXiv

  6. [7]

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer

    Accessed: 2025-07-09. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A “novel” challenge for long-context language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pp. 17048–17085, Miami, Flor...

    work page 2025

  7. [8]

    URL https://aclanthology

    Association for Computational Linguistics. URL https://aclanthology. org/2024.emnlp-main.948/. Dominique Kelly, Yimin Chen, Sarah E. Cornwell, Nicole S. Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L. Rubin. Bing chat: The future of search engines? Proceedings of the Association for Information Science and T echnology, 60(1):1007–1009, October

    work page 2024

  8. [9]

    Najoung Kim and Sebastian Schuster

    URL https://dl.acm.org/doi/abs/10.1002/pra2.927. Najoung Kim and Sebastian Schuster. Entity tracking in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3835–3855, Toronto, Canada, July

    doi:10.1002/pra2.927

  9. [10]

    URL https: //aclanthology.org/2023.acl-long.213/

    Association for Computational Linguistics. URL https: //aclanthology.org/2023.acl-long.213/. 11 Published as a conference paper at COLM 2025 Tom´aˇs Koˇcisk´y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G´abor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. T ransactions of the Association for Computa...

    work page 2023

  10. [13]

    Mosh Levy, Alon Jacoby, and Yoav Goldberg

    URL https://arxiv.org/abs/2406.10149. Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ,...

    Pith/arXiv arXiv

  11. [14]

    URL https: //aclanthology.org/2024.acl-long.818/

    Association for Computational Linguistics. URL https: //aclanthology.org/2024.acl-long.818/. Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, and Yixuan Su. Alr2: A retrieve-then-reason framework for long-context question answering, 2024a. URL https://arxiv.org/abs/2410.03227. Yanyang Li, Shuo Liang, Michae...

    Pith/arXiv arXiv 2024

  12. [15]

    Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A

    URL https: //aclanthology.org/2024.tacl-1.9/. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schuetze. Nolima: Long-context evaluation beyond literal matching. In 42nd International Conference on Machine Learning ,

    work page 2024

  13. [17]

    URL https://arxiv.org/abs/2307.08177

    Association for Computing Machinery. URL https://arxiv.org/abs/2307.08177. Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. Eliciting in-context retrieval and reasoning for long-context large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Associa...

    Pith/arXiv arXiv 2025

  14. [18]

    URL https:// aclanthology.org/2025.findings-acl.165/

    Association for Computational Linguistics. URL https:// aclanthology.org/2025.findings-acl.165/. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. ...

    work page 2025

  15. [19]

    URL https://aclanthology.org/2023.findings-emnlp.536/

    Association for Computational Linguistics. URL https://aclanthology.org/2023.findings-emnlp.536/. 12 Published as a conference paper at COLM 2025 Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars: A multi-evidence, position- aware, and scalable benchmark for evaluating long-context large language models. In Owen Rambow, Leo Wanner, Marianna Apidianak...

    work page 2023

  16. [20]

    URL https://aclanthology.org/2025.coling-main.253/

    Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.253/. Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. Source code summarization in the era of large language models. In Proceedings of the 47th International Conference on Software Engineering (ICSE) ,

    work page 2025

  17. [21]

    Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, and Xiaojiang Liu

    URL https://arxiv.org/abs/2407.07959. Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, and Xiaojiang Liu. Distance between relevant information pieces causes bias in long-context LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar...

    Pith/arXiv arXiv 2025

  18. [22]

    URL https:// aclanthology.org/2025.findings-acl.28/

    Association for Computational Linguistics. URL https:// aclanthology.org/2025.findings-acl.28/. Blerta Veseli, Simon Razniewski, Sneha Singhania, and Gerhard Weikum. Evaluating language models for knowledge base completion. In20th Extended Semantic Web Conference (ESWC), 2023,

    work page 2025

  19. [23]

    Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen

    URL https://arxiv.org/abs/2303.11082. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Hum...

    Pith/arXiv arXiv 2024

  20. [24]

    URL https://aclanthology.org/2024.naacl-long.205/

    Association for Computational Linguistics. URL https://aclanthology.org/2024.naacl-long.205/. Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. Primacy effect of ChatGPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 108–115, Singapore, December

    work page 2024

  21. [25]

    URL https://aclanthology

    Association for Computational Linguistics. URL https://aclanthology. org/2023.emnlp-main.8/. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad...

    work page 2023

  22. [26]

    Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie

    URL https://proceedings.neurips.cc/paper files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf. Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the emergence of position bias in transformers. In 42nd International Conference on Machine Learning ,

    work page 2022

  23. [27]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro

    URL https://arxiv.org/abs/2502.01951i. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The 12th International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ,

    Pith/arXiv arXiv 2024

  24. [28]

    org/abs/2310.03025

    URL https://arxiv. org/abs/2310.03025. Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogi- cal reasoners. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning , volume 2024, pp. 17019–17045,

    Pith/arXiv arXiv 2024

  25. [29]

    URL https://proceedings.iclr.cc/paper files/paper/2024/file/ 4990dad2c1696224de42573d0222554a-Paper-Conference.pdf. 13 Published as a conference paper at COLM 2025 Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. Lv- eval: A balanced long-context ben...

    work page 2024

  26. [30]

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B

    URL https://arxiv.org/abs/2402.05136. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summariza- tion. T ransactions of the Association for Computational Linguistics , 12:39–57, 2024a. URL https://aclanthology.org/2024.tacl-1.3/. Xinrong Zhang, Yingfa Chen, ...

    arXiv 2024

  27. [31]

    URL https://aclanthology.org/2023.findings-acl.167/

    Association for Computational Linguistics. URL https://aclanthology.org/2023.findings-acl.167/. 14 Published as a conference paper at COLM 2025 A Appendix A.1 Full-Length Examples of Base Instances The datasets MonoRel, PIR, and Simplified RuleTaker are taken from Levy et al. (2024), while BoxTracker is generated using the framework proposed by Kim & Schu...

    work page 2023

  28. [33]

    This preselection ensures that the answer depends on retrieval rather than memorized knowledge from pre-training, mirroring the controlled con- ditions of our synthetic datasets

    that a set of models (Gemma-2-27B, Llama- 3-70B, Mistral) fail to answer without supporting evidence but succeed with access to the corresponding Wikipedia page. This preselection ensures that the answer depends on retrieval rather than memorized knowledge from pre-training, mirroring the controlled con- ditions of our synthetic datasets. Each instance co...

    work page 2024

  29. [2018]

    Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev

    URL https: //aclanthology.org/Q18-1023/. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),Findings of the Association for Computational Linguistics: EMNLP 2022 , pp. 6536–6558, Abu Dhab...

    work page 2022

  30. [2022]

    URL https://aclanthology

    Association for Computational Linguistics. URL https://aclanthology. org/2022.findings-emnlp.488/. Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a- haystack. Advances in Neural Information Processing Systems , 37:106519–106554,

    work page 2022

  31. [2023]

    URL https://aclanthology.org/2023.tacl-1.17/. G. Kamradt. Needle in a haystack - llm test. https://github.com/gkamradt/LLMTest NeedleInAHaystack,

    work page 2023

  32. [2024]

    URL https://aclanthology.org/2024.acl-long.776/

    Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.776/. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins,...

    work page 2024

  33. [2025]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers

    URL https: //arxiv.org/abs/2502.05167. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ICSE ’24, New York, NY, USA,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.