{"id":"005a6931-a920-45b4-8108-aadcda62996e","arxiv_id":"2508.07479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.","lead":"Large language models are more likely to miss information placed in the middle of a long input, a known weakness, but this study shows the weakness appears mainly when inputs use up to half of a model's context window. The result reconciles conflicting earlier findings and implies benchmarks should measure input length relative to each model's context size.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval-reasoning 'prerequisite' claim is not established: Table 4's conditional gap is confounded by unmeasured reasoning difficulty, and P(RA=1|RT=0)=0.73 contradicts a strict prerequisite.","rationale":"The reader's weakest_assumption correctly identifies the equal-complexity assumption as load-bearing for the causal reading of the retrieval-reasoning analysis. I agree that this is the single most important concern: the headline 'prerequisite' claim rides entirely on the gap between two conditional probabilities, and that gap is confounded whenever retrieval difficulty and reasoning difficulty covary. The paper's own Table 4 provides a direct red flag: P(RA=1|RT=0)=0.73 for Llama-3-70B on MonoRel, which is far above the chance level expected if retrieval failure blocks reasoning. The authors' assertion of equal complexity in Appendix A.4 is an untested assumption, not a demonstrated control. Other issues — the absence of significance tests and the definitional clamping of LiMi in Eq. 2 — are real but secondary: the LiM/Lrel finding is supported by raw accuracy curves and a real-world replication, and the LiMi definition faithfully encodes the stated U-shape criterion. The causal retrieval claim, by contrast, would require either an intervention or a convincing difficulty-matching argument; neither is present. Therefore the reader's CONDITIONAL verdict is appropriate: the first contribution can stand with minor caveats, but the 'prerequisite' language must be softened or supported by additional evidence. My recommendation is UNCHANGED relative to the reader's verdict, because the concern is already incorporated in the CONDITIONAL rating.","tokens_in":20004,"tokens_out":8881,"duration_ms":102285,"concrete_test":"Run an oracle-retrieval intervention on the RT=0 subset: for every instance where the model answered the retrieval question incorrectly, append the correct retrieval answer (the relevant premise) to the input immediately before the reasoning question and re-measure reasoning accuracy. If P(RA=1|RT=0) rises to the P(RA=1|RT=1) level, retrieval failure is a causal bottleneck; if it stays near the original P(RA=1|RT=0), the Table 4 gap is a difficulty-composition effect and the 'prerequisite' claim should be dropped or softened to correlation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 concludes that 'successful retrieval is a prerequisite for effective reasoning' from the gap between P(RA=1|RT=1) and P(RA=1|RT=0) (Tables 3–4). This is observational conditioning, not an intervention; the gap is equally consistent with retrieval-easy instances also being reasoning-easy instances. The only defense is the equal-complexity assumption stated in Appendix A.4 and repeated in the Table 4 caption: 'all examples within the same dataset follow the same logical structure and reasoning depth, with identical distributions of operations and entities.' No evidence supports this, and the paper's own data strain it: Llama-3-70B on MonoRel has P(RA=1|RT=0)=0.73, far above the 0.5 chance floor that a true prerequisite plus random guessing would permit. If retrieval failure truly precluded reasoning, P(RA=1|RT=0) should be at or below chance; values well above chance show reasoning can succeed despite retrieval failure, so the 'prerequisite' claim collapses to correlation unless difficulty is shown to be matched. The flattening of P(RA=1|RT=1) in Figure 5 could likewise be a regression artifact. This is the most load-bearing weakness because it targets a headline claim, whereas the LiM/Lrel finding has independent support from Figures 3–4 and the real-world QA check in Appendix A.2.1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that positional biases in LLMs should be studied with input length normalized by each model's context window (Lrel), rather than with absolute token counts. Using four controlled synthetic datasets (MonoRel, PIR, RuleTaker, BoxTracker), six open-weight models with context windows from 8K to 128K, padding variants from prior work, and defined metrics PriMi, ReCi, and LiMi, the authors report that Lost in the Middle (LiM) appears consistently for Lrel ≤ 0.5 and fades beyond that point because primacy bias drops while recency bias persists, yielding a distance-based end preference. They additionally analyze retrieval and reasoning minimal pairs and claim that successful retrieval is a prerequisite for effective reasoning and that positional biases in reasoning are largely inherited from retrieval. The paper includes a real-world QA replication in Appendix A.2.1 and releases code.","tokens_in":20247,"tokens_out":5785,"duration_ms":61364,"significance":"If the relative-length finding holds, it would reconcile seemingly contradictory prior results (e.g., Liu et al. observing LiM at 6K tokens versus Zhang et al. not observing it at 100K tokens) and would provide a concrete methodological norm for future long-context evaluation. The study has notable strengths: controlled synthetic tasks that minimize reliance on in-weight knowledge, padding variants adopted from Levy et al. (2024), per-model and per-dataset results in appendices, a real-world QA check, and no fitted parameters in the core claims. The Lrel-based LiM pattern is credible and independently visible in the raw accuracy curves (Figures 3, 4, 13–16). However, the retrieval-reasoning 'prerequisite' claim is not established by the conditional-probability analysis; it requires either difficulty-matched instance controls or a substantial softening of the causal language. The paper is therefore promising but needs revision before the strongest claims can be accepted.","major_comments":[{"comment":"The headline claim in Section 4.3 that 'successful retrieval is indeed a prerequisite for effective reasoning' is not supported by the conditional analysis. Tables 3–4 compare P(RA=1 | RT=1) with P(RA=1 | RT=0); this is observational conditioning, not an intervention, and the gap is equally consistent with retrieval-easy instances also being reasoning-easy instances. The defense offered in Appendix A.4 ('all examples within the same dataset follow the same logical structure and reasoning depth...') is asserted, not demonstrated, and the paper's own data strain it: Llama-3-70B on MonoRel has P(RA=1 | RT=0)=0.73, well above the chance floor of 0.5 that a strict prerequisite would imply, and the same quantity ranges from 0.15 to 0.73 across model-dataset cells. The paper should either match or stratify instance difficulty (e.g., by number of operations or by performance under retrieval-guar","section":"Section 4.3, Table 4, Appendix A.4"},{"comment":"The claim that positional biases in reasoning are 'largely inherited from retrieval' is also not identified by conditioning on RT. Figure 5 plots P(RA=1 | RT=1) by position; if RT is a mediator of the position→RA path, conditioning on RT=1 blocks the very indirect effect the paper wants to measure, so flattened curves are expected even when reasoning itself has position effects. Conversely, if RT and RA share latent difficulty, conditioning on RT=1 selects a difficulty subset and can flatten any position curve. A mediation analysis, or a design where retrieval success is fixed by construction, is needed before inheritance can be concluded.","section":"Section 4.3, Figure 5"},{"comment":"LiMi is defined as a rectified sum that is set to 0 whenever either edge accuracy does not exceed the middle. Thus part of the message that 'the LiM effect disappears because primacy bias drops' is built into the metric: once PriMi goes non-positive, LiMi must be 0 even if recency is large. The raw curves in Figure 4 independently show the phenomenon, so the substantive claim is plausible, but the paper should separate the metric-defined quantity from the empirical accuracy pattern. Relatedly, the claim in Section 4.2 that primacy bias 'fades significantly' lacks significance tests; with 100 instances per dataset and six models, paired tests or confidence intervals across the seven Lrel levels would allow a quantitative check of the Lrel ≈ 0.5 crossover.","section":"Eq. (2); Sections 4.1–4.2"}],"minor_comments":[{"comment":"Typos: 'crutial' should be 'crucial' (fourth finding, Introduction); 'context widow' should be 'context window' (Introduction, third finding); 'adress' should be 'address' (start of Section 3).","section":"Introduction, Section 3"},{"comment":"The caption says the right panel shows P(RA=1 | RT=1) but the surrounding text describes the right panel as P(RA=1 | RT=0). The right panel label is also inconsistent with the left-panel formula in the same sentence.","section":"Figure 5 caption"},{"comment":"The increase column is formatted inconsistently: most entries are percentages (e.g., 52, 42) but the Mistral-Small-24B MonoRel row reports 0.69 as if it were a proportion. The column header '∼Increase (%)' is also malformed.","section":"Table 4"},{"comment":"The sentence 'We use only the instruction-tuned versions of these models.' is repeated verbatim in the same paragraph.","section":"Section 3.4"},{"comment":"In the BoxTracker false example, the name 'Hanna' appears in Table 1 but 'Hannah' appears in the full example in Appendix A.1; the name should be consistent.","section":"Table 1 / Appendix A.1"},{"comment":"Typo: 'ntheir' should be 'their'. Also, several reference titles contain spurious spaces (e.g., 'T ransactions', 'T ask'), likely from PDF extraction; these should be cleaned.","section":"Appendix A.2.1"}],"recommendation":"major_revision","confidential_remarks":"The Lrel-based analysis is the strongest part of the paper and is likely to be a useful contribution to the long-context evaluation literature. The Section 4.3 claims should be the main focus of revision; if the authors cannot provide difficulty-matched controls, they should be asked to replace 'prerequisite' and 'inherited' with explicit correlational language. The paper is otherwise transparent about its assumptions and provides good supporting material in the appendices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The relative-length framing is the real contribution and it mostly holds up. Six models, four controlled datasets, padding fixed, three positions, seven relative lengths; they find LiM is consistent when Lrel <= 0.5 and fades beyond because primacy decays while recency holds. That is a clean, reproducible result that plausibly reconciles Liu et al. 2024 with Zhang et al. 2024b. The real-world QA replication in Appendix A.2.1 matches the main trends, which gives the claim independent support beyond the synthetic setup.\n\nThe soft spot is Section 4.3. The \"successful retrieval is a prerequisite for reasoning\" headline is stronger than the evidence. The analysis is observational conditioning: P(RA=1|RT=1) vs P(RA=1|RT=0). The gap is real, but it is equally consistent with retrieval-easy instances also being reasoning-easy instances. The only defense is the equal-complexity assumption in Appendix A.4, stated but not tested. And the paper's own Table 4 strains it: P(RA=1|RT=0) = 0.73 for Llama-3-70B on MonoRel. If retrieval failure truly precluded reasoning, that number should be at or near chance; 0.73 means reasoning succeeds often despite retrieval failure. So \"prerequisite\" collapses to correlation unless difficulty is shown matched. Soften to \"supports\" or \"is associated with\" and the claim is fine. Similarly, the claim that reasoning biases are largely inherited from retrieval is suggestive; the flattening of P(RA=1|RT=1) in Figure 5 could be a regression artifact, since conditioning selects on a variable that is itself position-dependent.\n\nTwo smaller issues: no significance tests for the 50% threshold, though the pattern is consistent across models, so this is minor. And LiMi (Eq. 2) is defined to be zero whenever edge accuracy doesn't exceed middle, so part of the \"LiM disappears\" narrative is baked into the metric. The raw accuracy curves in Figure 4 support the substantive claim, so this is a presentation concern, not a fatal one.\n\nOverall: the Lrel result is a solid empirical contribution that deserves to be cited and engaged with. Accept for peer review; the referee should push on the causal language in Section 4.3.","headline":"The relative-length framing cleanly resolves the LiM contradiction; the retrieval-prerequisite claim is correlation dressed as causation and should be softened.","tokens_in":20843,"tokens_out":1853,"would_cite":true,"duration_ms":17979,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lost in the Middle is a real effect, but only up to half a model's context window.","keywords":["positional bias","Lost in the Middle","primacy bias","recency bias","relative input length","context window","long-context language models","retrieval-reasoning"],"falsifier":"Run the same positional protocol on a model at $L_{\\mathrm{rel}}$ values of 0.25, 0.5, and 0.9 while holding the absolute number of tokens between the relevant text and the end of the input constant: if accuracy tracks distance to the end, the distance-based bias is confirmed, and if it tracks $L_{\\mathrm{rel}}$, the window-normalization claim is confirmed. For the retrieval-prerequisite claim, build difficulty-matched retrieval-reasoning pairs and test whether $P(RA=1|RT=0)$ stays measurably below $P(RA=1|RT=1)$ when reasoning complexity is held fixed.","tokens_in":19806,"feed_emoji":"📏","tokens_out":6711,"duration_ms":65840,"temperature":0.7,"pith_summary":"The paper tries to establish that the conflicting reports about the Lost-in-the-Middle effect come from a measurement choice: studies used absolute token counts instead of input length normalized by each model's context window. Measured relative to the window, the effect appears consistently in every tested model when inputs occupy up to 50% of available context; beyond that, the bias for information at the start collapses while the bias for the end survives, leaving a distance-based preference for information near the end. The paper further argues that successful retrieval is a prerequisite for reasoning on long inputs, and that the positional biases seen in reasoning are mostly inherited from retrieval. If correct, this gives researchers a simple normalization rule—compare positions at comparable fractions of the context window—and redirects long-context reasoning failures toward retrieval failures.","feed_headline":"Lost in the Middle is real—up to half a context window","feed_subtitle":"Normalizing input length by context window reconciles conflicting studies and reveals that retrieval gates long-context reasoning.","key_machinery":"The central object is relative input length, $L_{\\mathrm{rel}} = L_{\\mathrm{input}} / L_{\\mathrm{max}}$, the ratio of an input's token count to the model's declared context-window size, evaluated at seven levels from 6% to 100%. Around it the paper defines three intensity metrics: primacy bias intensity (first-position accuracy minus middle-position accuracy), recency bias intensity (last-position accuracy minus middle-position accuracy), and LiM intensity, the sum of the first two only when both are positive. The supporting object is a set of retrieval-reasoning minimal pairs: for each reasoning question a matched retrieval question targets exactly the single premise needed, so reasoning ac","core_discovery":"The paper's central claim is that the inconsistent record on positional biases is an artifact of measuring input length in absolute token counts rather than relative to each model's context window. Across six instruction-tuned open-source LLMs with context windows from 8K to 128K tokens, on four controlled reasoning datasets, the Lost-in-the-Middle effect appears consistently whenever the input fills at most half the model's context window; beyond that threshold, primacy bias falls sharply, even dropping below middle-position accuracy for retrieval in some cases, while recency bias persists. The result is that the V-shaped LiM curve flattens and is replaced by a distance-based bias: the clos","pith_inferences":["Beyond the paper: the distance-based-bias result suggests that model internals track absolute distance to the end rather than ordinal position; a natural follow-up is probing attention or hidden-state dynamics as a function of remaining tokens.","Beyond the paper: if retrieval is truly a prerequisite, retrieve-then-think prompting should improve reasoning more at high $L_{\\mathrm{rel}}$ than at low $L_{\\mathrm{rel}}$—a direct, testable prediction of the inheritance claim.","Beyond the paper: the causal reading could be checked by constructing difficulty-matched pairs where retrieval fails for positional reasons rather than complexity reasons; if the conditional gap persists, the prerequisite claim strengthens.","Beyond the paper: the window-normalization rule may transfer to multimodal or recurrent models whose effective horizons differ from nominal context windows, but the paper only tests transformer-based LLMs.",""],"forward_implications":["Positional-bias evaluations should report $L_{\\mathrm{rel}}$ alongside token counts, because equal absolute lengths from different context windows are not comparable.","Prior conflicting results line up: studies below 50% of the window see LiM, while 100K-token studies mostly do not, because they are sampling different halves of the window.","Failures on long-context reasoning should be investigated as retrieval failures first; reasoning accuracy conditioned on successful retrieval is a cleaner measure of reasoning ability.","Long-context benchmarks need to include multiple relative lengths, since a single fixed length can conceal or fabricate a positional effect.","As inputs fill the window, distance from the end replaces first/middle/last as the organizing variable for accuracy.",""],"supporting_citations":[{"why":"Introduces the Lost-in-the-Middle effect and primacy/recency biases that this paper reanalyzes with relative input length.","marker":"Liu et al. (2024)"},{"why":"Supplies the padding methodology and the MonoRel, PIR, and Simplified RuleTaker datasets used for the positional-bias measurements.","marker":"Levy et al. (2024)"},{"why":"Reports no LiM effect at 100K+ token inputs, the conflicting result that motivates and is explained by the 50%-of-context threshold.","marker":"Zhang et al. (2024b)"},{"why":"Provides the BoxTracker entity-tracking dataset used for the retrieval-reasoning minimal pairs.","marker":"Kim & Schuster (2023)"},{"why":"Represents prior length-adaptable long-context evaluation work that observed LiM at short inputs and informs the relative-length comparison.","marker":"Hsieh et al. (2024)"},{"why":"Represents a long-context benchmark reporting no consistent LiM, one of the inconsistent findings the paper reconciles.","marker":"Modarressi et al. (2025)"},{"why":"Represents a multi-evidence, position-aware benchmark with no LiM, another inconsistent finding the paper's relative-length view explains.","marker":"Song et al. (2025)"},{"why":"Used by the paper to hypothesize why positional biases are more prominent at shorter relative input lengths.","marker":"Wu et al. (2025)"}],"fun_headline_variants":["Lost in the Middle is real—up to half context, then distance bias","Retrieval, not reasoning, carries positional bias in long-context LLMs","Context-window fill determines whether primacy or recency bias wins","Why Lost in the Middle disappears: it's a context-window artifact","LLM positional bias is relative: measure against context window"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The causal reading of the retrieval-reasoning results rests on the assumption, stated only in the appendix, that every example in a dataset has the same logical structure and reasoning difficulty; if retrieval-easy instances are also reasoning-easy, the conditional-accuracy gap could be a composition effect rather than evidence that retrieval enables reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Lost in the Middle is real—up to half context, then distance bias","Retrieval, not reasoning, carries positional bias in long-context LLMs","Context-window fill determines whether primacy or recency bias wins","Why Lost in the Middle disappears: it's a context-window artifact","LLM positional bias is relative: measure against context window"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000356,"raw_usage":{"total_tokens":1774,"prompt_tokens":751,"completion_tokens":1023,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":928}},"tokens_in":495,"tokens_out":1023,"duration_ms":9569,"temperature":1.0,"reasoning_tokens":928,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:07:04.422739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same positional protocol on a model at $L_{\\mathrm{rel}}$ values of 0.25, 0.5, and 0.9 while holding the absolute number of tokens between the relevant text and the end of the input constant: if accuracy tracks distance to the end, the distance-based bias is confirmed, and if it tracks $L_{\\mathrm{rel}}$, the window-normalization claim is confirmed. For the retrieval-prerequisite claim, build difficulty-matched retrieval-reasoning pairs and test whether $P(RA=1|RT=0)$ stays measurably below $P(RA=1|RT=1)$ when reasoning complexity is held fixed.","supporting_citations":[],"review_version":1}