{"id":"49e1d5e4-ec9f-4a98-88a4-9e0434b15fba","arxiv_id":"2507.22411","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs do well at classic needle-in-a-haystack retrieval but fail to integrate all-relevant chains of evidence, especially when the required reasoning order is right-to-left.","lead":"A new benchmark called NeedleChain tests whether language models can combine every piece of a short context when all of it is relevant, rather than just retrieving one snippet. It finds that models, including GPT-4o, collapse when evidence must be processed in reverse order, and it proposes a positional-embedding adjustment that partially restores performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NeedleChain's 'Needle Omission' scores a correct answer as wrong whenever a name is absent from the brief explanation, so the reported integration failures may partly be artifacts of the output protocol rather than of comprehension.","rationale":"The reader identified construct validity as the weakest assumption, specifically citing Sections 3.3 and Appendix B, where a missing name in the output is treated as proof that the corresponding evidence was not used. That is exactly the load-bearing concern: the accuracy numbers and the heatmaps are computed from a mention-based proxy for 'use,' which can misclassify correct answers whenever the model abbreviates its explanation. This is not an external objection about domain generalization; it is an internal measurement problem. The paper's own Appendix B makes the conflation explicit by labeling responses 'incorrect regardless of the correctness of the final answer.' Because the central claim is quantitative -- 'even advanced models fail to reliably integrate inputs as short as 200 tokens' -- a systematic bias in the error metric could turn what is actually a formatting or brevity effect into an apparent comprehension failure. A counterfactual answer-change test would settle the issue by asking whether the model's final answer is causally sensitive to a needle's content, independent of whether the needle's name is repeated in the explanation. If sensitivity is high, the reported 'needle omission' rates and the 'logically lost-in-the-middle' heatmaps should be reinterpreted as explanation-omission artifacts. The reader's conditional verdict already reflects the need for stronger evidence, so no verdict change is needed; the conditional should be understood as conditional on this specific scoring check.","tokens_in":15146,"tokens_out":4808,"duration_ms":64638,"concrete_test":"Run an ablation-based use test on the NeedleChain backward and mixed sets at k=100 and k=200. For each instance, generate the model's answer under the original context and under a counterfactual context where the salary in one needle is changed (e.g., doubled) while all other needles and the query stay fixed. If the final answer changes, the model demonstrably used that needle even if the needle's name does not appear in the explanation. Recompute accuracy and the position heatmaps using this use-based label instead of the mention-based label. If a substantial fraction of responses currently classified as 'needle omission' are use-positive under this test, the paper's central empirical claim is weakened and the error analysis must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that LLMs fail to 'integrate' all-relevant contexts rests on the accuracy numbers and on the position heatmap in Sections 3.2-3.3. The load-bearing step is the operational definition of 'needle omission': a response is classified as a needle-omission error if a name from the input is absent from the generated text, and Appendix B explicitly states that such responses are 'incorrect regardless of the correctness of the final answer.' This equates surface mention with information use. But the evaluation prompt in Table 3 asks for only a 'brief explanation' before the final answer. A model that silently uses all needles and then gives a correct final answer while compressing its explanation -- for example, writing 'since Kyren equals Justin, which equals Salma, which is equal to Ronin: $200' without restating the independent salary fact -- is automatically scored as wrong. The same conflation drives the position heatmap, which measures name-mention rates rather than whether the answer actually depends on each needle. Because needle-omission errors grow with k and dominate the large-k regime, the headline result that GPT-4o 'fails to reliably integrate' short contexts may substantially overstate the failure. The limitation section also admits the benchmark uses only numeric salary needles, but even before generalizing to naturalistic contexts, the scoring artifact threatens the internal validity of the reported accuracy and error taxonomy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NeedleChain, a synthetic benchmark for measuring whether LLMs integrate all query-relevant information in a context. Each instance is a chain of salary facts (one independent needle plus k-1 dependent needles), with three variants requiring forward, backward, or mixed reasoning order, and a parallel Needle-in-a-Haystack benchmark called NeedleStack built from the same needles. Experiments on Qwen, Llama, and GPT-4o show near-perfect accuracy on NeedleStack but substantial degradation on NeedleChain as k grows, especially for backward chains. The paper also proposes a training-free 'ROPE contraction' strategy, which increases the RoPE rotation angle at inference, and reports that it improves NeedleChain accuracy relative to YaRN extension. The central claim is that current LLMs fail to fully understand contexts even at short lengths when all context is relevant.","tokens_in":15414,"tokens_out":4369,"duration_ms":54386,"significance":"If the main result holds, it is an important qualification of long-context evaluation: retrieval-heavy benchmarks may overestimate holistic context comprehension. The controlled chain design, with the same name lists and needle compositions across the three variants and NeedleStack, is a useful contribution, and the authors state that they release the data generation code, which supports reproducibility. The comparison of forward, backward, and mixed orders gives a clean way to separate reasoning-direction effects from context-length effects. However, the benchmark's validity depends on two assumptions that are not adequately defended: that name-mention in the explanation is a faithful proxy for information use, and that synthetic salary-ratio chains represent 'intact context comprehension' in a way that supports the paper's broad conclusions. The reported ROPE contraction results are also under-specified. These issues are load-bearing for the abstract and Section 4 claims, so the manuscript needs substantial revision before the central claims can be accepted.","major_comments":[{"comment":"The 'Needle Omission' error category conflates surface mention with information use. Appendix B explicitly states that a response is incorrect 'regardless of the correctness of the final answer' if a needle's name is absent from the generated text. However, the evaluation prompt in Table 3 only asks for 'a brief explanation' and does not require the model to restate every input name. A model that correctly derives the final answer while summarizing the chain without repeating each name would be scored as an omission error. Because needle omission is reported as the dominant error mode at large k and drives the position heatmap in §3.3 (which measures name-mention rates, not actual evidential dependence), the accuracy numbers in Table 1 and the conclusion that models 'fail to integrate' short contexts may be substantially overstated. Please re-score the data with final-answer correctness assessed independently of explanation completeness, or change the prompt to require an exhaustive derivation, and report both the original and re-scored results.","section":"§3.2, §3.3, Appendix B"},{"comment":"The abstract's claim that 'even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens' is not supported for all chain variants. At k=20 (approximately 0.2K tokens), Table 1 shows GPT-4o at 98.0% on the forward chain and 88.5% on the mixed chain, with the backward chain at 78.5%. The headline failure at 200 tokens is thus really a backward-chain phenomenon at that length. Please qualify the claim to state which chain direction fails, or identify the length at which all variants show reliable failure.","section":"Abstract and Table 1"},{"comment":"The construct validity of NeedleChain is assumed rather than demonstrated. Section 7 concedes that the benchmark uses only numeric salary needles, yet Section 4 generalizes to the broad conclusion that 'LLMs do not yet fully comprehend given contexts.' No external validation is provided, such as correlation with established long-context tasks (e.g., LongBench, RULER, or a naturalistic all-relevant document task) or a human baseline on the same chains. Without such evidence, the paper should either restrict its conclusions to the synthetic setting or add a validation study showing that NeedleChain performance tracks a broader notion of intact context comprehension.","section":"§4, §7"},{"comment":"The ROPE contraction experiments are not sufficiently specified. The text says rotation angles of 2x and 4x were used and that contraction improves performance relative to YaRN, but it does not state which models and k values are shown in Figure 7, how the multipliers were selected, how many independent runs were performed, or whether the improvements are statistically significant. Since ROPE contraction is a central contribution, the experimental protocol and variability measures must be reported.","section":"§4, Figure 7"}],"minor_comments":[{"comment":"Typo: 'This refers to a sentences' should be 'This refers to a sentence'.","section":"Section 2.1"},{"comment":"Terminology is inconsistent: the main text uses 'Mixed Chain' but the appendix table label says 'Chaotic Chain'. Please unify the terminology.","section":"Appendix B and Section 2.2"},{"comment":"The sentence 'The experimental results clearly support our argument' appears twice in Section 4; one occurrence should be removed.","section":"Section 4"},{"comment":"The caption 'Performance variation with respect to the domain composition of training data' appears to be mismatched with the figure content, which shows example chain compositions. Please correct the caption.","section":"Figure 2 caption"},{"comment":"'ROPE' should be consistently spelled 'RoPE' (rotary position embedding), and 'Yarn' should be 'YaRN' when referring to the method of Peng et al.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a useful and well-controlled benchmark, and the authors are transparent about limitations. However, the needle-omission scoring rule is a genuine threat to the paper's central empirical claim, not just a presentation issue. If the authors re-score the data and adjust the conclusions accordingly, the paper could be suitable for publication. I would also ask the authors to position the work more carefully against existing all-relevant-context benchmarks such as NeedleBench; the current related-work discussion is brief."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The NeedleChain benchmark is a real contribution, but the paper's headline numbers are partly an artifact of its own scoring rule. The idea of testing models on contexts where every sentence is load-bearing, and separating forward, backward, and mixed reasoning order, is fresh and worth building on. The backward-chain performance collapse across several models is a new observation, and the simple ROPE contraction experiment is a clever twist. The paper deserves a serious look.\n\nThe soft spot is not the benchmark construct; it's the output protocol. Appendix B states that a response is labeled \"needle omission\" — and thus incorrect — if any input name is absent from the generated explanation, \"regardless of the correctness of the final answer.\" But the prompt asks only for a \"brief explanation.\" A model that correctly uses all evidence and then writes \"since Kyren equals Justin, which equals Salma, which is equal to Ronin: $200\" without restating Ronin's independent salary line is scored wrong, even though it did integrate the information. The position heatmap measures name-mention rates, not actual information use. This conflation directly inflates the reported accuracy drop at large k and the needle-omission error fractions. The abstract's claim that GPT-4o \"fails to reliably integrate inputs as short as 200 tokens\" also outruns the data: at k=20 (0.2K tokens), GPT-4o is at 78.5% on the backward chain, and near 100% on forward and mixed. That's a weakness, not a collapse.\n\nThe ROPE contraction section is under-supported — no error bars, no model/model-size detail, no external baseline beyond one Yarn comparison, and no artifact. The domain limitation (only numeric salary chains) is acknowledged, but the paper generalizes to \"LLMs do not yet fully comprehend given contexts\" anyway. If the scoring artifact is corrected, the backward-chain trend likely survives but is much weaker.\n\nWho is this for? Researchers building long-context benchmarks and anyone testing positional-embedding tricks. It's a useful benchmark to cite, but I would not trust the absolute accuracy numbers without re-scoring.\n\nMy advice: send to peer review, but with a strong request to re-analyze the scoring and temper the claims. The core idea is solid enough to warrant serious referee time.","headline":"The NeedleChain benchmark is a genuinely useful new probe of all-relevant context integration, but the paper's headline failure numbers are inflated by a scoring rule that counts correct final answers as wrong when a name is omitted from a brief explanation.","tokens_in":15913,"tokens_out":4044,"would_cite":true,"duration_ms":41456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When every sentence is relevant, LLMs fail to integrate even 200 tokens","keywords":["long-context comprehension","intact context comprehension","needle-in-a-haystack","chain reasoning","ROPE contraction","position bias","benchmark construction","LLM evaluation"],"falsifier":"Run the same backward-chain NeedleChain data rendered as a natural narrative (e.g., 'Cairo took home $1600, and Ramon makes twice Cairo's salary...') at k=20; if a model that fails the terse version scores high on the narrative version, the reported failure is an artifact of the benchmark's synthetic format rather than a general inability to integrate all-relevant context.","tokens_in":14932,"feed_emoji":"🔗","tokens_out":6177,"duration_ms":64113,"temperature":0.7,"pith_summary":"The paper argues that current long-context benchmarks overstate what LLMs understand, because they bury relevant facts in irrelevant text and therefore reward retrieval rather than integration. To test this, the authors build NeedleChain, a benchmark in which every sentence is needed: salary facts form a chain (e.g., 'A received $1600'; 'A earns twice as much as B'), and the only way to answer is to combine all of them. Across three orderings (forward, backward, mixed), models that score near 100% on a needle-in-a-haystack version of the same data fall sharply once the chain exceeds a few steps, with GPT-4o unreliable at roughly 200 tokens. The authors also propose ROPE contraction, increasing the rotary-position rotation angle at inference, which improves full-context integration. If right, the result means advertised context lengths say little about whether a model can use a short, information-dense passage as a whole.","feed_headline":"When every sentence matters, GPT-4o fails at 200 tokens","feed_subtitle":"New benchmark shows near-perfect retrieval but weak integration; a simple positional tweak helps.","key_machinery":"The load-bearing object is the NeedleChain benchmark, built from two kinds of 'needles': independent needles that fix one person's salary (e.g., 'A received $1600 last week') and dependent needles that relate salaries ('A earns twice/half/the same as B'). A chain of k−1 dependent needles plus one independent needle forms a single semantic unit, and the query asks for the salary at the end of the reasoning order, so omitting any needle makes the answer unreachable. The benchmark's three variants—forward, backward, and mixed—keep the same needles and only change the presented order, which isolates the effect of reasoning direction. A parallel NeedleStack benchmark concatenates independent needles, turning all but one into irrelevant 'haystack' content. The analysis tool that exposes positional weakness is a heatmap of which names appear in the model's output, used to separate presented position from reasoning position. The proposed intervention, ROPE contraction, increases the rotary position embedding rotation angle at inference (opposite of ROPE extension) to sharpen positional distinctions and reduce evidence omission.","core_discovery":"In the paper's own terms, the central discovery is that LLMs possess an 'intact context-understanding length' far shorter than their nominal context window: when the context contains only query-relevant text, models such as GPT-4o, Llama3.3-70B, and the Qwen family fail to reliably integrate inputs as short as roughly 200 tokens (about 20 salary relations), even though the same models are essentially perfect at locating a single relevant fact in a haystack built from the same sentences. The failure is order-dependent: forward chains, where the needed reasoning follows the presented order, are handled well, but backward chains, requiring right-to-left integration, show the largest drop, larger than arbitrarily mixed order. Error analysis attributes the failures first to calculation errors and increasingly, as the chain lengthens, to omission of entire evidence units, with the omitted positions following the 'logical' middle of the reasoning path rather than the middle of the text. The authors further report that changing the question from a single target to the total salary collapses needle-in-a-haystack performance, that adding a code interpreter rescues that retrieval setting but not NeedleChain, and that ROPE contraction—using a larger rotation angle at inference than at training—significantly improves intact comprehension, while the standard YaRN extension strategy degrades it.","pith_inferences":["The 'logically lost in the middle' pattern suggests a testable extension: reordering evidence so that the logical chain is monotone in presented order should eliminate most of the backward-chain deficit, which could be checked on the same benchmark without retraining.","Because the benchmark uses only salary comparisons, a natural next test is whether the same collapse appears with other all-relevant relations (causal, temporal, spatial); if it does not, the finding may be narrower than 'context comprehension' and partly about numerical-chaining format.","The success of ROPE contraction hints that position-signal sharpening may trade off against retrieval of very long contexts; combining contraction for dense passages with extension for sparse long documents could be a productive research direction.","Tool incorporation failing to help NeedleChain suggests the bottleneck is not computation but evidence maintenance; probing hidden states during backward chains could test whether earlier needles are attended to but then overwritten."],"forward_implications":["Accuracy on needle-in-a-haystack style evaluations should not be read as evidence of context comprehension; a model can be perfect at retrieval while failing to integrate a 200-token all-relevant passage.","The order in which evidence is presented materially changes reasoning quality; structuring documents so that the logical dependencies run left-to-right should improve LLM performance at no training cost.","Context-length extension methods that weaken positional distinctions (e.g., RoPE extension) can actively hurt full-context integration, while sharper position signals (ROPE contraction) can help.","Benchmarks for 'long context' should include information-dense, all-relevant conditions, because retrieval-style tests overestimate practical ability.","Error growth with length is dominated by omission of entire evidence pieces, so improving integration may require attention or memory mechanisms that keep all items active, not better arithmetic."],"supporting_citations":[{"why":"LongBench; the representative long-context benchmark that mixes relevant and irrelevant content, which the paper argues overestimates understanding.","marker":"Bai et al., 2024"},{"why":"NeedleBench; source of the needle-in-a-haystack paradigm used to build the parallel NeedleStack comparison.","marker":"Li et al., 2024b"},{"why":"RULER; prior evidence that effective context usage is shorter than nominal context length, which the paper extends.","marker":"Hsieh et al., 2024"},{"why":"BABILong; shows LLMs struggle on long-context reasoning-in-a-haystack, grounding the gap the paper quantifies.","marker":"Kuratov et al., 2024"},{"why":"Lost in the middle; supplies the 'lost in the middle' phenomenon that the paper re-examines along reasoning order.","marker":"Liu et al., 2024b"},{"why":"RoFormer; defines the rotary position embedding that ROPE contraction modifies.","marker":"Su et al., 2024"},{"why":"YaRN; the ROPE extension method that the paper compares against contraction and shows to degrade NeedleChain performance.","marker":"Peng et al., 2024"}],"fun_headline_variants":["LLMs fail to integrate all-relevant context at 200 tokens","GPT-4o stumbles when context is entirely relevant, at 200 tokens","NeedleChain: The 200-token wall for LLM context integration","Context comprehension fails early: LLMs stumble at 200 tokens","All relevant, no haystack: LLMs fail to integrate 200 tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that performance on synthetic salary-ratio chains is a faithful measure of 'intact context comprehension': the paper assumes that when a model omits a name from its answer, the corresponding evidence was not used, and that these arithmetic chains stand in for realistic all-relevant contexts.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail to integrate all-relevant context at 200 tokens","GPT-4o stumbles when context is entirely relevant, at 200 tokens","NeedleChain: The 200-token wall for LLM context integration","Context comprehension fails early: LLMs stumble at 200 tokens","All relevant, no haystack: LLMs fail to integrate 200 tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001253,"raw_usage":{"total_tokens":5166,"prompt_tokens":1004,"completion_tokens":4162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":4065}},"tokens_in":620,"tokens_out":4162,"duration_ms":29505,"temperature":1.0,"reasoning_tokens":4065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:41:22.136657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same backward-chain NeedleChain data rendered as a natural narrative (e.g., 'Cairo took home $1600, and Ramon makes twice Cairo's salary...') at k=20; if a model that fails the terse version scores high on the narrative version, the reported failure is an artifact of the benchmark's synthetic format rather than a general inability to integrate all-relevant context.","supporting_citations":[{"cited_title":"Babilong: Testing the limits of llms with long context reasoning-in-a-haystack","cited_arxiv_id":null,"evidence_quote":"BABILong; shows LLMs struggle on long-context reasoning-in-a-haystack, grounding the gap the paper quantifies."}],"review_version":1}