{"id":"6d2968b6-6c25-4350-bc19-8dfe9d87e386","arxiv_id":"2608.03297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Middle-removal truncation destroys the answer in most samples; when only distractors are removed, shorter context never hurts and sometimes improves small models.","lead":"This paper shows that cutting long prompts from the middle often deletes the answer itself, so it measures accidental signal loss, not true context-length effects. A distractor-aware replacement that removes only irrelevant content gives opposite results: shorter context never hurts and can help smaller models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distractor-aware truncation on GraphWalks and MRCR is not shown to meet its stated retention fraction; the equal-token comparison underlying the headline contrast is unverified.","rationale":"The paper is a well-designed paired study with strong internal controls: the signal-preservation diagnostics in Table 3 directly show why naive truncation collapses scores, the split-half analysis bounds seed sensitivity, the Oolong negative control is conceptually apt, and the released code and JSONL records support reproducibility. The central conceptual claim, that naive middle-removal confounds signal loss with context-length effects, is well supported and does not obviously require correction. However, the paper's stronger empirical assertion, that the distractor-aware protocol at 'the same nominal truncation' preserves or improves performance, depends on both protocols actually delivering the same token budgets. The method text enforces the target length only for BABILong; for GraphWalks and MRCR it describes dropping distractors without a stated token-count target, and no truncated token counts are reported. The reader's rationale already notes this issue, though the reader's formal 'weakest_assumption' points to signal-set completeness. The MRCR Sonnet residual harm, -0.112 with padj = 0.034, also shows that incomplete signal definitions can already produce small real effects, but the token-budget question is more load-bearing because it bears on every GraphWalks and MRCR cell and on the title's promise of disentangling context-length effects. If a quick audit of the released JSONL shows distractor-aware token fractions close to alpha, the paper's conclusions stand as stated; if not, the empirical contrast needs to be rerun under matched budgets. The existing CONDITIONAL verdict is therefore appropriate, and no verdict change is required, but the condition should explicitly include verification of realized token counts.","tokens_in":11797,"tokens_out":6483,"duration_ms":62417,"concrete_test":"From the released JSONL, compute tokens_in_per_record and verify the median retained-token fraction per (benchmark, model, protocol, alpha), especially GraphWalks and MRCR at alpha = 0.25. If distractor-aware medians are close to 25% of full-context tokens, the concern is resolved. If not, re-run the GraphWalks and MRCR comparisons with a matched-budget distractor-aware truncation that drops arbitrary distractor edges or turns until alpha L, re-verifies the signal is still present, and recomputes Table 1. If the flat or improved pattern persists under true equal budgets, the central claim survives; if it weakens, the reported contrast is partly a context-length artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical contrast is presented as 'same nominal truncation' at alpha in {1.0, 0.75, 0.5, 0.25} under both protocols (Section 3.2, Tables 1 and 2, Section 4.4). But Section 3.2 only enforces the target length for BABILong, where filler is compressed 'to reach the target alpha L'. For GraphWalks, the signal is the full depth-bounded reachable subgraph and truncation is described as dropping only edges whose source is unreachable; no mechanism compresses the retained reachable subgraph to alpha L. For MRCR, distractor-aware truncation drops turns from the middle of the message list, again without a stated token-budget target. No realized token counts by (protocol, alpha) are reported; Section 5.5 gives only full-context input medians. If the retained reachable subgraph or needle-turn set occupies substantially more than alpha L tokens, which is plausible for dense BFS graphs, then the apparent flat or improved distractor-aware curves reflect roughly the same or longer context rather than '25% context with signal preserved'. Because the abstract and conclusion claim a controlled comparison at equal nominal truncation, this missing budget check is load-bearing: it is the condition that lets the protocols be attributed to signal preservation rather than to different context lengths. The released JSONL records contain provider-reported token counts, so the check is feasible without new API calls.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the standard 'middle-removal' truncation protocol used in many long-context LLM studies confounds two effects: removal of task-irrelevant distractor content (which can help) and removal of answer-bearing signal (which hurts). To separate these, the authors propose a 'distractor-aware' truncation protocol that identifies and preserves a per-sample signal set defined from each benchmark's structure (bAbI fact sentences for BABILong, the depth-bounded reachable subgraph for GraphWalks, needle query/response pairs for MRCR v2, and a recomputed subsample gold for Oolong). They evaluate three Claude models and GPT-5.5 on two primary benchmarks (BABILong, GraphWalks) at four retention fractions (100%, 75%, 50%, 25%) under both protocols, plus MRCR v2 and Oolong as additional/control benchmarks. The headline finding is that naive truncation causes large, statistically significant score drops in all eight primary eval-by-model cells, while distractor-aware truncation at the same nominal retention either preserves performance or, for smaller Claude models on BABILong, significantly improves it. The authors conclude that naive truncation is not a valid measurement of context-length effects and that future studies must distinguish signal from distractor.","tokens_in":12143,"tokens_out":4258,"duration_ms":37672,"significance":"If the central comparison holds, the paper provides an important methodological correction for long-context evaluation. Its strengths are substantial: a clean paired design with Wilcoxon signed-rank and Holm correction; a negative control (Oolong) that behaves as predicted; split-half stability analysis bounding seed sensitivity; explicit per-sample signal definitions verified against dataset ground truth; and a fully released pipeline with raw JSONL records, provenance hashes, and verification scripts. The empirical demonstration of a protocol-induced sign flip (harm under naive, help/neutral under signal-aware) is a valuable caution for the field. The paper also makes a concrete, falsifiable prediction: that a perfect retriever, modeled by distractor-aware truncation, yields only modest gains and mostly helps smaller models, not ceiling-saturating large ones.","major_comments":[{"comment":"The central contrast is presented as 'same nominal truncation' at alpha in {1.0, 0.75, 0.5, 0.25} under both protocols, but the token budget is enforced only for BABILong, where filler is compressed 'to reach the target alpha L'. For GraphWalks the retained signal is the full depth-bounded reachable subgraph, with no mechanism to compress it to alpha L; for MRCR the protocol drops turns but sets no token budget. No realized token counts by (protocol, alpha) are reported (Section 5.5 gives only full-context medians). If the retained reachable subgraph or needle-turn set occupies substantially more than alpha L tokens, then the flat or improved distractor-aware curves reflect a longer effective context rather than '25% context with signal preserved'. Because the abstract and conclusion explicitly claim equality of nominal truncation, this missing budget check is load-bearing. The released JSONL records contain provider-reported token counts, so the check is feasible without new API calls; the authors should either report realized token counts and show they match alpha L, or re-run the comparison with enforced token budgets.","section":"§3.2 and Tables 1-2"},{"comment":"The completeness of the signal set is asserted rather than exhaustively verified. The paper states that the constrained-vocabulary regex achieves '100% recall on the no-filler 0k splits' and 'stable fact counts across all higher-filler splits', but it does not demonstrate that every answer-bearing fact is captured when facts are embedded in PG-19 filler, e.g., if filler is inserted inside a fact sentence. If any supporting fact is missed, the distractor-aware protocol would delete it and the 'preserved by construction' guarantee would fail, invalidating the recovery result on affected samples. The authors should either verify against per-sample supporting-fact annotations if available in BABILong, or perform a reconstructive check: confirm for every distractor-aware truncated prompt that the gold answer is still derivable from the retained facts. The verification scripts currently in the release should be extended to make this check explicit and exhaustive for all splits used.","section":"§3.2, BABILong signal definition"}],"minor_comments":[{"comment":"The abstract and conclusion state that distractor-aware truncation produces 'no harm' across cells, but Table 1 reports a statistically significant harm for Sonnet on MRCR v2 signal-aware (delta=-0.112, p_adj=0.034). The paper acknowledges this residual harm in §4.6, but the summary claims should be qualified to avoid overstatement.","section":"§4.6 and Abstract/Conclusion"},{"comment":"In the 100% column, the GraphWalks and MRCR rows show '(100%)' with parentheses while other cells are plain percentages; please make the notation consistent and clarify whether the parenthesized values are computed on the same basis as the others.","section":"Table 3"},{"comment":"The sentence 'the n_needles user-query needle pairs in MRCR v2' appears to be a typo for 'the n_needles user-query needle pairs' or 'the needle user-query pairs'; please correct the phrasing.","section":"§3.2, Oolong"},{"comment":"The realized input medians (~7k BABILong, ~50k MRCR, ~68k GraphWalks tokens) are reported only for full context; reporting the realized medians for each protocol and retention level would directly address the token-budget concern raised in the major comments and strengthen the paper's transparency.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"The token-budget issue is the key obstacle to accepting the paper's central claim. The authors have the data to resolve it, since the released JSONL contains per-record token counts, and the fix is within scope: either show that realized token counts match the nominal alpha L for GraphWalks and MRCR, or re-run with explicit budget enforcement. I would not recommend rejection, as the methodological contribution and the empirical demonstration are valuable and the paper otherwise shows exemplary statistical hygiene. The MRCR Sonnet harm overstatement is a smaller but easily fixed wording issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's a paper that deserves referee time, but with a clear caveat.\n\nWhat it does well: it formalizes a real confound. Naive middle-removal truncation doesn't measure context-length effects because it deletes the answer in most samples. The diagnostics are convincing: at 25% naive retention, bAbI facts survive in under 1% of BABILong samples, and the BFS gold in 26% of GraphWalks. On BABILong, where the distractor-aware condition actually compresses filler to the target alpha*L while preserving facts, the paired comparison is clean: naive collapses (Holm-significant harm in all cells), while distractor-aware is flat or improves, with significant gains for the two smaller Claude models. That is a real, useful result. The statistical design — paired Wilcoxon, Holm correction, split-half stability, and the Oolong negative control — is better than most of what gets published in this space. The code and data release is a plus.\n\nThe soft spot is load-bearing. The stress-test note checks out: for GraphWalks, the signal is the full reachable subgraph, and truncation drops only edges whose source is unreachable. There is no mechanism that reduces the retained graph to alpha*L. For MRCR, they drop middle turns but never state or enforce a token-budget target. No realized token counts by protocol and alpha are reported. So the 'same nominal truncation' comparison is only true on BABILong. The flat GraphWalks curves, for instance, are likely just full-context curves wearing a '25% retention' label. This doesn't undo the central warning — BABILong alone is enough to show the naive protocol is broken — but the abstract and conclusion claim a controlled contrast on all four benchmarks, and that's not established.\n\nOther limitations (temperature heterogeneity for Opus and GPT-5.5, n=50 on GraphWalks/MRCR, hand-crafted signal definitions) are acknowledged honestly and are minor in comparison. The fix is straightforward: report token counts, or actually truncate the distractor-aware prompts to alpha*L for GraphWalks and MRCR, e.g., by discarding non-gold paths after retaining a witness path for each answer node.\n\nMy take: bring it to the reading group. It's a good case study in how a well-intentioned protocol can quietly change what you're measuring. I wouldn't cite it until the token-budget issue is fixed, but I'd send it to a serious referee. The methodological message is important and the execution on BABILong is credible.","headline":"The paper's warning about naive truncation is right and the BABILong experiment is clean, but the equal-token comparison is unverified on GraphWalks and MRCR, so the headline overclaims.","tokens_in":12560,"tokens_out":4544,"would_cite":false,"duration_ms":39559,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Middle-removal benchmarks measure how often the answer survives, not context-window effects.","keywords":["long-context benchmarks","distractor-aware truncation","signal loss","middle-removal protocol","context-length effects","retrieval-augmented generation","paired Wilcoxon test","BABILong"],"falsifier":"Audit the released per-sample records for a counterexample: a BABILong sample at 25% distractor-aware retention whose score drops below full context, together with a sentence dropped by the distractor-aware truncation that is required to derive the gold answer, would falsify the 'preserved by construction' guarantee; a GraphWalks sample whose kept reachable subgraph fails to derive a gold answer node would do the same.","tokens_in":11550,"feed_emoji":"✂️","tokens_out":7264,"duration_ms":57331,"temperature":0.7,"pith_summary":"The paper argues that the standard way of testing whether shorter context helps language models, truncating long prompts by deleting the middle, cannot distinguish two opposite effects: removing irrelevant distraction versus deleting the very content needed to answer. It proposes a distractor-aware protocol that first defines, for each sample, the minimal task-relevant signal and deletes only the rest. Measured this way, on BABILong and GraphWalks across three Claude model sizes and GPT-5.5, the same nominal truncation no longer hurts: the smaller models improve significantly at 25% retention, and the larger models stay at their full-context ceiling. The naive protocol's collapse coincides with a signal-preservation rate below 1% for BABILong and 26% for GraphWalks at 25% retention. The paper's conclusion is that context-length studies must specify how signal and distractor are separated, or their reported direction is ambiguous.","feed_headline":"Middle-truncation tests measure answer survival, not context effects","feed_subtitle":"Keeping the answer in context flips 'shorter hurts' into 'shorter helps' on long-context benchmarks.","key_machinery":"The load-bearing object is the sample-level signal set: a minimal subset of the prompt sufficient to determine the gold label, with the distractor defined as everything else. The paper instantiates this per benchmark: bAbI fact sentences in BABILong matched by a constrained regex, the depth-bounded BFS-reachable subgraph in GraphWalks recomputed from the query, and the needle query-response pair in MRCR v2 located through the dataset's desired_msg_index field, and verifies each against the dataset's own ground truth. Distractor-aware truncation keeps this set verbatim and compresses or drops only the remainder, so the answer is preserved by construction. The paired statistical design, Wilcoxon signed-rank tests on sample IDs with Holm correction plus paired-bootstrap confidence intervals, is what makes the protocol comparison interpretable as a within-sample effect.","core_discovery":"The central claim is that 'shorter context hurts' and 'shorter context helps' can both be true of the same samples at the same nominal context fraction, depending only on what the truncation deletes. Naive middle-removal is a measurement of how often the answer-bearing content survives the cut; distractor-aware truncation, which preserves the task-sufficient subset by construction, isolates the effect of distractor reduction. On the primary benchmarks, all eight benchmark-by-model cells show significant harm under naive truncation at 25% retention, while distractor-aware truncation produces no harm and, for Haiku and Sonnet on BABILong, statistically significant gains of +0.083 and +0.104. The pattern replicates on GPT-5.5, ruling out a single-provider artifact, and the negative control (Oolong, where every item is signal) shows no protocol difference. The paper frames this as a correction: prior middle-removal results are ambiguous between two opposite hypotheses unless they separate signal from distractor.","pith_inferences":["If the signal sets are incomplete for other splits or task families, the 'preserved by construction' guarantee silently becomes 'preserved only on known signal'; the paper's hand-verified definitions should be re-checked on every new task before the protocol is adopted.","The same distractor/signal separation could be extended to natural-language and multilingual long-context tasks, where signal definition would need to be learned or annotated rather than matched by regex or reachability.","The upper-bound framing suggests a testable scaling prediction: retrieval headroom should shrink as base-model context utilization improves, so 'less is more' claims should be stratified by model capability.","Published naive-truncation results could be re-analyzed post hoc by recomputing signal-preservation rates on the released samples; if preservation is low, those results are consistent with signal loss rather than context-window effects."],"forward_implications":["Any future context-length study that uses middle-removal must report how often the answer-bearing content survives truncation; otherwise a 'shorter is worse' result cannot be distinguished from signal loss.","A perfect retriever, which removes only distractors, gives a controlled upper bound on retrieval-augmented and memory-augmented system headroom; on BABILong this bound is modest in absolute terms and depends on baseline model strength.","At 25% retention, distractor-aware truncation significantly improves Haiku and Sonnet on BABILong, while Opus and GPT-5.5 remain at their full-context ceiling, suggesting saturation rather than context-window failure.","For GraphWalks, naive truncation asks the model the wrong question in 74% of samples at 25% retention, so the naive curve is not about reasoning degradation.","Oolong, where no distractor exists, shows no protocol difference, confirming that the distractor-aware advantage comes from identifying signal, not from shorter prompts per se."],"supporting_citations":[{"why":"Supplies the BABILong benchmark and its embedded bAbI fact sentences, the primary testbed for the signal-loss diagnostics.","marker":"[4]"},{"why":"Supplies the GraphWalks BFS dataset whose reachable-subgraph edges define the task-relevant signal.","marker":"[9]"},{"why":"Supplies the MRCR v2 conversations and the desired_msg_index field used to locate the needle pairs.","marker":"[8]"},{"why":"Supplies the Oolong aggregation task that serves as a negative control with no distractor concept.","marker":"[1]"},{"why":"Documents the lost-in-the-middle effect that motivates middle-removal protocols and that the paper argues has been over-generalized.","marker":"[7]"},{"why":"States the retrieval-augmented 'less context is more' claim that distractor-aware truncation is designed to test under perfect retrieval.","marker":"[6]"},{"why":"Provides the bAbI synthetic QA tasks that BABILong embeds in filler text.","marker":"[13]"}],"fun_headline_variants":["Shorter context helps when the answer survives truncation","Truncation method, not length, decides if shorter context helps","Distractor-aware truncation flips 'shorter hurts' to 'shorter helps'","Shorter context isn't worse when you keep the signal","Answer preservation, not context length, drives long-context results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each benchmark's signal definition captures every piece of content the model could need to answer correctly; if it misses any answer-bearing fact, edge, or turn, the distractor-aware protocol deletes it and the 'preserved by construction' guarantee fails.","fun_headline_variants_meta":{"raw":{"variants":["Shorter context helps when the answer survives truncation","Truncation method, not length, decides if shorter context helps","Distractor-aware truncation flips 'shorter hurts' to 'shorter helps'","Shorter context isn't worse when you keep the signal","Answer preservation, not context length, drives long-context results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3478,"prompt_tokens":1154,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":770,"completion_tokens_details":{"reasoning_tokens":2234}},"tokens_in":770,"tokens_out":2324,"duration_ms":17122,"temperature":1.0,"reasoning_tokens":2234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:51:30.827058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the released per-sample records for a counterexample: a BABILong sample at 25% distractor-aware retention whose score drops below full context, together with a sentence dropped by the distractor-aware truncation that is required to derive the gold answer, would falsify the 'preserved by construction' guarantee; a GraphWalks sample whose kept reachable subgraph fails to derive a gold answer node would do the same.","supporting_citations":[{"cited_title":"BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack","cited_arxiv_id":null,"evidence_quote":"Supplies the BABILong benchmark and its embedded bAbI fact sentences, the primary testbed for the signal-loss diagnostics."},{"cited_title":"GraphWalks dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the GraphWalks BFS dataset whose reachable-subgraph edges define the task-relevant signal."},{"cited_title":"MRCR dataset.https://huggingface.co/datasets/openai/mrcr, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the MRCR v2 conversations and the desired_msg_index field used to locate the needle pairs."},{"cited_title":"Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"Documents the lost-in-the-middle effect that motivates middle-removal protocols and that the paper argues has been over-generalized."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","cited_arxiv_id":null,"evidence_quote":"States the retrieval-augmented 'less context is more' claim that distractor-aware truncation is designed to test under perfect retrieval."}],"review_version":2}