{"id":"7cc1d775-a33c-4e1f-bf3d-9f5f28b22206","arxiv_id":"2412.10079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In multi-hop QA, language models answer worse when the required evidence documents are separated by distractor documents, not just when the evidence sits in the middle of the context.","lead":"This paper tests whether long-context language models lose track of information not just in the middle of the input, but when the clues needed to answer a question are far apart from each other. It finds that multi-hop question answering gets harder when the supporting documents are separated by distractors, and that summarization or knowledge-graph compression reduces position bias but costs accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central distance claim is not isolated: Figure 4 averages adjacent and separated configurations that differ in absolute positions, and no significance testing is reported.","rationale":"The reader's weakest assumption identifies essentially the same load-bearing concern: the hand-picked adjacent and separated configurations are averaged without isolating distance from absolute position. I sharpen this to a concrete, testable confound: the adjacent set includes edge-edge pairs like (1,2) and (19,20), while the separated set does not, so the classic U-shaped position effect can inflate the adjacent average even with no true distance effect. This matters because the paper's practical message, that RAG pipelines must care about the arrangement of supporting evidence, rests entirely on the distance effect being real. The paper deserves credit for publishing full per-configuration results, which make the requested reanalysis straightforward, but the current aggregate Figure 4 and absence of statistical inference leave the central claim under-supported. I do not see grounds for rejection: the reanalysis could confirm the effect, and the paper already includes a limitations statement acknowledging the subset. The reader's CONDITIONAL verdict is therefore appropriate, and I would not change it.","tokens_in":17210,"tokens_out":5963,"duration_ms":63697,"concrete_test":"Reanalyze the Appendix A full-document accuracy tables with a per-question logistic mixed model per model and dataset: predict correctness from an adjacency indicator while controlling for absolute-position covariates (first evidence position, last evidence position, and their mean, or exact strata on first evidence position). If the adjacency coefficient becomes non-significant or changes sign after controlling for position, the Figure 4 distance effect is confounded; if it remains significant and negative across most model-dataset cells, the claim survives. Also report paired bootstrap confidence intervals for the adjacent-minus-separated difference within position-matched strata.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 defines adjacent placements such as (1,2) and (19,20) and separated placements such as (1,5) and (15,20), and Figure 4 compares the group means. These groups are not matched on the marginal distribution of document positions: the adjacent set includes pairs at both extreme edges, while the separated set never places both evidence documents at a single edge and is shifted toward the interior. Because Liu et al. (2024) established a U-shaped absolute-position effect, an adjacent-vs-separated difference can arise purely from this imbalance even if inter-evidence distance has no causal effect. No regression, matching, or position-controlled paired analysis is provided, and no confidence intervals or significance tests appear anywhere in the paper. The full tables in Appendix A show the fragility of the aggregate comparison: for MPT on 2Wiki 4-hop full documents, the adjacent and separated averages are 51.67% and 51.68%, respectively, and several other model-dataset cells differ by less than one percentage point. Thus the paper's headline claim, that performance degrades with the relative distance between relevant documents, is not yet established from the reported aggregate comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the \"lost in the middle\" phenomenon in multi-hop question answering, where multiple relevant documents must be integrated. The authors position two or more gold documents among 20 documents and measure accuracy for three models (GPT-3.5-turbo, MPT-7b, Llama-2-longlora) on HotpotQA, 2WikiMultihopQA, and MuSiQue. They report a new effect: performance degrades not only with the absolute position of evidence but also with the relative distance between relevant documents. They also test chain-of-thought prompting and two context-reduction methods (knowledge-graph triple extraction and summarization). The paper includes full result tables in an appendix and releases code and data.","tokens_in":17378,"tokens_out":5202,"duration_ms":54796,"significance":"If the central claim is correct, it extends the known absolute-position bias to a pairwise layout effect, which has direct implications for retrieval-augmented generation over multi-hop questions. The study is useful as a systematic benchmark: the position conditions are pre-specified, the evaluation uses standard public datasets and a standard metric, generation uses temperature 0, and the code and data are released. These are real strengths. However, the central claim is currently supported by an aggregate comparison that does not control for absolute position and by qualitative inspection of point estimates without uncertainty quantification, so the significance of the contribution is not yet firmly established.","major_comments":[{"comment":"The headline claim that inter-evidence distance degrades performance is not isolated from absolute position. The adjacent set in Figure 4 contains edge pairs (1,2) and (19,20), while the separated set contains no pair in which both documents are at the same edge; given the known U-shaped absolute-position effect (Liu et al., 2024), a group-mean difference can arise from this distributional imbalance even if spacing has no causal effect. No significance tests or confidence intervals are reported anywhere in the paper. The data also contain direct counterexamples: in MuSiQue 4-hop for GPT-3.5 (Table 3), adjacent condition (15,16,17,18) gives 51.72% while separated condition (14,16,18,20) gives 53.20%, and in 2Wiki 4-hop for MPT (Table 4), the adjacent and separated averages are essentially identical (51.67% vs. 51.68%). The authors should redo the analysis position-matched (e.g., comparing (1,2) with (1,5), (5,6) with (5,10), etc.) or fit a regression with controls for absolute positions, and report uncertainty.","section":"Section 6, Figure 4"},{"comment":"The choice of position combinations is a small, hand-picked subset: 5 adjacent and 4 separated for 2-hop and 4-hop, and 3 separated for 3-hop, out of 190, 1140, and 4845 possible combinations, respectively. No justification is given for why these particular combinations are representative of the full space, and the paper's own Limitations section concedes that the analysis \"may not capture the full extent of potential variations.\" Because the aggregate adjacent-versus-separated difference is the sole basis for the central claim, the subset selection could be driving the result. A more defensible approach would be to exhaustively evaluate all combinations for the 2-hop case and to sample uniformly (with reported variance) for higher-hop cases, or at least to show that the chosen subset is balanced on absolute-position margins.","section":"Section 4, Methodology"},{"comment":"The claimed distance effect is not consistent across the reported conditions. The paper states that \"models perform better when relevant documents are adjacent compared to when they are separated,\" but the Appendix tables show multiple cells where the opposite holds: besides the GPT MuSiQue 4-hop case noted above, MPT on 2Wiki 4-hop with standard prompts shows separated (14,16,18,20)=53.00% outperforming every adjacent condition except (1,2,3,4)=52.42%, and GPT on 2Wiki 4-hop with KG+CoT has separated (1,3,5,7)=66.62% versus adjacent (17,18,19,20)=64.13%. The aggregate plot hides this variability. The authors should report per-condition comparisons and statistical tests of the interaction between spacing and absolute position, rather than only group means.","section":"Section 6, Figure 4"}],"minor_comments":[{"comment":"In panel (a), the document labels repeat \"Document [3]\" before \"Document [2]\"; this appears to be a typo and should be corrected.","section":"Figure 1"},{"comment":"The terms \"Adjacent Documents\" and \"Separated Documents\" are not precisely defined in the caption; please state which position pairs/triples are included in each group, since the reader must otherwise infer them from Section 4.","section":"Figure 4 caption"},{"comment":"The tabular layout separates hop counts by row blocks inside the same table, which is hard to parse. Using separate tables per hop count or adding explicit horizontal rules would improve readability. Also, the column headers repeat the position tuples without a separator from the prompt type; consider adding a subheader row.","section":"Appendix A, Tables 3-5"},{"comment":"For Llama-2-longlora, the MuSiQue 3-hop and 4-hop results are all 0.00% in every condition (Table 5). The paper relegates these to the appendix with a note about poor performance, but it would be useful to explicitly state that floor effects make these conditions uninformative for the distance analysis, rather than including them in any summary statement.","section":"Section 3.1.1 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid: the central claim is currently supported by a non-matched aggregate comparison without inferential statistics, and the paper's own tables contain counterexamples. The manuscript is not beyond repair—the raw data are public, the position conditions are pre-specified, and the 2-hop matched comparisons often go in the claimed direction—but the analysis needs to be redone with position controls and uncertainty quantification before the distance effect can be considered established. I would not reject, but I would require a substantive revision, not just a wording change."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper does something genuinely new: it takes the lost-in-the-middle setup and moves it to multi-hop QA, measuring accuracy as a function of the separation between multiple gold evidence documents. That specific measurement hasn't been done before, as far as I can tell from the citations. The authors run it on three standard multi-hop datasets across three models, publish full tables, release code and data, and are upfront about the limits. That's real work, and the per-position numbers in the appendix give some support to their distance effect—e.g., for GPT-3.5 on MuSiQue 2-hop, the separated configurations are often a few points below adjacent ones at similar start positions.\n\nThe soft spot is the headline claim. Figure 4 compares the average of five adjacent placements against the average of four separated placements, and those two sets are not matched on absolute document positions. The adjacent set includes (1,2) and (19,20); the separated set never has both documents at one edge. Since Liu et al. established a U-shaped effect of absolute position, the aggregate difference can be explained at least in part by that confound. There are also no confidence intervals or significance tests anywhere, and several 4-hop MuSiQue cells go the other way. The authors acknowledge they only sampled a subset of combinations, and they do qualify the conclusion, but the way the conclusion is phrased in the abstract and intro overstates what the data show.\n\nThe mitigation experiments are preliminary and messy, but they're honest about that. The CoT results are mixed, and the context-reduction methods flatten the position curve while dropping accuracy. Not a breakthrough, but useful as a negative result about off-the-shelf mitigations.\n\nWho should read this? People building or evaluating RAG pipelines and anyone studying positional bias in long-context models. The methodological lesson—that you need to match absolute positions or control for them when comparing 'adjacent' vs 'separated'—is worth internalizing.\n\nMy recommendation: send it to peer review, but ask the reviewers to require a reanalysis that controls for absolute position and reports uncertainty. The underlying idea is worth publishing; the current analysis doesn't yet nail the central claim.","headline":"A useful extension of lost-in-the-middle to multi-hop QA with a plausible but not yet isolated inter-evidence distance effect; needs a reanalysis before the headline claim is accepted.","tokens_in":17924,"tokens_out":2851,"would_cite":true,"duration_ms":31282,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the known 'lost in the middle' failure of long-context language models is not only about where a single piece of evidence sits relative to the context edges.","keywords":["lost in the middle","multi-hop question answering","long-context language models","evidence position","inter-evidence distance","chain-of-thought prompting","retrieval-augmented generation","context reduction"],"falsifier":"Run the same models on a random sample of all possible evidence-position pairs in a 20-document context and check whether, after controlling for each evidence document's distance from the context edges, accuracy still decreases as the gap between the two evidence documents increases; if it does not, the inter-evidence distance claim fails.","tokens_in":17005,"feed_emoji":"🧩","tokens_out":6728,"duration_ms":58965,"temperature":0.7,"pith_summary":"The paper argues that the known 'lost in the middle' failure of long-context language models is not only about where a single piece of evidence sits relative to the context edges. In multi-hop question answering, where an answer requires combining evidence from several documents, performance also drops when the relevant documents are far apart from each other, even when each one individually is in a favorable position. If this is right, systems that feed many retrieved documents to a model must care how supporting evidence is arranged, not just whether it is present.","feed_headline":"QA accuracy drops when evidence documents are spread apart","feed_subtitle":"Beyond 'lost in the middle,' spacing between relevant documents changes model performance.","key_machinery":"The central object is the set of evidence-position combinations in a fixed 20-document prompt: five adjacent placements such as (1,2) and (10,11), plus separated placements such as (1,5) and (10,15) for two-hop questions, with parallel schemes for three- and four-hop questions. Comparing the averaged accuracy of adjacent versus separated placements is what isolates the relative distance between evidence documents from their absolute positions. The argument rests on treating this comparison as the measurable signature of how well models integrate disconnected evidence.","core_discovery":"The central discovery is that multi-hop QA accuracy degrades with the distance between multiple pieces of evidence, not just with their distance from the edges of the input. The paper fixes a 20-document context, places the gold evidence documents either adjacent to each other or separated by distractor documents, and measures best-subspan accuracy on HotpotQA, 2WikiMultihopQA, and MuSiQue-Ans. Across instruction-tuned models, adjacent evidence consistently outperforms separated evidence. The paper also reports that chain-of-thought prompting helps instruction-tuned models identify relevant documents but does not erase the positional disparity, and that context reduction via summarization or knowledge-graph triples flattens the position curve at the cost of overall accuracy.","pith_inferences":["If inter-evidence distance is causal, then accuracy should fall monotonically as the gap between two evidence documents grows, a prediction the paper's selected configurations only partially test; a systematic sweep over all gaps would sharpen or refute it.","The finding suggests a 'lost in-between' component interacts with attention mechanisms: models may need a second retrieval or a mid-context 'bridge' summary to reconnect separated evidence, which is a testable engineering response.","The results imply that evaluation protocols for long-context reasoning should report evidence-spacing conditions, otherwise two systems with identical average accuracy could differ sharply in how they handle dispersed evidence.","For non-instruction-tuned models, the paper's observation that summaries and triples improve performance hints that these models may be failing on surface-level format rather than reasoning, a possibility worth probing with controlled prompts."],"forward_implications":["Retrieval-augmented generation systems should consider the spacing between supporting passages, not only their top-k relevance scores, when ordering documents for a prompt.","Re-ranking methods designed for single-hop 'lost in the middle' may not transfer to multi-hop settings because the number of relevant orderings grows combinatorially with the number of hops.","Chain-of-thought prompting can lift overall accuracy for instruction-tuned models but should not be expected to remove position-based performance gaps.","Context compression can reduce the model's sensitivity to evidence position, but only at the price of losing information, so it is not a free fix.","The distance effect should be treated as a benchmark target for long-context models: a model that reasons robustly over dispersed evidence should show a small adjacent-versus-separated gap."],"supporting_citations":[{"why":"Defines the 'lost in the middle' phenomenon and supplies the fixed-position methodology and contriever setup that the paper extends to multiple evidence documents.","marker":"Liu et al. (2024)"},{"why":"Provides the HotpotQA multi-hop question answering dataset used for one of the three evaluations.","marker":"Yang et al. (2018)"},{"why":"Provides the MuSiQue multi-hop dataset and the notion of disconnected reasoning that motivates the distance comparison.","marker":"Trivedi et al. (2020)"},{"why":"Supplies LongLoRA, the long-context fine-tuning method used to build one of the evaluated models.","marker":"Chen et al. (2023)"},{"why":"Provides the Llama 2 base model used both as one evaluated model and as the extractor for knowledge-graph triples.","marker":"Touvron et al. (2023)"},{"why":"Provides BART-large-CNN, the summarization model used to reduce document length in the context-reduction experiments.","marker":"Lewis et al. (2019)"},{"why":"Supplies the chain-of-thought prompting strategy whose effects on positional bias are tested.","marker":"Zhou et al. (2023)"},{"why":"Supplies ALiBi, the position-bias mechanism used by MPT-7b, one of the evaluated models.","marker":"Press et al. (2022)"}],"fun_headline_variants":["Multi-hop QA accuracy drops when evidence documents are spaced apart","Evidence document distance degrades multi-hop QA accuracy","Beyond lost-in-middle: evidence gap hurts multi-hop reasoning","Chain-of-thought can't fix evidence-distance bias in QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's evidence-distance claim depends on the nine hand-picked position combinations being representative of all possible placements; if those configurations are not representative or are confounded with absolute position, the averaging that isolates distance would not establish the effect.","fun_headline_variants_meta":{"raw":{"variants":["Multi-hop QA accuracy drops when evidence documents are spaced apart","Evidence document distance degrades multi-hop QA accuracy","Beyond lost-in-middle: evidence gap hurts multi-hop reasoning","Chain-of-thought can't fix evidence-distance bias in QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2850,"prompt_tokens":865,"completion_tokens":1985,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1931}},"tokens_in":481,"tokens_out":1985,"duration_ms":15735,"temperature":1.0,"reasoning_tokens":1931,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:22:19.877116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same models on a random sample of all possible evidence-position pairs in a 20-document context and check whether, after controlling for each evidence document's distance from the context edges, accuracy still decreases as the gap between the two evidence documents increases; if it does not, the inter-evidence distance claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MuSiQue multi-hop dataset and the notion of disconnected reasoning that motivates the distance comparison."}],"review_version":1}