{"id":"fd094683-b182-447a-9159-3125af467790","arxiv_id":"2505.11754","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Ordering retrieved documents along the reasoning chain and replacing the causal mask with a prefix mask during LoRA fine-tuning improves multi-hop QA accuracy; peak attention scores can select the best context order.","lead":"Language models answer multi-hop questions better when the supporting documents appear in the order of the reasoning chain, and swapping in a bidirectional attention mask during fine-tuning improves accuracy. The paper also shows that a model's own peak attention score can pick the most promising document ordering, lifting Qwen 7B accuracy from 28.6% to 33.7%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prefix-mask gains may stem from question-aware document encoding, not bidirectional document attention; missing prompt-order control","rationale":"The paper is a systematic empirical study, and the bidirectional-attention result is internally consistent across model families and on a second dataset (Appendix H). My concern is not that the reported Finetuned+Bi > Finetuned numbers are wrong, but that the design conflates two effects of the prefix mask: (i) bidirectional document-document attention, which the paper emphasizes, and (ii) question-aware document encoding, because the prompt in Appendix C places the question after the documents. In a causal model, document representations cannot incorporate the later-occurring question; under Eq. 4 they can. The reader's weakest assumption concerned the IC-based selection heuristic (Section 6.1 and Figure 5), a genuine issue for that secondary contribution. However, that heuristic is evaluated partly on 2WikiMultihopQA (Appendix H), so the dev-set overfitting risk is at least partially mitigated. The prompt-order confound affects the primary architectural claim and is untested. The proposed control is inexpensive and would decisively separate mechanism from prompt layout. If the question-first causal baseline matches Finetuned+Bi, the paper's practical recommendation should shift from 'modify the mask' to 'place the question before the context'; if the gap remains, the causal-mask bottleneck is real. The existing CONDITIONAL verdict is appropriate; the condition should include this prompt-order control.","tokens_in":19888,"tokens_out":14140,"duration_ms":139901,"concrete_test":"Run the Finetuned (causal LoRA) condition for Qwen2.5 7B on the MuSiQue dev set using the same hyperparameters and seed, but with the Appendix C prompt reordered so the question appears immediately after the instruction and before the documents (answer-only format, same greedy decoding). Compare Acc and Delta-B/Delta-F against Table 1 Finetuned and Finetuned+Bi. If question-first causal accuracy reaches within ~1 point of Finetuned+Bi, the headline gain is mostly prompt-order; if it remains at least 2 points below, bidirectional attention is the operative factor. Additionally run Finetuned+Bi with the same reordered prompt to confirm there is no interaction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that replacing the causal mask with a prefix mask (Eq. 4) boosts MHQA performance—rests on a comparison in which the prompt places the question after the documents (Appendix C). In a causal decoder-only model, document tokens cannot attend to the question, so document hidden states are encoded without access to the query; the prefix mask allows all context tokens to attend bidirectionally, so documents become question-aware. The paper attributes the gain to bidirectional information flow between documents (Figure 1, Section 7.2), but the mask change also grants documents access to the question. A causal model prompted with the question before the documents would obtain question-aware document encodings via ordinary causal attention, and this baseline is not reported. If that baseline approaches Finetuned+Bi accuracy, the conclusion that 'the causal mask hinders multi-hop reasoning' is a prompt-layout artifact rather than a property of the mask; if the gap persists, the bidirectional document-document explanation is confirmed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how context permutation affects multi-hop QA in encoder-decoder (Flan-T5) and causal decoder-only (Qwen2.5, Llama 3.x) language models. It evaluates three manipulations: the order of gold documents (forward/backward/original), the distance between them, and their completeness (removing the first-hop document). The main empirical findings are that Flan-T5 models outperform similarly sized decoder-only models in zero-shot settings, fine-tuned decoder-only models prefer forward ordering, increasing inter-document distance hurts non-fine-tuned models, and replacing the causal mask with a prefix mask (Eq. 4) improves accuracy and robustness. The paper further analyzes attention via an Information Contribution (IC) score, finding that higher peak IC scores correlate with correct answers, and uses the peak IC to select among 20 random document permutations, improving Qwen 7B accuracy from 28.6% to 33.7%. Experiments are conducted on MuSiQue and supplemented by 2WikiMultihopQA results in the appendix.","tokens_in":19990,"tokens_out":9715,"duration_ms":94552,"significance":"If the findings hold, the paper provides practical guidance for retrieval-augmented generation: document order and inter-document distance matter, and a simple prefix-mask fine-tuning recipe can make decoder-only readers more accurate and robust. The paper has notable strengths: it covers three model families across a range of sizes, validates core trends on two datasets, reports a Spearman/Kendall check showing that training-set order is not spuriously correlated with the forward/backward distinction, and includes a parametric-knowledge probe in Section 5.3. The code is released. The main risks are the prompt-layout confound in the Finetuned+Bi comparison and the lack of uncertainty quantification for the peak-IC selection heuristic; both are addressable with additional experiments.","major_comments":[{"comment":"The comparison between Finetuned and Finetuned+Bi is potentially confounded by the prompt layout. The prompt places documents before the question (Appendix C), so under a causal mask document tokens cannot attend to the question, whereas under the prefix mask defined by Eq. (4) all context tokens—depending on the definition of c—attend bidirectionally, which would make document encodings question-aware. The paper attributes the gains to bidirectional document-document attention (Section 7.2, Figure 1), but it does not report a causal-mask baseline with the question placed before the documents, which would make document encodings question-aware without changing the mask. If that baseline matches Finetuned+Bi, the conclusion that 'the causal mask hinders multi-hop reasoning' would instead be a prompt-layout artifact; if the gap persists, the bidirectional-document account is confirmed. Please clarify the value of c in Eq. (4) (does the prefix include the question tokens?) and add the question-first control, or soften the mechanistic claim accordingly.","section":"§4, Eq. (4), and Appendix C"},{"comment":"The peak-IC selection heuristic is discovered and evaluated on the same MuSiQue development set (2,417 queries), and the reported improvement (28.6% to 33.7% for Qwen 7B Answer Only) is a single point estimate with no confidence interval, bootstrap, or other significance assessment. Because the heuristic is arguably selected based on this very dataset, the improvement could reflect dev-set overfitting rather than a generalizable signal. The 2WikiMultihopQA results in Appendix H provide some cross-dataset transfer evidence, but they are also reported without variance. Please report paired bootstrap or per-question standard errors on MuSiQue (and ideally on 2Wiki), for example by computing the accuracy of the max-IC selection over repeated bootstrap samples of the 2,417 queries, and state whether the same heuristic was fixed before evaluating on MuSiQue or whether it was tuned there.","section":"§6.1 and Figure 5"}],"minor_comments":[{"comment":"The definitions of ΔB and ΔF are not explicitly stated; the text says they are 'performance differences between original documents and re-ordered backward and forward documents respectively,' but it is not clear whether the difference is new minus original or original minus new. Please state the sign convention explicitly (e.g., ΔB = Acc_backward − Acc_original).","section":"Table 1"},{"comment":"The set A is defined as 'the set of answer tokens in the prediction,' but when the prediction is not boxed (e.g., for CoT when models fail to follow the instruction), the definition of answer tokens is unclear. Please specify how answer tokens are identified in those cases.","section":"§3.3, Eq. (3)"},{"comment":"The abstract states that encoder-decoder models 'generally outperform' causal decoder-only LMs despite being smaller, but this comparison is only for the zero-shot Answer Only and CoT setups; fine-tuned decoder-only models such as Qwen 2.5 14B reach higher accuracy than Flan-T5 xxl. Please clarify that the superiority claim is for off-the-shelf/zero-shot use, not for fine-tuned settings.","section":"Abstract and §5.1"},{"comment":"In panel (a), the legend contains a typo: 'Inorrect' should be 'Incorrect'.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a thorough empirical study with a clear practical message, but the two major issues—the prompt-order confound in the prefix-mask experiment and the lack of statistical validation for the IC heuristic—need to be addressed before publication. The prompt-order control is particularly important because the paper's central mechanistic claim ('the causal mask hinders multi-hop reasoning') may be reinterpreted as a prompt-layout effect. The IC heuristic is a promising idea, but the current evidence is not sufficient to rule out dev-set overfitting. If the authors can run the additional control and add significance/uncertainty reporting, the paper would be suitable for a strong journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time, but don't take the bidirectional-mask result at face value: the prompt puts the question after the documents, so the prefix-mask gain is confounded with giving documents access to the question.\n\nWhat is actually new and good. The systematic comparison of causal vs. prefix-mask LoRA finetuning across Qwen and Llama sizes is real. The finding that finetuned decoder-only LMs develop a forward-order preference on MuSiQue, despite no order correlation in the training split, is solid and interesting. The peak-IC selection heuristic is a clever use of internal attention as a confidence signal, and the 20-shuffle selection experiment is a nice idea. Appendix F also earns credit: the authors show that another encoder-decoder model, T0, does not favor forward order, which partially walks back their own broader claim about encoder-decoder superiority and points to training data rather than architecture.\n\nThe soft spots, in proportion. The main problem is the missing control identified in the stress test. The prompt is \"Documents then Question\" (Appendix C). In a causal decoder-only model, document tokens cannot attend to the question, so the documents are encoded blind. The prefix mask changes that: documents become question-aware. The paper attributes the entire gain to bidirectional document-document information flow, but it never runs the obvious control: a finetuned causal model with the question placed before the documents. Without that baseline, the claim that \"the causal mask hinders multi-hop reasoning\" is not established. This is a fixable experiment, not a fatal flaw, but it should be required before publication.\n\nSecond, the peak-IC heuristic is discovered and evaluated on the same MuSiQue development set (2,417 queries). The 2WikiMultihopQA appendix helps, but those are also development subsets, and the heuristic setup (20 shuffles, rank selection) was presumably chosen on these data. The reported 28.6-to-33.7 accuracy gain needs out-of-sample confirmation on a held-out test set.\n\nThird, all results are point estimates with no significance or variance estimates. Given the consistency across model sizes and datasets, this is not fatal, but it makes small differences hard to interpret, and the 1-3 point gains from bidirectional attention could be within noise.\n\nWho this is for. Anyone working on RAG, multi-hop QA, or context ordering for decoder-only LMs. The practical guidance about forward order, distance, and placing important documents near the end is useful even if the mask conclusion is still conditional. The code is public, so replication is feasible.\n\nRecommendation. Send it to serious peer review. The paper deserves referee time; with a prompt-order control and a held-out evaluation of the heuristic, it would be a solid contribution. I would cite the order and distance findings, and possibly the heuristic with appropriate caveats.","headline":"The paper is worth your time, but don't take the bidirectional-mask result at face value: the prompt puts the question after the documents, so the prefix-mask gain is confounded with giving documents access to the question.","tokens_in":20599,"tokens_out":2386,"would_cite":true,"duration_ms":27669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Swapping the causal mask for bi-directional attention reliably improves multi-hop question answering across decoder-only language models.","keywords":["multi-hop question answering","causal mask","prefix mask","bi-directional attention","context permutation","Information Contribution score","retrieval-augmented generation","MuSiQue"],"falsifier":"Run the identical peak-IC reranking experiment on a held-out test set of MuSiQue or on a different multi-hop dataset (for instance, HotpotQA or 2WikiMultihopQA with fresh questions) and check whether the best-permutation accuracy still beats the average shuffled accuracy by the same margin; if the gap shrinks to near zero, the peak-IC signal does not generalise. A second check: train the prefix-masked models on a dataset where gold documents are explicitly in backward order and see whether the forward-order preference still emerges, which would test the claim that the bias is an emergent fine-tuning property.","tokens_in":19641,"feed_emoji":"🧠","tokens_out":2421,"duration_ms":24658,"temperature":0.7,"pith_summary":"This paper asks whether the causal mask that forces decoder-only language models to read contexts left-to-right is a genuine bottleneck for multi-hop question answering, where a model must combine evidence from several retrieved documents. The authors show that it is: fine-tuning the same models with a simple prefix mask, which lets earlier documents attend to later ones, improves accuracy on the MuSiQue benchmark for every model tested, while also making performance less sensitive to document order. They also find that when a model answers correctly, its attention on the final supporting document peaks higher, and they exploit that signal to pick the best of several shuffled contexts, lifting Qwen 7B accuracy from 28.6% to 33.7%. A sympathetic reader would take away a practical recipe for RAG systems: order documents along the reasoning chain, keep them close together, and consider replacing the causal mask with a prefix mask if fine-tuning is affordable.","feed_headline":"Bi-directional attention lifts multi-hop QA accuracy in every model tested","feed_subtitle":"A simple prefix-mask swap during fine-tuning beats the causal mask and makes answers less sensitive to document order.","key_machinery":"The central mechanism is the attention mask itself: a causal mask forbids earlier tokens from attending to later ones, and the paper replaces it with a prefix mask $\\mathbf{M}_{i,j}=0$ if $i\\ge j$ or if both $i,j$ are within the context, and $1$ otherwise, converting a decoder-only model into a prefix LM during fine-tuning. To analyse behaviour, the paper introduces grouped attention weights $\\mathrm{GA}_{l,h}(X,Y)$ that aggregate attention between token blocks, and the Information Contribution score $\\mathrm{IC}_l(d)$ that sums attention from answer tokens to a document across heads and layers. The peak IC score is used both to explain correctness and as a heuristic to select among context permutations.","core_discovery":"The paper's central claim is that the causal mask is a measurable and fixable limitation for multi-hop reasoning in decoder-only language models. Replacing the causal mask with a prefix mask (where every context token can attend to every other context token, while the question and answer region remain causally masked) and fine-tuning with LoRA yields consistent gains over ordinary fine-tuning: for example, Qwen2.5 7B rises from 58.05% to 62.96% and Llama3.1 8B from 63.51% to 65.48%, with smaller sensitivity to forward/backward document order. The paper further claims that encoder-decoder models such as Flan-T5 outperform similarly sized decoder-only models out of the box, that fine-tuned models prefer contexts where gold documents appear in reasoning-chain order, and that the distance between gold documents matters more than their order for non-fine-tuned models. Finally, the paper argues that the peak of the Information Contribution score -- the maximum attention a model gives to any single document while generating the answer -- is a signal of answer correctness, and that selecting the permutation with the highest peak score improves accuracy.","pith_inferences":["The peak-IC selection heuristic resembles a self-contained reranking method, and if it generalises beyond MuSiQue, it could be used to choose between multiple retrievals or generation drafts without any additional training or external scorers.","The finding that fine-tuning instils a forward-order bias, even when training data has no such order, suggests that the bias is an emergent inductive bias of the fine-tuning objective rather than a property of the base model; testing this on other reasoning datasets would clarify whether it is a general phenomenon.","Because the bi-directional gain is demonstrated with LoRA fine-tuning, one natural extension is to test the same prefix-mask modification at larger scale or with full fine-tuning, where the gap between causal and bi-directional reading may widen or narrow.","The paper evaluates peak-IC selection on the same development set used to discover the signal; a genuinely held-out test would settle whether the heuristic overfits to MuSiQue's particular answerable questions."],"forward_implications":["RAG pipelines that use off-the-shelf causal decoder-only readers should place high-relevance documents near the end of the context and minimise the distance between supporting documents, since non-fine-tuned models are order-invariant but distance-sensitive.","When fine-tuning is an option, replacing the causal mask with a prefix mask is a cheap, LoRA-compatible change that improves multi-hop accuracy and robustness to document order across model families.","The peak Information Contribution score can serve as a confidence or selection signal: among multiple context orderings, the one with the highest peak IC tends to give the correct answer more often.","Fine-tuned models, including those with bi-directional attention, still mostly fail to refuse answering when the first-hop document is missing, so evidence attribution and completeness checks remain necessary in deployed QA systems."],"supporting_citations":[{"why":"Supplies the MuSiQue dataset and its train/development splits, which are the experimental substrate for all accuracy measurements.","marker":"Trivedi et al. (2022)"},{"why":"Documents the 'lost in the middle' ordering effect that motivates the document-position experiments and the recommendation to place relevant documents near the end.","marker":"Liu et al. (2024)"},{"why":"Shows that premise order matters in logical reasoning, providing the prior finding the paper generalises to multi-hop QA with forward/backward permutations.","marker":"Chen et al. (2024)"},{"why":"Introduces the prefix LM / prefix mask as an architectural alternative to causal masking, which the Finetuned + Bi setup directly adapts.","marker":"Raffel et al. (2020)"},{"why":"Defines the Flan-T5 family of instruction-tuned encoder-decoder models used as the comparison baseline for encoder-decoder vs decoder-only performance.","marker":"Chung et al. (2024)"},{"why":"Provides the Qwen2.5 model family whose causal-mask limitation is diagnosed and whose prefix-masked variant is evaluated.","marker":"Qwen (2025)"},{"why":"Provides the Llama 3.x model family used as the second causal decoder-only baseline and for the bi-directional fine-tuning experiments.","marker":"Llama Team (2024)"}],"fun_headline_variants":["Prefix mask outperforms causal mask in multi-hop QA","Bi-directional attention fixes multi-hop QA reasoning","Swap causal mask for bi-directional in multi-hop QA","Multi-hop QA gains from letting contexts attend mutually","Causal mask swap improves multi-hop QA in decoder LMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the peak Information Contribution score observed on the MuSiQue development set is a stable, transferable signal of answer correctness, and that the improvement from selecting the highest-peak permutation will hold on genuinely new data rather than being an artifact of the same 2,417 queries used to discover the correlation.","fun_headline_variants_meta":{"raw":{"variants":["Prefix mask outperforms causal mask in multi-hop QA","Bi-directional attention fixes multi-hop QA reasoning","Swap causal mask for bi-directional in multi-hop QA","Multi-hop QA gains from letting contexts attend mutually","Causal mask swap improves multi-hop QA in decoder LMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000685,"raw_usage":{"total_tokens":3166,"prompt_tokens":1066,"completion_tokens":2100,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":682,"tokens_out":2100,"duration_ms":15617,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:48:27.225690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical peak-IC reranking experiment on a held-out test set of MuSiQue or on a different multi-hop dataset (for instance, HotpotQA or 2WikiMultihopQA with fresh questions) and check whether the best-permutation accuracy still beats the average shuffled accuracy by the same margin; if the gap shrinks to near zero, the peak-IC signal does not generalise. A second check: train the prefix-masked models on a dataset where gold documents are explicitly in backward order and see whether the forward-order preference still emerges, which would test the claim that the bias is an emergent fine-tuning property.","supporting_citations":[],"review_version":1}