{"id":"d7558af9-1f95-42ae-90f8-c42da9ac7147","arxiv_id":"2505.16806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"ESA-DGR combines two-way evidence self-alignment with dual-gated knowledge fusion and GRPO training to improve multi-hop question answering on HotpotQA, 2WikiMultiHopQA, and MuSiQue.","lead":"The paper introduces ESA-DGR, a training framework that helps large language models answer multi-step questions by selecting the most relevant evidence and merging it with the model's own knowledge. It reports accuracy gains of about 4% EM and 5% F1 over existing methods on three multi-hop QA benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GRPO reward in Algorithm 1 scores only format, order, and conflict; it never checks the golden answer, so Eq. (6) cannot train the reported EM/F1 gains. The central empirical claim is therefore unverified as described.","rationale":"The reader's CONDITIONAL verdict already identifies table errors, missing error bars, and causal overclaim. My stress-test finds a more fundamental, internal inconsistency that should be checked first: in the paper's own Algorithm 1, the reward used for GRPO is purely structural (format, order, conflict) and never consults the golden answer. Since Eq. (6) is the only term in L_final that directly rewards answer quality, and L_align does not involve answers, the described training objective cannot explain how the model learns to improve EM/F1. This is not a matter of consensus but of the method's internal coherence. I would not reject outright because the anonymous code may include an omitted answer reward; hence UNVERDICTED pending the code check. If the check confirms format-only rewards, the headline empirical claim is unsupported; if it reveals a gold-answer reward, the paper still needs fixes for the impossible recall value (336.3), duplicated DSLR rows, and missing significance tests, and the conditional verdict would stand.","tokens_in":16101,"tokens_out":6601,"duration_ms":57256,"concrete_test":"Inspect the released implementation (anonymous.4open.science/r/ESA-DGR-2BF8) and locate the reward computation feeding Eq. (6). If the reward includes exact-match/QA correctness or a gold-answer check, the concern is resolved. Then run a controlled HotpotQA experiment with the format-only reward from Algorithm 1 (no answer signal) and report EM/F1; if gains persist, reward mismatch is not fatal, and if they vanish, the paper's Algorithm 1 cannot explain the headline numbers.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is empirical: ESA-DGR's training must improve EM/F1. In the described method, the only answer-sensitive training signal is the GRPO objective J(θ) in Eq. (6), whose advantage A_i derives from rewards in Algorithm 1. Algorithm 1 returns r = α1·CHECKFORMAT + α2·CHECKORDER + α3·CHECKCONFLICT; none of these functions compares the completion with the golden answer. A perfectly formatted wrong answer and a correct answer receive identical rewards, so the group advantage carries no information about answer accuracy and the policy gradient cannot drive the reported +4 EM / +5 F1 gains. L_align (Eq. 3) aligns token-selection and hidden states with the LLM's attention/representations, not with gold answers, and no supervised answer loss appears in the full objective (Eq. 7). Thus, as written, the training procedure is internally inconsistent with the reported improvements. This is aggravated by tabular anomalies: LLaMA Table 3 lists a HotpotQA RAG recall of 336.3, and the DSLR baseline rows are numerically identical across the Qwen and LLaMA tables, suggesting unreliable reporting. A code check is needed before the empirical claim can be evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ESA-DGR, a framework for knowledge-intensive multi-step reasoning (KIMSR) built on a 'claim-select-reason' pipeline. A rationale information extraction (RIE) module selects tokens from retrieved evidence; the two-way evidence self-alignment (TW-ESA) module aligns the selected-token distribution and hidden states with the LLM's attention and representations; and a dual-gated reasoning enhancement (DGR) module fuses the strict selected-evidence representation with the fine-tuned and original LLM hidden states. The model is trained with an alignment loss plus GRPO-based policy optimization. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue with Qwen2.5-7B and LLaMA3.1-8B report improvements over ten baselines, with average gains of about 4 EM and 5 F1. The full text includes ablation, sensitivity, and efficiency analyses, an anonymous code release link, and an explicit Limitations section noting that the underlying mechanism needs further investigation.","tokens_in":16443,"tokens_out":6864,"duration_ms":55390,"significance":"If the reported gains are reproducible, the combination of token-level and hidden-state alignment with a two-level gating mechanism is a plausible and potentially useful recipe for improving retrieval-augmented multi-hop reasoning; the paper also evaluates on three standard benchmarks with an extensive baseline set and provides an anonymous code release, which are strengths. However, the significance is currently bounded by verification problems: the described GRPO reward does not use the golden answer, the tables contain impossible or identical values, and the 'causal' interpretation of the objective is not formally supported. I therefore cannot yet assess whether the central claim holds.","major_comments":[{"comment":"The advantage A_i in Eq. (6) is computed from the reward in Algorithm 1, whose three components (CHECKFORMAT, CHECKORDER, CHECKCONFLICT) never compare the completion with the gold answer. A polished but wrong answer receives the same reward as a correct one, so the policy gradient described cannot be the source of the reported +4 EM / +5 F1 gains. Since L_align in Eq. (3) also uses only LLM attention and hidden states rather than gold labels, the training procedure as written is internally inconsistent with the empirical claim. Please add an answer-correctness term to the reward (or a supervised answer loss) and rerun, or identify the missing answer-sensitive signal.","section":"Section 4.4 and Algorithm 1"},{"comment":"The Recall value 336.3 for RAG on HotpotQA exceeds 100 and is therefore not a valid percentage, and the entire DSLR row is numerically identical to the DSLR row in Table 1 for all three datasets, which is implausible across two different backbone models. These anomalies strongly suggest tabulation or copy-paste errors and undermine confidence in the reported comparisons; please correct the tables and provide the underlying runs or a reproducibility check.","section":"Table 3"},{"comment":"All results appear to be single-run numbers without error bars, confidence intervals, or significance tests, despite the abstract's claim that ESA-DGR 'significantly surpasses' state-of-the-art methods. Please report multiple seeds with mean and standard deviation and a paired test (e.g., bootstrap or signed test) for the headline EM/F1 differences, or temper the significance wording.","section":"Section 6.1 and Tables 1-3"},{"comment":"The lower bound I(a;Z_final) >= E[log P(a|Z_final)] - H(a) is just a conditional log-likelihood plus a constant, so maximizing it does not, by itself, enforce a 'causal mapping' from evidence to answer; the paper's causal language in the abstract and contributions is therefore not supported by the derivation. Either supply a formal argument for why the alignment and gating losses enforce causal selection, or soften the causal claims.","section":"Section 4.4, Eq. (5)"}],"minor_comments":[{"comment":"The structural parameters k=8 and maximum claim steps=3 are selected on the development data and then used for the final results; please state the model-selection protocol and whether the test numbers were produced only after fixing this configuration, to rule out selection bias.","section":"Section 6.4 and Table 4"},{"comment":"The case studies end with 'Incorrect answer' but the text does not identify which system produced these outputs; please clarify whether these are baseline failures or failures of ESA-DGR, as the current placement is ambiguous.","section":"Appendix F and Appendix G"},{"comment":"Section 4.2 describes lambda_5 as a Lagrange multiplier that 'should be estimated', while Section 6.3 treats it as a tuned loss weight; please clarify the estimation procedure and report the estimated or tuned value.","section":"Section 4.2 and Section 6.3"},{"comment":"There are several typos and inconsistencies: 'Figuurnov' should be 'Figurnov', 'MuSiQue' and 'Musique' are used interchangeably, and the example 'the capital of the United States is Washington' in Section 4.3 is confusingly phrased.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The anonymous code link should be checked before acceptance; if Algorithm 1 is faithfully implemented, the central empirical claim is unverified as written. The identical DSLR rows and impossible recall value need to be corrected and verified, and the revised version should include statistical support for the headline gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a reasonably well-structured framework for evidence-grounded multi-hop QA, but the training objective as described cannot produce the reported answer gains. The GRPO reward in Algorithm 1 only checks format, tag order, and internal conflict; it never compares the completion to the golden answer. So the +4 EM / +5 F1 improvements over state-of-the-art baselines are not supported by the method as written. That is the load-bearing problem.\n\nWhat is actually new: the two-way evidence self-alignment (TW-ESA) that aligns token selection with LLM attention and hidden states, combined with the dual-gated fusion of strict and LLM reasoning (DGR), is a sensible combination, and the ablation study is reasonably designed. The idea of using the strict model to constrain the LLM's evidence representation is worth exploring. The sensitivity analysis of the five loss weights and the structural parameters (k, max steps) is useful. The case studies, while anecdotal, illustrate the intended behavior.\n\nThe soft spots beyond the missing answer reward: the tables contain impossible values (Recall 336.3 for LLaMA RAG on HotpotQA) and the DSLR baseline rows are numerically identical across Qwen and LLaMA tables, which suggests copy-paste errors. There are no error bars or significance tests, so we cannot tell if the gains are stable. The claim that the mutual information bound in Eq. (5) yields a 'causal mapping' is overreach; what they actually maximize is the likelihood of the answer, which is not causal. The ablation w/o GRPO uses standard cross-entropy, which implies the full model drops answer supervision; that makes the missing answer reward even more puzzling.\n\nThe stress-test note holds up. The only way the reported gains make sense is if the actual code includes an answer-based reward that is omitted from Algorithm 1. The authors promise anonymous code; a referee should check that first. As written, the empirical claim is unverified.\n\nThis paper is for researchers working on retrieval-augmented reasoning and evidence extraction. It deserves a serious referee because the framework is coherent and the flaw may be a reporting error, but the referee should require code and corrected tables. My recommendation: send to peer review, but with a clear request for the training details and code. If the code confirms the missing answer reward, reject; if it reveals an answer-aware reward, the paper could be a solid, incremental contribution.","headline":"Plausible architecture, but the GRPO reward never checks the gold answer, so the reported gains are unexplained as written.","tokens_in":16967,"tokens_out":4112,"would_cite":false,"duration_ms":34210,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ESA-DGR improves knowledge-intensive multi-step reasoning by aligning evidence selection with the LLM's attention and hidden states, then fusing the evidence with the model's own knowledge through two learned gates.","keywords":["knowledge-intensive multi-step reasoning","retrieval-augmented generation","evidence extraction","self-alignment","dual-gated fusion","GRPO","multi-hop question answering","hallucination reduction"],"falsifier":"Run a sufficiency test on correctly answered examples: erase every token that ESA-DGR did not select from the input and regenerate the answer; if exact match and F1 stay flat, the selection mask is not causally load-bearing. As a control, replace the LLM attention target $\\alpha_{R,i}$ in $L_{\\mathrm{align}}$ with a random attention vector of matched entropy; if scores do not drop, the two-way alignment is not the source of the reported gains.","tokens_in":15925,"feed_emoji":"🧠","tokens_out":14566,"duration_ms":98724,"temperature":0.7,"pith_summary":"The paper proposes ESA-DGR, a training framework for knowledge-intensive multi-step reasoning that combines two ideas: a two-way evidence self-alignment module that links evidence selection to the LLM's attention and hidden states, and a dual-gated reasoning module that fuses selected evidence with knowledge the LLM already has. The authors claim this addresses the two main failure modes of retrieval-augmented reasoning: selecting semantically similar but logically irrelevant evidence, and hallucinating when retrieved evidence is incomplete or noisy. On HotpotQA, 2WikiMultiHopQA, and MuSiQue, they report average improvements of about 4 percentage points in exact match and 5 percentage points in F1 over prior LLM fine-tuning methods, with the largest absolute gains on HotpotQA and 2WikiMultiHopQA. If the result holds, ESA-DGR offers a concrete recipe for making multi-hop question answering both more accurate and more robust to imperfect evidence.","feed_headline":"Two-way evidence alignment lifts multi-hop QA by 4-5 points","feed_subtitle":"It fuses selected evidence with model knowledge, beating prior retrieval baselines on three benchmarks.","key_machinery":"The load-bearing object is the two-way evidence self-alignment loss in TW-ESA: $L_{\\mathrm{align}} = \\lambda_3 \\, \\mathrm{CE}(\\alpha_{R,i}, M_i) + \\lambda_4 \\, \\mathrm{JS}(Z_{R,i} \\| Z_{U,i}) + \\lambda_5 L_s$, where $M_i$ is the binary token-selection mask over the retrieved evidence, $\\alpha_{R,i}$ is the LLM's attention over the full evidence, $Z_{R,i}$ and $Z_{U,i}$ are hidden states of the full and selected evidence, and $L_s$ is a sparsity-and-continuity regularizer on the mask. This loss pushes the selector to mimic where the LLM looks, and pushes the full-evidence representation to match the selected-evidence representation, which is the mechanism claimed to teach causal relevance rather than semantic similarity. The dual-gated mechanism then computes two learned gates: the first fuses $Z_{U,i}$ with $Z_{R,i}$, and the second fuses that result with the original LLM hidden state $Z_i$, producing the final state $Z_{\\mathrm{final}}$. A GRPO objective with a structure-aware reward supplies the reasoning-quality signal, and a KL term keeps the second gate from drifting too far from first-level reasoning.","core_discovery":"ESA-DGR's central claim is that the two known weaknesses of retrieval-augmented multi-hop reasoning—semantic-logic mismatch in evidence extraction and uncertainty-aware hallucination—can be treated as alignment problems. The TW-ESA module trains a rationale information extraction module by aligning the token-selection distribution with the LLM's attention over the full evidence, and by aligning the hidden states computed from full evidence with those computed from selected evidence. The DGR module then combines three sources: the strict model over selected evidence, the LLM-enhanced representation over full evidence, and the original LLM's hidden state, with two learned gates that decide how much of each source to trust. The whole framework is trained jointly with a GRPO objective and a KL regularizer. On three multi-hop QA benchmarks the authors report the highest exact match and F1 among compared methods, including +5.5 EM/+6.1 F1 on HotpotQA and +4.3 EM/+5.7 F1 on 2WikiMultiHopQA over the previous best baselines, with additional gains on MuSiQue.","pith_inferences":["The same two-way alignment could be lifted out of the GRPO training loop and used as a plug-in evidence selector for existing retrieval-augmented systems; if the causal-alignment claim is right, it should improve accuracy even without the policy-optimization component.","A stronger test of the paper's causal story would compare selected evidence against minimal human-authored rationale sets; the paper reports separation statistics and evidence-quality scores, but not whether the unselected tokens are individually unnecessary for the answer.","The dual-gated fusion recipe generalizes beyond multi-hop QA: any task with noisy retrieved evidence and a model with useful parametric knowledge could use the same two-gate structure to decide when to trust the evidence and when to trust the model.","Because the reported gains are smallest on MuSiQue, the framework's benefit appears tied to settings where retrieved passages are compact and reasoning chains are long; applying it to tasks with very long or multi-document evidence may require re-tuning the gates."],"forward_implications":["On HotpotQA and 2WikiMultiHopQA, ESA-DGR reports the largest gains over prior methods: +5.5 EM/+6.1 F1 and +4.3 EM/+5.7 F1 respectively, with additional gains on MuSiQue.","ESA-DGR achieves higher answer quality with a comparable or lower average number of retrieval queries than Search-o1, so the alignment and gating modules improve retrieval efficiency rather than trading accuracy for more search.","The extracted rationale tokens separate more cleanly from non-rationale tokens in representation space and receive higher evidence-quality scores than SEER, which supports the claim of better interpretability.","Ablation results show that removing either alignment loss, either gate, or the GRPO objective lowers performance, with hidden-state alignment and the second gate among the largest individual contributors."],"supporting_citations":[{"why":"Supplies the IRCoT setup ESA-DGR builds on: interleaved retrieval with chain-of-thought, BM25 indices, and the retrieve-on-claim pipeline.","marker":"Trivedi et al., 2023"},{"why":"SEER is the main self-aligned evidence-extraction baseline and the source of the evidence-quality protocol used to compare rationale extraction.","marker":"Zhao et al., 2024b"},{"why":"Search-o1 is the strongest retrieval-based baseline and the system ESA-DGR is compared against on answer quality and query efficiency.","marker":"Li et al., 2025"},{"why":"GRPO is the policy-optimization objective that drives the collaborative training of the evidence selector and the gated reasoner.","marker":"Shao et al., 2024"},{"why":"mPLUG-Owl2's modal feature alignment motivates the two-way evidence self-alignment design.","marker":"Ye et al., 2024"},{"why":"Provides the differentiable binary-selection reparameterization and the sparsity/continuity regularizer used by the rationale extraction module.","marker":"Bastings et al., 2019"},{"why":"Supplies the implicit reparameterization gradient used to train the hard token-selection indicators.","marker":"Figuurnov et al., 2018"},{"why":"DARE contributes the selection regularizer and is also a rationale-extraction baseline ESA-DGR must outperform.","marker":"Yue et al., 2022"},{"why":"Supports the premise that LLM hidden states encode factual knowledge, which justifies fusing the original LLM hidden state in the second gate.","marker":"Azaria and Mitchell, 2023"},{"why":"Adds evidence that LLM hidden states contain factual knowledge usable for reasoning, reinforcing the dual-gated fusion design.","marker":"Hu et al., 2024"}],"fun_headline_variants":["Two-way evidence alignment lifts multi-hop QA by 4% EM and 5% F1","Dual-gated reasoning fuses evidence and LLM knowledge for +4 EM, +5 F1","Self-aligned causal evidence beats baselines across three QA benchmarks","ESA-DGR: gated evidence fusion lifts multi-hop reasoning by 4-5 points","Aligning evidence tokens yields 5.5 EM and 6.1 F1 gains on HotpotQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework rests on the assumption that aligning the evidence selector with the LLM's attention, and aligning the two evidence representations, trains the model to pick causally relevant evidence rather than merely statistically similar text.","fun_headline_variants_meta":{"raw":{"variants":["Two-way evidence alignment lifts multi-hop QA by 4% EM and 5% F1","Dual-gated reasoning fuses evidence and LLM knowledge for +4 EM, +5 F1","Self-aligned causal evidence beats baselines across three QA benchmarks","ESA-DGR: gated evidence fusion lifts multi-hop reasoning by 4-5 points","Aligning evidence tokens yields 5.5 EM and 6.1 F1 gains on HotpotQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3485,"prompt_tokens":992,"completion_tokens":2493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2375}},"tokens_in":608,"tokens_out":2493,"duration_ms":14637,"temperature":1.0,"reasoning_tokens":2375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:46.231827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a sufficiency test on correctly answered examples: erase every token that ESA-DGR did not select from the input and regenerate the answer; if exact match and F1 stay flat, the selection mask is not causally load-bearing. As a control, replace the LLM attention target $\\alpha_{R,i}$ in $L_{\\mathrm{align}}$ with a random attention vector of matched entropy; if scores do not drop, the two-way alignment is not the source of the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the differentiable binary-selection reparameterization and the sparsity/continuity regularizer used by the rationale extraction module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the implicit reparameterization gradient used to train the hard token-selection indicators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DARE contributes the selection regularizer and is also a rationale-extraction baseline ESA-DGR must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the premise that LLM hidden states encode factual knowledge, which justifies fusing the original LLM hidden state in the second gate."}],"review_version":1}