REVIEW 3 major objections 7 minor 18 references
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · glm-5.2
Learned controller beats fixed pipelines for multi-hop evidence retrieval
2026-07-08 03:22 UTC pith:K5BPEMOC
load-bearing objection Solid systems paper with a real contribution, but the headline claim rests on a 0.62-point margin with no variance reported. the 3 major comments →
DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Generation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central result is that a learned, state-conditioned policy over atomic evidence operations outperforms both fixed RAG pipelines and uniform selection among the same operations. The validity-utility separation is the load-bearing design: deterministic rules prevent undefined or premature transitions, while a random-forest regressor trained on support-recall changes ranks only the feasible actions. This controller converges on a stable action pattern across datasets -- frontier retrieval, one sufficiency check, then gap-focused queries -- rather than collapsing to repeated retrieval. The policy trained on Qwen2.5-7B trajectories transfers to GPT-4o-mini and Llama-3.1-8B without re-
What carries the argument
The action-value model (Eq. 1) assigns each valid state-action pair a score: acquisition actions receive the change in annotated support recall, sufficiency checking receives an indicator for full support, and stopping receives current support recall. A random-forest regressor fits these labels offline. At inference, the controller selects the highest-scoring valid action minus a cost term (set to zero in main runs). A separate continuation model can suppress premature stopping. The validity layer enforces preconditions: gap queries require a recorded gap, bridge expansion requires a detected bridge entity, retrieval actions are removed after the budget is exhausted, and so on.
Load-bearing premise
The controller is trained on changes in annotated supporting-document recall as a proxy for answer quality. If support recall and answer quality diverge -- for example, when retrieved support documents contain the right facts but the generator fails to use them, or when non-support documents contribute to correct answers -- the learned policy optimizes the wrong objective. The MuSiQue four-hop result, where DynaKRAG underperforms the strongest baseline, is consistent with the
What would settle it
If the correlation between support-recall changes and F1 changes is low at the trajectory level, or if non-support documents frequently contribute to correct answers, the training signal would be optimizing a proxy that diverges from the actual objective of answer quality.
If this is right
- The transfer of a Qwen-trained policy to GPT-4o-mini and Llama-3.1-8B without retraining suggests that evidence-acquisition preferences are partially model-independent, which could reduce the cost of deploying adaptive RAG across new generators.
- The finding that MuSiQue peaks at two retrieval calls and degrades at three directly challenges the assumption that more evidence is always better; retrieval-budget optimization should be state-dependent rather than fixed.
- The learned controller's convergence on gap-focused queries after sufficiency checks -- rather than query rewriting or bridge expansion -- implies that explicit diagnostic feedback may be more valuable than reformulation for multi-hop evidence acquisition.
- The validity-utility separation pattern could generalize to other sequential decision problems where some actions are structurally infeasible in certain states, reducing the learning problem to ranking among feasible options.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DynaKRAG, a framework that formulates multi-hop evidence acquisition as state-conditioned control over atomic evidence operations (retrieval, query rewriting, gap-directed retrieval, sufficiency checking, stopping, and terminal compression). A hard validity layer constructs the executable action set at each step, and a learned random-forest action-value model ranks valid actions. The action-value model is trained on changes in annotated supporting-document recall (SR). Experiments on HotpotQA, 2Wiki, and MuSiQue with Qwen2.5-7B, GPT-4o-mini, and Llama-3.1-8B show gains over controlled baselines sharing the same backbone, corpus, and retriever, alongside token reductions relative to S2G-RAG. Ablations isolate the contributions of the learned controller, sufficiency feedback, and terminal compression. A retrieval-cap sweep demonstrates that additional retrieval is not uniformly beneficial.
Significance. The paper addresses a genuine gap: most adaptive RAG systems fix their control topology, and the question of how to learn a shared policy that selects among heterogeneous evidence operations is well-motivated. The experimental design is a strength—all baselines share the same backbone, corpus, retriever, and evaluation pipeline, and the ablations cleanly separate the learned controller, sufficiency check, and compression. The retrieval-cap sweep provides a falsifiable test of the 'more retrieval is better' alternative. The cross-backbone transfer experiment (Qwen-trained policy applied to GPT-4o-mini and Llama-3.1-8B) is a useful test of controller generality. The MuSiQue four-hop breakdown (where DynaKRAG underperforms S2G-RAG) is reported transparently.
major comments (3)
- §Experiments, Table 1 (MuSiQue, Qwen2.5-7B): The headline claim is 'outperforming the strongest controlled baseline on all three benchmarks.' On MuSiQue with Qwen2.5-7B, DynaKRAG scores 0.3061 vs S2G-RAG's 0.2999 — a margin of 0.62 F1 points on 2,417 examples. The paper uses a single seed (seed 13) with deterministic decoding and reports no variance, confidence intervals, or significance tests. Randomness remains in random-forest fitting and retrieval nondeterminism. A 0.62-point difference could fall within run-to-run variance. If the MuSiQue difference is not statistically significant, the 'all three benchmarks' claim does not hold. The paper should either (a) report multi-seed variance and a significance test for this specific comparison, or (b) qualify the claim to 'two of three benchmarks with a competitive result on the third.' This is load-bearing because the abstract and main结果s1
- §Method, Eq. (1): The training signal for the action-value model is the change in annotated supporting-document recall (SR). The paper assumes SR improvement is a good proxy for answer-quality improvement but does not report the trajectory-level correlation between SR changes and F1 changes. The MuSiQue four-hop result (Appendix: DynaKRAG 0.1798 vs S2G-RAG 0.2060 F1) is consistent with this proxy degrading on harder compositions — the controller may improve support recall without improving answer quality, or non-support documents may contribute to correct answers. A diagnostic showing the SR–F1 correlation, or at least acknowledging this limitation and its connection to the four-hop underperformance, would strengthen the paper. This is load-bearing because the entire learned policy optimizes SR as a surrogate for answer quality.
- §Experiments, Table 1 (Llama-3.1-8B, 2Wiki): The paper states DynaKRAG 'leads on five and remains competitive on the sixth' of six non-Qwen dataset–backbone combinations. On Llama 2Wiki, DynaKRAG scores 0.3933 F1 vs CRAG's 0.3668 — but the text says this is 'within 0.38 points of the best baseline,' implying CRAG is the best baseline at 0.3668, which would make DynaKRAG the leader by 2.65 points, not 0.38 behind. This inconsistency needs clarification: either the comparison baseline is different from what the table shows, or the text contains an error. The 'lightweight inference-time calibration' mentioned for this result is not described in the Method section; its nature and justification should be specified.
minor comments (7)
- §Method, Eq. (2): The main configurations use λ=0, meaning cost does not affect action ranking. The paper should clarify earlier in the Method section (not just in the Appendix) that cost-sensitive optimization is not evaluated in the main results, as the framing in the Introduction emphasizes cost-effectiveness.
- Table 1: The 'Tok.' column for fixed-K uses a different accounting convention (prompt tokens) than iterative methods (total generated-run tokens), as noted in the Appendix. This makes direct token comparisons between fixed-K and other methods misleading. A footnote or annotation in the table itself would help readers.
- Figure 3: The y-axis scale for MuSiQue (0.18–0.30) differs from HotpotQA (0.50–0.60) and 2Wiki (0.40–0.55). While the per-dataset scaling is understandable, a note clarifying that scales differ would prevent misreading the relative magnitude of the cap-3 decline on MuSiQue.
- §Related Work: The references to S2G-RAG (Li et al. 2026a) and PAR2-RAG (Li et al. 2026b) cite arXiv papers from 2026. If these are concurrent or unpublished, the relationship to the present work should be clarified.
- §Method, Terminal Evidence Compression: The dataset-specific compression parameters (HotpotQA/2Wiki: 8 snippets, 256 tokens; MuSiQue: 12 snippets, 384 tokens) are described only in the Appendix. A brief mention in the main text would improve reproducibility for readers who skip the Appendix.
- Table 2: The 'Fixed-K static reference' row is included as a reference point, not a single-component ablation. This is explained in the text but could be more clearly marked in the table itself (e.g., with a separator or footnote).
- §Experiments: The paper states 'The main runs have no explicit retrieval-call cap and may issue a fourth retrieval on some examples.' The relationship between this statement and the 'Maximum control steps: 4' setting (Table 4) could be made clearer — specifically, whether the fourth step is always a retrieval or can be a sufficiency check or other action.
Simulated Author's Rebuttal
We thank the referee for the careful and constructive review. The referee raises three major comments: (1) the MuSiQue/Qwen2.5-7B margin of 0.62 F1 is reported without variance or significance testing, making the 'all three benchmarks' claim potentially unsupported; (2) the SR-to-F1 proxy assumption underlying the training signal is not validated, and the MuSiQue four-hop underperformance may be consistent with proxy degradation; and (3) an inconsistency in the Llama 2Wiki comparison text, plus an undescribed 'lightweight inference-time calibration.' We address each below.
read point-by-point responses
-
Referee: §Experiments, Table 1 (MuSiQue, Qwen2.5-7B): The headline claim is 'outperforming the strongest controlled baseline on all three benchmarks.' On MuSiQue with Qwen2.5-7B, DynaKRAG scores 0.3061 vs S2G-RAG's 0.2999 — a margin of 0.62 F1 points on 2,417 examples. The paper uses a single seed (seed 13) with deterministic decoding and reports no variance, confidence intervals, or significance tests. Randomness remains in random-forest fitting and retrieval nondeterminism. A 0.62-point difference could fall within run-to-run variance. If the MuSiQue difference is not statistically significant, the 'all three benchmarks' claim does not hold. The paper should either (a) report multi-seed variance and a significance test for this specific comparison, or (b) qualify the claim to 'two of three benchmarks with a competitive result on the third.' This is load-bearing because the abstract and main结果s1
Authors: The referee is correct that the 0.62-point MuSiQue margin is small and that we report no variance or significance test. We will address this in revision. Specifically, we plan to: (1) run the MuSiQue/Qwen2.5-7B configuration across multiple random seeds (varying the random-forest initialization seed) and report the resulting F1 distribution; (2) perform a paired bootstrap significance test between DynaKRAG and S2G-RAG on the 2,417 MuSiQue examples. We will report these results in the revised paper. If the difference is not statistically significant, we will qualify the abstract and main-results claim accordingly—for example, to 'outperforming the strongest controlled baseline on two of three benchmarks, with a competitive result on MuSiQue.' We agree this is load-bearing for the headline claim and appreciate the referee flagging it. revision: yes
-
Referee: §Method, Eq. (1): The training signal for the action-value model is the change in annotated supporting-document recall (SR). The paper assumes SR improvement is a good proxy for answer-quality improvement but does not report the trajectory-level correlation between SR changes and F1 changes. The MuSiQue four-hop result (Appendix: DynaKRAG 0.1798 vs S2G-RAG 0.2060 F1) is consistent with this proxy degrading on harder compositions — the controller may improve support recall without improving answer quality, or non-support documents may contribute to correct answers. A diagnostic showing the SR–F1 correlation, or at least acknowledging this limitation and its connection to the four-hop underperformance, would strengthen the paper. This is load-bearing because the entire learned policy optimizes SR as a surrogate for answer quality.
Authors: This is a fair and important point. The proxy assumption—that improving support recall improves answer quality—is central to the learned policy, and we have not explicitly validated it. We will add a diagnostic in the revision: for each dataset, we will compute the trajectory-level correlation between SR changes (the training target) and F1 changes (the evaluation metric) across training examples. We expect this correlation to be positive overall but potentially weaker on harder compositions, which would be consistent with the MuSiQue four-hop underperformance the referee identifies. We will also add an explicit discussion of this limitation in the Method section, noting that SR is an imperfect surrogate for answer quality, that non-support documents may contribute to correct answers, and that this proxy may degrade on longer evidence chains—connecting it to the four-hop result. We cannot fully resolve the proxy limitation within this revision (switching to an F1-based reward would require a fundamentally different training procedure), but we can make the limitation transparent and provide the requested diagnostic. revision: partial
-
Referee: §Experiments, Table 1 (Llama-3.1-8B, 2Wiki): The paper states DynaKRAG 'leads on five and remains competitive on the sixth' of six non-Qwen dataset–backbone combinations. On Llama 2Wiki, DynaKRAG scores 0.3933 F1 vs CRAG's 0.3668 — but the text says this is 'within 0.38 points of the best baseline,' implying CRAG is the best baseline at 0.3668, which would make DynaKRAG the leader by 2.65 points, not 0.38 behind. This inconsistency needs clarification: either the comparison baseline is different from what the table shows, or the text contains an error. The 'lightweight inference-time calibration' mentioned for this result is not described in the Method section; its nature and justification should be specified.
Authors: The referee has identified a genuine inconsistency. Examining Table 1 for the Llama-3.1-8B / 2Wiki block: DynaKRAG scores 0.3933 F1, while the best baseline is CRAG at 0.3668. DynaKRAG therefore leads by 2.65 points, not trails by 0.38. The text stating 'within 0.38 points of the best baseline' is incorrect. The source of the error appears to be a comparison against a different baseline or a leftover figure from an earlier draft. We will correct the text to accurately reflect that DynaKRAG leads on all six non-Qwen dataset–backbone combinations, not five of six. Regarding the 'lightweight inference-time calibration' mentioned in the table footnote: this refers to a minor adjustment of the continuation model's decision threshold for the Llama backbone, applied because Llama-3.1-8B's evidence-state distributions differ from Qwen2.5-7B's (the training backbone). We will add a clear description of this calibration in the Method or Implementation Details section, including its nature (threshold adjustment only, no retraining), its justification (cross-backbone distribution shift), and its scope (applied only to the Llama 2Wiki configuration). We thank the referee for catching both issues. revision: yes
Circularity Check
No circularity found: training targets use external gold annotations, inference uses only observable state, and no self-citation chain is load-bearing.
full rationale
The paper's derivation chain is self-contained. The action-value model (Eq. 1) is trained on changes in support recall (SR), computed from gold supporting-document annotations. At inference time (Eq. 2, Algorithm 1), the controller selects actions using only observable evidence-state features; the paper explicitly states 'Gold answers, answer scores, supporting facts, and support recall are never runtime features.' The training signal (SR from gold annotations) and the inference signal (observable state features) are distinct, so the 'prediction' at test time is not equivalent to the training labels by construction. The ablation replacing the learned controller with uniform-valid selection (Table 2) provides an independent baseline showing the learned ranking contributes measurably. No self-citation chain is load-bearing for the central claim: the paper cites prior RAG methods (S2G-RAG, PAR2-RAG, CoRAG, etc.) as baselines, not as premises that force its conclusion. The framework's validity layer and action space are defined independently of any cited result. The MuSiQue four-hop underperformance (0.1798 vs S2G-RAG's 0.2060) is reported transparently and is inconsistent with a circular construction that would guarantee gains. The single-seed and small-margin concerns raised by the skeptic are statistical fragility issues, not circularity. No step in the derivation reduces to its inputs by definition.
Axiom & Free-Parameter Ledger
free parameters (9)
- λ (cost weight) =
0
- Max control steps =
4
- Documents per acquisition =
3
- Max final context =
12
- Random forest trees =
300
- Random forest min leaf =
8
- Compression token limits =
256 (HotpotQA/2Wiki), 384 (MuSiQue)
- Continuation threshold =
permissive (exact value not stated)
- Training examples per dataset =
1000
axioms (4)
- domain assumption Support-document recall improvement is a valid proxy for answer-quality improvement.
- domain assumption The seven atomic operations (retrieve_more, gap_query, rewrite_query, bridge_entity_expand, sufficiency_check, stop_answer, compress_answer_evidence) are sufficient to cover the evidence-acquisition needs of multi-hop QA.
- domain assumption A random-forest regressor can capture the state-action value function for evidence acquisition.
- domain assumption Dataset-specific controllers trained on 1,000 examples generalize to the full evaluation split.
invented entities (2)
-
Evidence state representation
independent evidence
-
Validity layer
independent evidence
read the original abstract
Multi-hop retrieval-augmented generation (RAG) acquires evidence sequentially, with each new document potentially revealing missing facts, bridge entities, query defects, or sufficient support for answering. Existing methods provide useful operations such as iterative retrieval, query reformulation, evidence critique, and sufficiency judging, but typically organize them within method-specific pipelines or predefined control topologies. This leaves underexplored how to learn a shared state-conditioned policy that chooses among currently valid evidence operations. We introduce DynaKRAG, which formulates multi-hop evidence acquisition as state-conditioned control over atomic evidence operations. At each step, a validity layer constructs the executable action set, and a learned controller selects the next operation. The resulting transition updates the evidence state and may enable new operations at subsequent steps. With Qwen2.5-7B-Instruct, DynaKRAG achieves F1 scores of 0.5998 on HotpotQA, 0.5340 on 2Wiki, and 0.3061 on MuSiQue, outperforming the strongest controlled baseline on all three benchmarks. Replacing the learned controller with a uniform-valid policy reduces F1 by 3.96--5.78 points, while removing sufficiency feedback hurts all three datasets. Controlled retrieval-cap experiments further show that additional retrieval is not uniformly beneficial. Together, these results demonstrate the benefit of coordinating retrieval, diagnosis, and gap-directed acquisition under an evolving evidence state.
Figures
Reference graph
Works this paper leans on
-
[1]
Retrieval-Augmented Generation for Knowledge-Intensive
Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , year =
-
[2]
and Salakhutdinov, Ruslan and Manning, Christopher D
Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle =. 2018 , pages =
work page 2018
-
[3]
Ho, Xanh and Nguyen, Anh-Khoa Duong and Sugawara, Saku and Aizawa, Akiko , booktitle =. Constructing A Multi-Hop. 2020 , pages =
work page 2020
-
[4]
Trivedi, Harsh and Balasubramanian, Niranjan and Khot, Tushar and Sabharwal, Ashish , journal =. 2022 , pages =
work page 2022
-
[5]
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year =
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year =
-
[6]
International Conference on Learning Representations , year =
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval , author =. International Conference on Learning Representations , year =
-
[7]
Findings of the Association for Computational Linguistics: EMNLP 2023 , year =
Measuring and Narrowing the Compositionality Gap in Language Models , author =. Findings of the Association for Computational Linguistics: EMNLP 2023 , year =
work page 2023
-
[8]
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =
Active Retrieval Augmented Generation , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =
work page 2023
-
[9]
Su, Weihang and Tang, Yichen and Ai, Qingyao and Wu, Zhijing and Liu, Yiqun , booktitle =. 2024 , pages =
work page 2024
-
[10]
Chan, Chi-Min and Xu, Chunpu and Yuan, Ruibin and Luo, Hongyin and Xue, Wei and Guo, Yike and Fu, Jie , booktitle =. 2024 , url =
work page 2024
-
[11]
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle=. 2023 , url=
work page 2023
-
[12]
Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle =. 2024 , url =
work page 2024
- [13]
-
[14]
Jeong, Soyeong and Baek, Jinheon and Cho, Sukmin and Hwang, Sung Ju and Park, Jong , booktitle =. 2024 , pages =
work page 2024
-
[15]
Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighoff, Niklas and Lian, Defu and Nie, Jian-Yun , year =
-
[16]
Advances in Neural Information Processing Systems , year =
Chain-of-Retrieval Augmented Generation , author =. Advances in Neural Information Processing Systems , year =
-
[17]
Li, Minghan and Zou, Junjie and Lv, Xinxuan and Zhang, Chao and Zhou, Guodong , year =
-
[18]
Li, Xingyu and Wang, Rongguang and Wang, Yuying and Guo, Mengqing and Li, Chenyang and Sheng, Tao and Ravi, Sujith and Roth, Dan , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.