{"id":"1a3bbe2c-d30a-4cc2-b9c5-3778ceaea941","arxiv_id":"2505.19112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SiGIR trains a language model to decompose multi-hop questions, self-critique each retrieval and reasoning step, and use cumulative self-rewards in a beam search, beating prior methods by up to 14.4 F1 points on MuSiQue.","lead":"The paper introduces SiGIR, a system that answers multi-step questions by breaking them into smaller pieces, checking each step with its own critique, and searching over reasoning paths with self-assigned scores. It reports large F1 gains over prior methods on three question-answering benchmarks and shows a smaller model can approach full-data performance using self-generated training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval corpus in Appendix A.4 is not open-domain and the cited SOTA baselines were not run in it, so the 8.6% comparison may mix incompatible settings.","rationale":"The reader's conditional verdict is well-founded, but the condition needs to be stronger. The paper's only description of retrieval (A.4) describes a curated corpus, contradicting Section 4.1's 'open-domain' claim; moreover, two of the three SOTA baselines used in the 8.6% calculation are derived from other papers with unstated retrieval settings. My proposed check settles whether the headline comparison is fair. If the check shows the gold paragraphs are in the index and the derived baselines came from full-corpus retrieval, the paper must re-run those baselines under the same setup or weaken its claim; if it shows the setups match, the claim stands. I therefore keep the reader's CONDITIONAL verdict, with the added condition of demonstrated retrieval equivalence. I do not see a more load-bearing issue: the method's components are standard, the ablations are consistent, and the training/inference procedures are clearly specified apart from the retrieval corpus ambiguity.","tokens_in":22737,"tokens_out":11004,"duration_ms":97567,"concrete_test":"Inspect the released repository's retriever/index construction script. For a sample of 100 development questions, verify whether the gold evidence paragraphs are in the BM25/Contriever index and measure the recall of gold paragraphs in the top-10 retrieved results. Then check the original papers of BeamAggR and RAG-Star (and the provided configuration files) to determine what retrieval corpus those published numbers used. If the gold paragraphs are indexed and those baselines used full Wikipedia, the evaluation is not open-domain and the 8.6% claim must be recomputed under a single shared retrieval corpus.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 says 'The experiments are conducted in the open-domain setting,' but Appendix A.4 says 'we construct a retrieval corpus using supporting facts and irrelevant documents from the training instances, and conduct retrieval within this scope.' That is a small, curated pool, not a Wikipedia-scale open-domain index. The paper never states whether development-set gold paragraphs are added to this pool, never reports pool size, and never reports gold recall. If the gold paragraphs are in the pool, the task is not open-domain; if they are not, the reported F1 could not be achieved for unseen dev questions. The internal inconsistency alone is enough to question the headline. The problem is compounded by Table 1's note that ProbTree, BeamAggR, RAG-Star, and GenGround results are 'derived from the original paper.' Those numbers were produced under the retrieval setups of their respective papers, which are not shown to match the restricted pool described in A.4. On HotpotQA and MuSiQue, the previous SOTA used in the 8.6% calculation (RAG-Star at 60.00 and BeamAggR at 32.30) are both derived numbers. Without evidence that those baselines retrieved from the same corpus, the reported improvements are not apples-to-apples, and the central claim that SiGIR 'surpasses the previous SOTA by 8.6%' is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SiGIR, a training and inference framework for multi-hop question answering in which a single model iteratively decomposes the question into atomic sub-questions, triggers external retrieval, performs retrieval-augmented reasoning, emits self-critique rewards for retrieval relevance and reasoning utility, and uses these rewards in a beam-style search to keep the most promising trajectories. The model is obtained by SFT on synthetic iterative reasoning rationales produced by DeepSeek-V2.5, with a separate distilled critic used to annotate intermediate and outcome rewards and then merged into the reasoner. On HotpotQA, 2WikiMQA, and MuSiQue the authors report token-level F1 of 63.09, 74.47, and 37.15 with Mistral-7B, claiming an average relative improvement of 8.6% over previous state-of-the-art methods. The paper also includes ablations on search, reward granularity, reward modeling, retrieval systems, and self-improvement data synthesis, and it releases code, data, and models.","tokens_in":23035,"tokens_out":5154,"duration_ms":35252,"significance":"If the reported evaluation setup is valid, SiGIR is a practically relevant contribution: the idea of using a single model for iterative decomposition, retrieval, reasoning, and self-critique, with cumulative process rewards guiding a small beam search, is clearly presented, and the ablations (especially the comparison of cumulative vs. outcome reward selection and the training-scaling analysis) give useful insights. The release of code, data, and models, and the moderate training cost (about 3 hours on 4 A100 GPUs) are strengths. However, the significance of the headline 8.6% claim depends entirely on the fairness of the evaluation, which is currently not established (see major comments). The method appears internally consistent and the circularity risk is low because final scoring uses external ground-truth answers, but the numerical claims need to be placed on solid evaluative ground before the contribution can be assessed.","major_comments":[{"comment":"The paper states in Section 4.1 that 'The experiments are conducted in the open-domain setting,' but Appendix A.4 describes a retrieval corpus built from 'supporting facts and irrelevant documents from the training instances.' This is an in-domain, closed retrieval pool, not a Wikipedia-scale open-domain index. The paper never states whether development-set gold paragraphs are added to this pool, never reports the pool size, and never reports gold recall. If dev gold paragraphs are absent, the reported F1 values could not be achieved for unseen dev questions; if they are present, the task is not open-domain. The authors must specify the exact composition of the retrieval pool, report gold recall, and either re-run the experiments in a genuine open-domain setting or rename the task (e.g., 'in-scope retrieval') and adjust the claims accordingly.","section":"4.1 vs. A.4"},{"comment":"The footnote ':' indicates that the numbers for ProbTree, BeamAggR, RAG-Star, and GenGround are 'derived from the original paper' and were not obtained under the retrieval setup described in Appendix A.4. In particular, the previous-SOTA values used in the 8.6% calculation on HotpotQA and MuSiQue (RAG-Star 60.00 and BeamAggR 32.30) come from different retrieval environments. Without evidence that those baselines used the same retrieval corpus (or a comparable open-domain index), the comparison is not apples-to-apples, and the central claim that SiGIR 'surpasses the previous SOTA by 8.6%' is unsupported. The authors should re-run these baselines in the same retrieval setting or, failing that, clearly restrict the scope of the comparison and re-compute the claimed improvement.","section":"Table 1"},{"comment":"All reported F1 numbers in Table 1 are from a single run, while the inference procedure relies on stochastic sampling (temperature 1.15 in Table 9) and beam search. Differences between SiGIR and the strongest baseline are as small as 3.09 F1 on HotpotQA. The paper should provide at least the variance over multiple seeds (or bootstrap over test instances) and a statistical test or confidence interval to show the headline improvements are not sampling noise. This is essential before the method can be recommended as a reliable SOTA.","section":"Section 5 / Table 1"}],"minor_comments":[{"comment":"Equation (7) has malformed underbrace labels ('rc t loooomoooon previous reward'); please fix the LaTeX/typesetting.","section":"Equation 7"},{"comment":"Table 9 reports hyperparameters such as 'N 3' and '# Retrieval Reward 1.0 / 0.5 / -1.0' without explaining what they correspond to; please add a note or define N in the caption.","section":"Table 9"},{"comment":"Figure 4 contains corrupted Unicode tokens that make the axis labels unreadable; please replace with proper labels.","section":"Figure 4"},{"comment":"The self-improvement threshold in Algorithm 2 (line 37) is never specified; please report the threshold or state how it was chosen.","section":"Algorithm 2"},{"comment":"The relationship between the separate critic C and the merged self-critique reasoner is described somewhat ambiguously; it would help to state explicitly whether C is used at test time in the 'Separated' mode of Table 6.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and the skeptic's assessment are spot-on: the retrieval corpus description in Appendix A.4 is a genuine internal inconsistency with Section 4.1, and the 'derived from original paper' baselines make the headline comparison invalid as written. The paper has merit and the central method is plausible, but the evaluation must be substantially strengthened. I would not reject the paper outright because the issue is fixable; however, the authors should be asked to clarify the retrieval setup, re-run or re-implement baselines under the same conditions, and report variance. If they cannot provide an open-domain evaluation, the title and abstract should be revised to avoid claiming open-domain SOTA."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the SiGIR recipe — train a single small model to decompose, retrieve, reason, and emit critique tokens, then use cumulative rewards to do beam search — is a clean and sensible extension of Self-RAG/IRCoT/BeamAggR. The ablations show the search and reward signals matter, and the self-improvement experiment on 40% data is a nice touch. If the only question were 'does this method work?', I'd say provisionally yes, with the usual caveat that these are single runs without error bars.\n\nThe problem is the evaluation. Section 4.1 says open-domain, but Appendix A.4 says the retrieval corpus is built from 'supporting facts and irrelevant documents from the training instances' and retrieval is 'within this scope.' That's a small curated pool. The paper never says whether dev-set gold paragraphs are in it. If they are, it's not open-domain, and the comparison with baselines that used full Wikipedia-scale indices is not apples-to-apples. If they aren't, the reported F1s are impossible. Either way the headline 8.6% claim is unsupported as written. The stress-test note is on target.\n\nAlso, the 'derived from original paper' baselines (ProbTree, BeamAggR, RAG-Star, GenGround) were run under their own retrieval setups, which we have no reason to believe match this pool. So the SOTA comparison is doubly unstable.\n\nThe rest of the paper is careful: hyperparameters are listed, prompts are in the appendix, and the training recipe is reproducible. The lack of error bars or significance testing is a secondary concern because the effect sizes are large, but it still means the margins are unverified. The self-improvement threshold is unspecified.\n\nBottom line: this is a real method worth engaging with, but the central benchmark claim needs a corrected evaluation or a clear statement of the retrieval pool before the paper can be accepted. Send it to review, but ask the authors to clarify or re-run.","headline":"Useful method paper whose headline SOTA claim is not supported by the reported retrieval setup — needs a re-run or a clear caveat before I'd trust the numbers.","tokens_in":23570,"tokens_out":2069,"would_cite":true,"duration_ms":18530,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model that criticizes each intermediate retrieval and reasoning step beats the previous best multi-hop QA systems by 8.6%.","keywords":["multi-hop question answering","self-critique","iterative reasoning","retrieval-augmented generation","process reward","beam search","question decomposition","open-domain QA"],"falsifier":"Inspect the released retrieval pool: if any development-set question's gold supporting paragraphs are missing, re-run SiGIR with those documents included or with an unrestricted Wikipedia index and compare F1; the 8.6% margin would shrink if the current gain comes from searching a closed, easier pool.","tokens_in":22543,"feed_emoji":"🧠","tokens_out":9338,"duration_ms":53945,"temperature":0.7,"pith_summary":"SiGIR claims that multi-hop question answering gets substantially better when the model scores every intermediate retrieval and reasoning step it takes, instead of only receiving the final answer as feedback. The paper trains a 7-billion-parameter Mistral model to decompose questions into sub-questions, trigger retrieval, reason over documents, and emit self-critique tokens rating retrieval relevance and reasoning utility. At inference the model branches over decomposition, retrieval, and reasoning, and keeps only the top-k trajectories according to cumulative process rewards. On 2WikiMQA, HotpotQA, and MuSiQue the authors report F1 of 74.47, 63.09, and 37.15, an average relative gain of 8.6% over the previous best results. A sympathetic reader should read this as evidence that intermediate self-critique, not just more retrieval or decomposition, is what carries difficult multi-hop reasoning.","feed_headline":"Self-critique search lifts multi-hop QA by 8.6% over prior best","feed_subtitle":"It scores its own retrieval and reasoning steps, beating prior best answers on three multi-hop benchmarks.","key_machinery":"The load-bearing mechanism is the merged reasoner-critic, called SC-Reasoner, which writes self-critique tokens after each sub-process (decomposition, retrieval, reasoning, and question reduction) and therefore produces its own rewards without a separate scoring pass. At inference, iteration-level beam search uses cumulative process rewards $r^c_t = r^c_{t-1} + r^{retr}_t + r^{reas}_t$ to keep the top-k candidate trajectories at each step, and the trajectory with the highest cumulative reward supplies the final answer. The design works because the rewards are fine-grained and process-level: ablations show that coarse binary rewards lose their discriminative power exactly when the search branches broadly, while graded retrieval and reasoning rewards keep the search on promising paths.","core_discovery":"The central discovery the paper argues for is that iterative reasoning can be guided from inside the generator: instead of a separate verifier that only judges complete answers, the model itself rates each retrieval as relevant, partially relevant, or irrelevant, and each reasoning step as fully, partially, or not supported, and these fine-grained process rewards are accumulated across iterations. A beam search then retains the most promising trajectories, so bad decompositions and bad retrieval calls are pruned before they cascade into wrong final answers. The paper reports that this system, SiGIR, surpasses the prior state of the art on all three benchmarks, with the largest absolute gains on MuSiQue, and that the gain is 15.0% on MuSiQue overall, including 47.1% and 24.37% improvements on 3-hop and 4-hop questions respectively.","pith_inferences":["A reader should test the benchmark setup before trusting the 8.6% claim: the retrieval pool is built only from supporting facts and irrelevant documents of training instances, and the paper never states that development-set gold paragraphs are included, so the reported F1 may not be open-domain as claimed.","The paper's own limitations point to the next experiment: replace temperature-sampled sub-question branching with a dedicated decomposition or query-rewriting module and measure whether wider exploration converts into further gains.","The self-critique-plus-cumulative-reward pattern is not QA-specific; it is a general design for any multi-step agent task in which intermediate actions can be scored, such as tool use, code execution, or database lookup.","Following the ablations, an adaptive inference scheme using coarse rewards under greedy decoding and fine-grained rewards only during wide search would likely capture most of the benefit at lower cost, though the paper does not propose it."],"forward_implications":["Fine-tuning a small model to emit its own step-level scores can substitute for expensive Monte Carlo tree search or a large external verifier in multi-hop QA, since reward-guided beam search performs better than greedy inference at similar cost.","Fine-grained process rewards should be preferred over binary rewards when the inference budget permits branching over decomposition and retrieval candidates, because coarse rewards lose discrimination as search width grows.","Exploration must span question decompositions, not just retrieved documents: setting the decomposition branch width to one costs 13.57 points in average F1.","Self-critique can generate its own training data: after two self-improvement iterations on 40% labeled data, the model reaches about 97.8% of the full-SFT performance, so the pipeline should transfer to low-resource settings.","Hybrid sparse-dense retrieval can be added without retraining because each retrieved document is evaluated independently, and it improves F1 across the three benchmarks."],"supporting_citations":[{"why":"Introduces IRCoT, the interleaved retrieval-and-reasoning baseline whose setup SiGIR inherits and whose retrieval-corpus construction is adopted.","marker":"Trivedi et al. (2023)"},{"why":"Provides ProbTree, a decomposition-tree baseline that SiGIR compares against and improves on.","marker":"Cao et al. (2023)"},{"why":"Provides BeamAggR, the beam-aggregation decomposition baseline that set a previous high score on MuSiQue.","marker":"Chu et al. (2024a)"},{"why":"Provides Self-RAG, the self-reflective retrieval and generation baseline that motivates merging retrieval, generation, and critique in one model.","marker":"Asai et al. (2024)"},{"why":"Provides DR-Distillation, the previous best training-based method on 2WikiMQA that SiGIR's 74.47 F1 surpasses.","marker":"Li et al. (2024)"},{"why":"Provides the HotpotQA benchmark and its bridge and comparison question types.","marker":"Yang et al. (2018)"},{"why":"Provides the 2WikiMQA benchmark and its four reasoning types.","marker":"Ho et al. (2020)"},{"why":"Provides MuSiQue, the hardest 2-to-4-hop benchmark where SiGIR reports the largest gains.","marker":"Trivedi et al. (2022)"},{"why":"Supplies BM25, the sparse retriever used in the main experiments and in the baseline comparisons.","marker":"Robertson and Zaragoza (2009)"},{"why":"Defines the Mistral-7B backbone on which SiGIR is trained and evaluated.","marker":"Jiang et al. (2023a)"}],"fun_headline_variants":["Self-critique prunes bad reasoning, lifts multi-hop QA by 8.6%","Model critiques its own steps to beat prior multi-hop QA best","Self-scored retrieval guides reasoning to SOTA multi-hop QA","SiGIR yields 8.6% gain in multi-hop QA via self-feedback","Iterative self-critique sharpens multi-hop reasoning, boosts QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the retrieval corpus being a fair open-domain test, but the corpus is built only from training-instance supporting facts and irrelevant documents, and the paper does not state that development-set evidence documents are in it.","fun_headline_variants_meta":{"raw":{"variants":["Self-critique prunes bad reasoning, lifts multi-hop QA by 8.6%","Model critiques its own steps to beat prior multi-hop QA best","Self-scored retrieval guides reasoning to SOTA multi-hop QA","SiGIR yields 8.6% gain in multi-hop QA via self-feedback","Iterative self-critique sharpens multi-hop reasoning, boosts QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1743,"prompt_tokens":905,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":737}},"tokens_in":521,"tokens_out":838,"duration_ms":7441,"temperature":1.0,"reasoning_tokens":737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:19:11.565242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released retrieval pool: if any development-set question's gold supporting paragraphs are missing, re-run SiGIR with those documents included or with an unrestricted Wikipedia index and compare F1; the 8.6% margin would shrink if the current gain comes from searching a closed, easier pool.","supporting_citations":[],"review_version":1}