{"id":"a23d3371-bf9d-40cc-97d9-d58d9c432da1","arxiv_id":"2501.10053","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AirRAG integrates MCTS with five reasoning actions and self-consistency to improve RAG accuracy on multi-hop and open-domain QA, but the main results run without active tree search.","lead":"AirRAG couples Monte Carlo tree search with five hand-designed reasoning actions to steer retrieval-augmented generation, and reports accuracy gains on seven question-answering benchmarks. The headline comparison uses only one rollout and one sampled answer per action, so the central claim that tree search expands the solution space is not directly exercised in the main results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline results in Table 1 use rollout=1 and n=1, so MCTS and self-consistency are effectively inactive; the central attribution of the reported gains to tree-based solution-space expansion is not tested in the primary comparison.","rationale":"The Pith reader's weakest assumption is exactly the same load-bearing issue I find, and I do not see a reason to move the verdict. AirRAG is a plausible framework, and the paper does provide genuine evidence for inference scaling: Figure 4 shows accuracy rising with the number of rollouts, Table 2 shows computational-allocation strategies helping, and Figure 7 compares verification methods. Those results support a weaker claim that AirRAG can use additional generated reasoning paths to improve accuracy. What they do not support is the abstract's stronger claim that the state-of-the-art margins in Table 1 arise from MCTS-based expansion of the solution space and self-consistency verification. Because Table 1, the primary comparison, is run at rollout=1 and n=1, the operative mechanism has no opportunity to branch, compare alternatives, or aggregate votes. The Appendix's attempt to explain away rollout=1 does not change the fact that a single root-to-leaf pass is not a tree search. This is not an internal mathematical inconsistency; it is an evidential gap between the mechanism named in the title and the configuration evaluated. The fix is straightforward and inexpensive: run the headline evaluation with search enabled, or compare directly against a greedy planner using the same five actions. Until that is done, a conditional verdict is appropriate: the framework may well work, but the central attribution is unverified. I would not reject the paper, because the action space, the scaling results, and the pluggable architecture are useful contributions independent of the MCTS framing. I recommend keeping the conditional verdict and requiring the proposed control experiment as the condition for acceptance.","tokens_in":19219,"tokens_out":4648,"duration_ms":48731,"concrete_test":"On Qwen2.5-14B with the same 1,000-sample test sets, run three variants of AirRAG: (A) the current Table 1 configuration with rollout=1, n=1; (B) the scaling configuration from Table 2/Figure 4 with rollout=32, n=4; and (C) a greedy controller that executes the same five actions in the same order but with no UCT, no backpropagation, no parallel candidate paths, and no self-consistency aggregation, matching (A) in sampling parameters. If (A) and (C) produce statistically indistinguishable F1/Acc, the headline comparison contains no measurable MCTS signal, and the paper must be reframed as an action-design and inference-scaling method. If (B) clearly beats (C), the search benefit exists but should be reported as the headline result with a matched compute budget, including total LLM calls and tokens for each method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is an attribution gap in the main evaluation. Table 1, and Table 4, fix rollouts=1 and output sequences n=1. With one rollout and one sample per action, the MCTS loop has no branching: UCT in Eq. (1) never compares alternative child nodes, no Q-values are back-propagated from multiple leaves, and the self-consistency verifier in Section 3.3 receives only one candidate answer to score. The Appendix's statement that rollout=1 'does not imply that MCTS is not executed' is true only in the trivial sense that one root-to-leaf pass is run; that pass is functionally a greedy action sequence. If the implementation still emits parallel actions at the root, the paper must demonstrate that this creates multiple candidate leaves and that UCT/backpropagation affects them; the reported configuration provides no such evidence. Consequently, Table 1's large margins over IterDRAG and Search-o1 may be caused by the five-action decomposition, prompt design, or extra retrieval calls, rather than by 'expanding the solution space' via MCTS. The only direct evidence that search and self-consistency help is Figures 1 and 4 and Table 2, which use rollout=32 and n=4; those runs are separate from the headline comparison and are not matched to the baselines in compute, number of LLM calls, or retrieval budget. The paper's central claim, as stated in the abstract and contributions, therefore rests on an untested configuration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AirRAG, a retrieval-augmented generation framework that combines five hand-designed reasoning actions (SAY, DA, RA, QT, SA) with Monte Carlo Tree Search and self-consistency-based verification. The authors claim that this expands the solution space for complex QA and yields significant improvements over iterative and agentic RAG baselines on NQ, TriviaQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, PopQA, and WebQA, across Qwen2.5-7B/14B/32B, Qwen3-235B, and Llama3-8B. The paper also studies inference-scaling strategies (retrieved-document counts, context length, rollouts, output sequences), verification methods, and a trained process-supervision reward model. The headline comparisons, however, are run with one rollout and one output sequence per action, so the proposed MCTS and self-consistency mechanisms are not exercised in the primary evaluation.","tokens_in":19578,"tokens_out":11555,"duration_ms":104648,"significance":"If the claims survive scrutiny, AirRAG would be a useful plug-and-play contribution to agentic RAG: it is training-free for the base generator, provides a modular action space that can absorb new tools, and shows consistent gains over vanilla RAG and several iterative baselines across many datasets and model sizes. The inference-scaling analysis (Figures 1, 3, 4) and the verification-method comparison (Figure 7) are valuable empirical contributions, and the qualitative case study in Figures 13-14 is instructive. The main reservation is that the headline configuration does not exercise the proposed MCTS/self-consistency machinery, so the central attribution is not yet established.","major_comments":[{"comment":"The headline results are obtained with rollouts=1 and n=1, so the MCTS and self-consistency components of AirRAG are effectively inactive in the primary comparison. With a single rollout and one output sequence per action, UCT in Eq. (1) never accumulates visit counts that distinguish among alternative children, no meaningful Q-values are back-propagated (the Appendix assigns every leaf Q=1 when nall=1), and the self-consistency scorer in Section 3.3 receives only one candidate. The Appendix statement that 'rollout=1 does not imply that MCTS is not executed' is true only in the trivial sense that one root-to-leaf pass is run. The gains in Table 1 over IterDRAG and Search-o1 therefore cannot be attributed to 'expanding the solution space' via MCTS; they may come from the five-action decomposition, the prompts, or additional retrieval calls. The same issue applies to Table 4, whose caption also sets rollouts=1 and n=1 for AirRAG. The experiments that do activate MCTS (Figures 1 and 4, Table 2 with rollout=32) are not matched to baselines in LLM calls, retrieval budget, or total tokens, so they do not close the attribution gap. Please either provide a headline comparison with rollouts>=2 and n>=2 against baselines at matched compute budgets, or restrict the central claim to the action-decomposition contribution.","section":"§4.2, Tables 1 and 4; Appendix A"},{"comment":"No measure of variability or statistical significance is reported. All tables use a single fixed seed and 1,000 sampled questions per dataset, yet the abstract and text claim 'significant performance gains.' Some headline differences are small (e.g., Qwen2.5-7B average F1: AirRAG 61.1 vs. DeepResearcher 61.0), so without error bars, bootstrap confidence intervals, or paired significance tests, the reader cannot tell whether the reported margins are stable. Please add multi-seed or bootstrap uncertainty estimates for at least the main comparisons.","section":"§4.2, Appendix A"},{"comment":"Baseline coverage is incomplete at the scales where the strongest claims are made. ReSearch and DeepResearcher appear only at 7B (and ReSearch at 32B), not at 14B or Qwen3-235B; on Qwen2.5-32B, ReSearch actually beats AirRAG on MuSiQue (F1 38.3 vs. 36.5). Table 5 (Llama3-8B) omits Search-o1, ReSearch, and DeepResearcher entirely, and Table 4 compares only IterDRAG and AirRAG variants under budget scaling. The statements 'surpassing the trainable approaches' and 'state-of-the-art among all datasets' are therefore not established across the tested model sizes. Please add the missing baselines or temper the claims to the configurations actually compared.","section":"§4.2, Tables 1 and 5; §4.3"},{"comment":"The robustness comparison in Table 5 mixes reproduced numbers with results 'quoted from Jin et al. (2024) and Yu et al. (2024)' without reporting the retrieval corpus, retriever, or prompt settings for the quoted entries. If those baselines were evaluated in a different pipeline, the large margins (e.g., Auto-RAG 44.9 vs. AirRAG 67.6 F1 on HotpotQA) may reflect pipeline differences rather than the proposed method. Please rerun all baselines in the identical setting or clearly state the protocol and corpus for each quoted number.","section":"Table 5"}],"minor_comments":[{"comment":"The hop-distribution sentence contains the typo 'aboout'; please correct it to 'about'.","section":"Appendix A"},{"comment":"The figure caption contains the untranslated Chinese phrase '以下展示的是推理过程的动作序列' and the unexplained label 'Ours RAG-O1-plus'; please translate or remove these.","section":"Figure 2"},{"comment":"Table 1's caption refers to 'test sets' while Appendix A states that 1,000 samples were selected from 'validation sets' with a fixed seed; please align the terminology.","section":"Table 1 caption vs. Appendix A"},{"comment":"In Eq. (1), the function UCT(s,p) is written with Q(s,a) and N(s), but the action a is not formally introduced as part of the node state, and the update Q(si,ai)=Q(si,ai)+Q(sd,ad) conflates node rewards with action rewards; please clarify the notation.","section":"Eq. (1) and Section 3.2.1"},{"comment":"The sentence 'MCTS selects the action a1 and a2' should be rephrased to 'selects actions a1 and a2' and should specify whether two branches are created at the root, since this affects the interpretation of the rollout=1 configuration.","section":"Section 3.1"},{"comment":"The claim that the method is 'training-free' is in tension with the reward-model fine-tuning in Section 3.3; please qualify the claim as applying to the base generator and the non-reward-model configuration.","section":"§3.2.3 and §3.3"},{"comment":"The Appendix states that Table 2 uses rollout=32, but the table caption does not report this; please state rollout and n settings in every table caption.","section":"Table 2 caption / Appendix A"},{"comment":"In Table 3, the column 'database_size' reports '100w' without defining the unit; the text says the database contains approximately one million documents, so please spell out the units directly in the table.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable candidate after a major revision focused on the attribution gap. The strongest fix would be a matched-budget comparison with rollouts>1/n>1, plus error bars and complete baseline coverage. I do not see a circularity problem in the main evaluation because the reward model is not used in the headline configuration. There are also copyediting issues suggesting the manuscript needs a careful pass before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful RAG paper, but the main table doesn't actually exercise the tree search. The authors report strong gains with rollout=1 and n=1, which reduces their MCTS to a single-minded action sequence. That means the headline claim — that MCTS 'expands the solution space' — is not what the numbers in Table 1 are testing.\n\nWhat's new and good: the five-action decomposition (SAY, DA, RA, QT, SA) is a sensible, flexible action space for complex QA, and the plug-and-play design is a nice touch. The scaling experiments (docs, context length, rollouts, n) are broad and show the expected trends. The self-consistency verification methods, including the Jaccard/embedding-based scoring and the process-supervised reward model, are interesting. The paper does not hide the configuration: the appendix says rollout=1 is a deliberate choice for efficiency, and Figures 1/4 and Table 2 show that increasing rollouts and n improves performance. So the evidence that the search mechanism helps is there, just not in the headline comparison.\n\nThe soft spots: first, the attribution gap is real. With one rollout, you get a single root-to-leaf path; UCT never compares alternatives and backpropagation is a no-op. The gains over IterDRAG and Search-o1 in Table 1 likely come from the action space and prompt design, not from the search. Second, the compute budget is not matched when the paper does use rollouts=32 and n=4 against baselines — those runs are presented separately. Third, no error bars, and the 'computationally optimal' strategy in Table 2 seems tuned on the same test sets, risking overfitting. The reward model is trained on AirRAG-generated data, but since the main results don't use it, circularity is not a real issue.\n\nWho should read this: RAG researchers, especially those interested in inference scaling or agentic retrieval. The framework is a reasonable alternative to IterDRAG/Search-o1 and the scaling analysis is useful. It deserves a serious referee, but the authors need to either run Table 1 with actual MCTS or reframe the central claim. As it stands, the paper is overclaiming its own best evidence.\n\nI'd send it to peer review with that caveat.","headline":"A useful RAG framework, but the headline numbers don't actually run the tree search; the paper overclaims its central mechanism.","tokens_in":20080,"tokens_out":2940,"would_cite":true,"duration_ms":27390,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AirRAG claims that turning RAG reasoning into a tree of five basic actions expands the solution space a model can explore, beating iterative and agentic RAG baselines on multi-hop question answering.","keywords":["AirRAG","retrieval-augmented generation","Monte Carlo tree search","self-consistency","reasoning actions","inference scaling","multi-hop question answering","test-time compute"],"falsifier":"Run AirRAG on MuSiQue and 2Wiki with the five actions executed in a fixed order—say, system analysis, then query transformation, retrieval-answer, summary-answer—without any tree branching, UCT selection, or backpropagation, at the same token budget as the rollout=1 configuration. If the fixed-order chain matches AirRAG's reported accuracy, the MCTS expansion is not what produces the improvement; if increasing rollouts from 1 to 32 fails to improve accuracy beyond the fixed chain, the solution-space-expansion claim is unsupported.","tokens_in":19036,"feed_emoji":"🧠","tokens_out":6000,"duration_ms":51554,"temperature":0.7,"pith_summary":"The paper proposes AirRAG, a retrieval-augmented generation (RAG) method that frames answering as a tree of five basic reasoning actions: system analysis, direct answer, retrieval-answer, query transformation, and summary-answer. The tree is searched with Monte Carlo Tree Search (MCTS), and multiple sampled answer paths are merged through self-consistency scoring. The aim is to show that this expands the solution space a model can explore, escaping the single chain of reasoning that iterative RAG methods get stuck in. On five complex QA datasets, AirRAG reports higher accuracy and F1 than iterative and agentic RAG baselines across several model sizes. If the claim holds, it means test-time computation can be spent to make even small models handle multi-hop questions.","feed_headline":"AirRAG beats iterative RAG with tree-searched reasoning actions","feed_subtitle":"Five basic reasoning actions searched by Monte Carlo tree search lift accuracy on complex multi-hop QA.","key_machinery":"The central machinery is the action space and the tree built from it: five reasoning actions (system analysis, direct answer, retrieval-answer, query transformation, summary-answer) that the language model chooses among at each step, with MCTS (using the UCT formula) balancing exploration and exploitation over the resulting tree. The state transition concatenates prior reasoning steps with the chosen action, and rollouts end at a maximum depth or terminal state; terminal rewards backpropagate along the path. Self-consistency verification (Jaccard-based or embedding-based clustering, or a trained reward model) selects the final answer from the generated trajectories.","core_discovery":"AirRAG's central claim is that combining a small set of human-like reasoning actions with tree search and self-consistency verification gives a controlled way to scale inference computation in RAG. The authors design five actions—system analysis, direct answer, retrieval-answer, query transformation, and summary-answer—as the fundamental action space, expand it into a tree via MCTS with UCT node selection, and select final answers by clustering candidates with Jaccard similarity or embeddings, or by a process-supervised reward model. The paper reports that AirRAG outperforms iterative and agentic baselines such as IterDRAG, Search-o1, ReSearch, and DeepResearcher, and that performance improves as retrieved documents, context length, rollouts, and sampled outputs increase. It also reports that a simplified three-action version, AirRAG-Lite, keeps most of the gain at lower cost.","pith_inferences":["If the rollout=1 results are representative, the practical contribution may be the action prompts rather than the search; a greedy fixed action chain is a cheaper way to test this directly.","The process-supervised reward model trained on MCTS-generated paths could be reused as a step-level verifier for other reasoning frameworks, independent of AirRAG.","The observation that a smaller, curated document database improves accuracy suggests that database partitioning tuned to query complexity could compound AirRAG's gains.","A direct comparison of AirRAG against a chain-of-thought baseline with the same five action prompts, matched for token budget, would isolate the contribution of the tree search."],"forward_implications":["The five-action decomposition itself, even before branching, appears to beat single-chain iterative RAG on most datasets, suggesting prompt-level planning is the primary driver.","Scaling retrieved documents, effective context length, rollouts, and sampled outputs all improve performance, consistent with an inference scaling law for RAG.","Allocating extra output samples and higher sampling diversity specifically to the planning and query-rewriting actions (SAY and QT) yields the largest gains per token.","Because AirRAG is training-free and pluggable, other RAG strategies and stronger models can be added as new action branches without retraining."],"supporting_citations":[{"why":"Supplies the IterDRAG baseline AirRAG compares against and the inference-scaling setup (Lmax, document counts) that AirRAG extends.","marker":"(Yue et al., 2024)"},{"why":"Provides the self-consistency idea and voting/scoring mechanism used to verify and aggregate candidate reasoning paths.","marker":"(Wang et al., 2023)"},{"why":"Gives the UCT node-selection formula that drives MCTS exploration-exploitation in AirRAG.","marker":"(Kocsis and Szepesvári, 2006)"},{"why":"Defines the agentic search workflow (Search-o1) that AirRAG positions itself against and reproduces as a baseline.","marker":"(Li et al., 2025)"},{"why":"Provides ReSearch, an RL-trained agentic baseline that AirRAG aims to surpass without training.","marker":"(Chen et al., 2025)"},{"why":"Provides DeepResearcher, another agentic retrieval baseline for comparison.","marker":"(Zheng et al., 2025)"},{"why":"Defines the MuSiQue multi-hop dataset that anchors AirRAG's evaluation of complex reasoning.","marker":"(Trivedi et al., 2022)"},{"why":"Defines the 2WikiMultiHopQA dataset used in the main evaluation.","marker":"(Ho et al., 2020)"},{"why":"Defines the HotpotQA multi-hop dataset used in the main evaluation.","marker":"(Yang et al., 2018)"},{"why":"Supplies the Wikipedia retrieval corpus and dense passage retrieval setup used in all experiments.","marker":"(Karpukhin et al., 2020)"}],"fun_headline_variants":["Tree-search steers RAG: five actions beat iterative agents","AirRAG scales inference with MCTS and five reasoning actions","RAG gets a tree: AirRAG's MCTS expands single-shot answers","Five reasoning actions, MCTS expansion: AirRAG lifts QA accuracy","From single path to tree: AirRAG's control over RAG inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results in Table 1 are run with a single rollout and a single output per action, so the tree search and self-consistency components are not actually exercised in that comparison; the paper nonetheless attributes the reported gains to the MCTS-based framework.","fun_headline_variants_meta":{"raw":{"variants":["Tree-search steers RAG: five actions beat iterative agents","AirRAG scales inference with MCTS and five reasoning actions","RAG gets a tree: AirRAG's MCTS expands single-shot answers","Five reasoning actions, MCTS expansion: AirRAG lifts QA accuracy","From single path to tree: AirRAG's control over RAG inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1446,"prompt_tokens":917,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":533,"tokens_out":529,"duration_ms":5418,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:23:07.330623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AirRAG on MuSiQue and 2Wiki with the five actions executed in a fixed order—say, system analysis, then query transformation, retrieval-answer, summary-answer—without any tree branching, UCT selection, or backpropagation, at the same token budget as the rollout=1 configuration. If the fixed-order chain matches AirRAG's reported accuracy, the MCTS expansion is not what produces the improvement; if increasing rollouts from 1 to 32 fails to improve accuracy beyond the fixed chain, the solution-space-expansion claim is unsupported.","supporting_citations":[],"review_version":1}