Pith. sign in

REVIEW 4 major objections 8 minor 7 cited by

AirRAG: Autonomous Strategic Planning and Reasoning Steer Retrieval Augmented Generation

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AirRAG claims that turning RAG reasoning into a tree of five basic actions expands the solution space a model can explore, beating iterative and agentic RAG baselines on multi-hop question answering.

desk verdict A useful RAG framework, but the headline numbers don't actually run the tree search; the paper overclaims its central mechanism. read the letter →

arxiv 2501.10053 v3 pith:6FW55WTH submitted 2025-01-17 cs.AI

Wenfeng Feng , Chuzhan Hao , Yuewei Zhang , Guochao Jiang , Jingyi Song , Hao Wang This is my paper · ORCID
classification cs.AI
keywords AirRAGretrieval-augmentedgenerationMonteCarlotreesearchself-consistencyreasoningactionsinferencescalingmulti-hopquestionansweringtest-timecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AirRAG, a retrieval-augmented generation (RAG) method that frames answering as a tree of five basic reasoning actions: system analysis, direct answer, retrieval-answer, query transformation, and summary-answer. The tree is searched with Monte Carlo Tree Search (MCTS), and multiple sampled answer paths are merged through self-consistency scoring. The aim is to show that this expands the solution space a model can explore, escaping the single chain of reasoning that iterative RAG methods get stuck in. On five complex QA datasets, AirRAG reports higher accuracy and F1 than iterative and agentic RAG baselines across several model sizes. If the claim holds, it means test-time computation can be spent to make even small models handle multi-hop questions.

What carries the argument

The central machinery is the action space and the tree built from it: five reasoning actions (system analysis, direct answer, retrieval-answer, query transformation, summary-answer) that the language model chooses among at each step, with MCTS (using the UCT formula) balancing exploration and exploitation over the resulting tree. The state transition concatenates prior reasoning steps with the chosen action, and rollouts end at a maximum depth or terminal state; terminal rewards backpropagate along the path. Self-consistency verification (Jaccard-based or embedding-based clustering, or a trained reward model) selects the final answer from the generated trajectories.

What would settle it

Run AirRAG on MuSiQue and 2Wiki with the five actions executed in a fixed order—say, system analysis, then query transformation, retrieval-answer, summary-answer—without any tree branching, UCT selection, or backpropagation, at the same token budget as the rollout=1 configuration. If the fixed-order chain matches AirRAG's reported accuracy, the MCTS expansion is not what produces the improvement; if increasing rollouts from 1 to 32 fails to improve accuracy beyond the fixed chain, the solution-space-expansion claim is unsupported.

Watch

Extended reading notes

Core claim

AirRAG's central claim is that combining a small set of human-like reasoning actions with tree search and self-consistency verification gives a controlled way to scale inference computation in RAG. The authors design five actions—system analysis, direct answer, retrieval-answer, query transformation, and summary-answer—as the fundamental action space, expand it into a tree via MCTS with UCT node selection, and select final answers by clustering candidates with Jaccard similarity or embeddings, or by a process-supervised reward model. The paper reports that AirRAG outperforms iterative and agentic baselines such as IterDRAG, Search-o1, ReSearch, and DeepResearcher, and that performance improves as retrieved documents, context length, rollouts, and sampled outputs increase. It also reports that a simplified three-action version, AirRAG-Lite, keeps most of the gain at lower cost.

Load-bearing premise

The headline results in Table 1 are run with a single rollout and a single output per action, so the tree search and self-consistency components are not actually exercised in that comparison; the paper nonetheless attributes the reported gains to the MCTS-based framework.

Editorial extensions

If this is right

  • The five-action decomposition itself, even before branching, appears to beat single-chain iterative RAG on most datasets, suggesting prompt-level planning is the primary driver.
  • Scaling retrieved documents, effective context length, rollouts, and sampled outputs all improve performance, consistent with an inference scaling law for RAG.
  • Allocating extra output samples and higher sampling diversity specifically to the planning and query-rewriting actions (SAY and QT) yields the largest gains per token.
  • Because AirRAG is training-free and pluggable, other RAG strategies and stronger models can be added as new action branches without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the rollout=1 results are representative, the practical contribution may be the action prompts rather than the search; a greedy fixed action chain is a cheaper way to test this directly.
  • The process-supervised reward model trained on MCTS-generated paths could be reused as a step-level verifier for other reasoning frameworks, independent of AirRAG.
  • The observation that a smaller, curated document database improves accuracy suggests that database partitioning tuned to query complexity could compound AirRAG's gains.
  • A direct comparison of AirRAG against a chain-of-thought baseline with the same five action prompts, matched for token budget, would isolate the contribution of the tree search.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes AirRAG, a retrieval-augmented generation framework that combines five hand-designed reasoning actions (SAY, DA, RA, QT, SA) with Monte Carlo Tree Search and self-consistency-based verification. The authors claim that this expands the solution space for complex QA and yields significant improvements over iterative and agentic RAG baselines on NQ, TriviaQA, HotpotQA, MuSiQue, 2WikiMultiHopQA, PopQA, and WebQA, across Qwen2.5-7B/14B/32B, Qwen3-235B, and Llama3-8B. The paper also studies inference-scaling strategies (retrieved-document counts, context length, rollouts, output sequences), verification methods, and a trained process-supervision reward model. The headline comparisons, however, are run with one rollout and one output sequence per action, so the proposed MCTS and self-consistency mechanisms are not exercised in the primary evaluation.

Significance. If the claims survive scrutiny, AirRAG would be a useful plug-and-play contribution to agentic RAG: it is training-free for the base generator, provides a modular action space that can absorb new tools, and shows consistent gains over vanilla RAG and several iterative baselines across many datasets and model sizes. The inference-scaling analysis (Figures 1, 3, 4) and the verification-method comparison (Figure 7) are valuable empirical contributions, and the qualitative case study in Figures 13-14 is instructive. The main reservation is that the headline configuration does not exercise the proposed MCTS/self-consistency machinery, so the central attribution is not yet established.

major comments (4)
  1. [§4.2, Tables 1 and 4; Appendix A] The headline results are obtained with rollouts=1 and n=1, so the MCTS and self-consistency components of AirRAG are effectively inactive in the primary comparison. With a single rollout and one output sequence per action, UCT in Eq. (1) never accumulates visit counts that distinguish among alternative children, no meaningful Q-values are back-propagated (the Appendix assigns every leaf Q=1 when nall=1), and the self-consistency scorer in Section 3.3 receives only one candidate. The Appendix statement that 'rollout=1 does not imply that MCTS is not executed' is true only in the trivial sense that one root-to-leaf pass is run. The gains in Table 1 over IterDRAG and Search-o1 therefore cannot be attributed to 'expanding the solution space' via MCTS; they may come from the five-action decomposition, the prompts, or additional retrieval calls. The same issue applies to Table 4, whose caption also sets rollouts=1 and n=1 for AirRAG. The experiments that do activate MCTS (Figures 1 and 4, Table 2 with rollout=32) are not matched to baselines in LLM calls, retrieval budget, or total tokens, so they do not close the attribution gap. Please either provide a headline comparison with rollouts>=2 and n>=2 against baselines at matched compute budgets, or restrict the central claim to the action-decomposition contribution.
  2. [§4.2, Appendix A] No measure of variability or statistical significance is reported. All tables use a single fixed seed and 1,000 sampled questions per dataset, yet the abstract and text claim 'significant performance gains.' Some headline differences are small (e.g., Qwen2.5-7B average F1: AirRAG 61.1 vs. DeepResearcher 61.0), so without error bars, bootstrap confidence intervals, or paired significance tests, the reader cannot tell whether the reported margins are stable. Please add multi-seed or bootstrap uncertainty estimates for at least the main comparisons.
  3. [§4.2, Tables 1 and 5; §4.3] Baseline coverage is incomplete at the scales where the strongest claims are made. ReSearch and DeepResearcher appear only at 7B (and ReSearch at 32B), not at 14B or Qwen3-235B; on Qwen2.5-32B, ReSearch actually beats AirRAG on MuSiQue (F1 38.3 vs. 36.5). Table 5 (Llama3-8B) omits Search-o1, ReSearch, and DeepResearcher entirely, and Table 4 compares only IterDRAG and AirRAG variants under budget scaling. The statements 'surpassing the trainable approaches' and 'state-of-the-art among all datasets' are therefore not established across the tested model sizes. Please add the missing baselines or temper the claims to the configurations actually compared.
  4. [Table 5] The robustness comparison in Table 5 mixes reproduced numbers with results 'quoted from Jin et al. (2024) and Yu et al. (2024)' without reporting the retrieval corpus, retriever, or prompt settings for the quoted entries. If those baselines were evaluated in a different pipeline, the large margins (e.g., Auto-RAG 44.9 vs. AirRAG 67.6 F1 on HotpotQA) may reflect pipeline differences rather than the proposed method. Please rerun all baselines in the identical setting or clearly state the protocol and corpus for each quoted number.
minor comments (8)
  1. [Appendix A] The hop-distribution sentence contains the typo 'aboout'; please correct it to 'about'.
  2. [Figure 2] The figure caption contains the untranslated Chinese phrase '以下展示的是推理过程的动作序列' and the unexplained label 'Ours RAG-O1-plus'; please translate or remove these.
  3. [Table 1 caption vs. Appendix A] Table 1's caption refers to 'test sets' while Appendix A states that 1,000 samples were selected from 'validation sets' with a fixed seed; please align the terminology.
  4. [Eq. (1) and Section 3.2.1] In Eq. (1), the function UCT(s,p) is written with Q(s,a) and N(s), but the action a is not formally introduced as part of the node state, and the update Q(si,ai)=Q(si,ai)+Q(sd,ad) conflates node rewards with action rewards; please clarify the notation.
  5. [Section 3.1] The sentence 'MCTS selects the action a1 and a2' should be rephrased to 'selects actions a1 and a2' and should specify whether two branches are created at the root, since this affects the interpretation of the rollout=1 configuration.
  6. [§3.2.3 and §3.3] The claim that the method is 'training-free' is in tension with the reward-model fine-tuning in Section 3.3; please qualify the claim as applying to the base generator and the non-reward-model configuration.
  7. [Table 2 caption / Appendix A] The Appendix states that Table 2 uses rollout=32, but the table caption does not report this; please state rollout and n settings in every table caption.
  8. [Table 3] In Table 3, the column 'database_size' reports '100w' without defining the unit; the text says the database contains approximately one million documents, so please spell out the units directly in the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AirRAG's central results are measured on external QA benchmarks, and the MCTS/self-consistency machinery is not defined in terms of the quantities it predicts.

full rationale

The paper's derivation chain is empirically self-contained rather than circular. The proposed method combines five reasoning actions, MCTS, and self-consistency verification, and the main claims are tested on independent benchmarks (HotpotQA, MuSiQue, 2WikiMultiHopQA, NQ, TriviaQA, etc.) against externally defined baselines. No equation defines a predicted quantity in terms of the fitted input: Eq. (1) is the standard UCT selection rule, Eqs. (2)-(3) are similarity scores for candidate answers, and the process-supervised reward model is trained on MCTS-generated trajectories with ground-truth labels, which is a conventional supervised training loop rather than a circular derivation. The paper does cite prior work for components such as self-consistency and inference scaling, but those citations are not self-citations and the components are independently evaluated. The most notable weakness is the attribution gap noted in the appendix: Tables 1 and 4 set both rollouts and output sequences to 1, so the headline gains cannot be cleanly attributed to tree-based solution-space expansion. The appendix's statement that 'a rollout value of 1 does not imply that MCTS is not executed' is a defense of that configuration, but this is an empirical-validity concern, not a circularity: the reported numbers are still measured on external benchmarks and are not forced by construction. No fitted parameter is renamed as a prediction, no load-bearing claim reduces to a self-citation, and no known result is merely relabeled. The central novelty claims may be under-tested in the primary configuration, but the paper does not define its outputs in terms of its inputs, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central performance claims depend on a set of hyperparameters (rollouts, n, w, d, sampling temperature) and on assumptions about the retrieval corpus and the action space. The paper provides no code or data release, so these choices are not independently checkable.

free parameters (5)
  • MCTS rollout count = 1 in Table 1; 1 to 32 in scaling experiments
    The number of rollouts is a key variable for the search expansion, tuned in experiments; the main results use 1.
  • Output sequences per action (n) = nall=1 in Table 1; na1,a4=3, others=1 in optimal strategy
    The 'computationally optimal' allocation is derived from experiments on the same benchmarks.
  • UCT exploration weight w = not stated
    The UCT formula in Eq. (1) contains a weight w that appears to balance exploration and exploitation, but its value is not reported.
  • Maximum tree depth d = 10
    Set to 10 in Appendix A: SAY and SA once, RA-QT/QT-RA up to 4 iterations.
  • Sampling parameters (top-p, top-k, temperature, qdiv) = 0.8/50/0.7 default; 1.0/1.0 for qdiv
    Sampling diversity knobs, adjusted in the optimal strategy.
assumptions (3)
  • domain assumption The Wikipedia dump from December 2018 with the multilingual-e5-base retriever provides sufficient and unbiased evidence for these QA benchmarks.
    Used as the retrieval corpus and retriever in all experiments; the paper does not quantify retrieval noise.
  • ad hoc to paper The five fundamental actions (SAY, DA, RA, QT, SA) cover the reasoning space needed for the tested QA tasks.
    The action set is designed by the authors from an informal summary of common RAG practices; no formal coverage argument is given.
  • domain assumption The reward model and the SA action can reliably select the correct answer from candidate trajectories.
    The verification methods are evaluated on the same datasets used to train the reward model, and the SA action is an LLM prompt, not a certified validator.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AirRAG: Autonomous Strategic Planning and Reasoning Steer Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/6FW55WTH

@misc{pith2026250110053,
  author       = {Pith},
  title        = {Pith review of: AirRAG: Autonomous Strategic Planning and Reasoning Steer Retrieval Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6FW55WTH}},
  note         = {Machine review of arXiv:2501.10053}
}
read the original abstract

Leveraging the autonomous decision-making capabilities of large language models (LLMs) has demonstrated superior performance in reasoning tasks. However, despite the success of iterative or agentic retrieval-augmented generation (RAG) techniques, these methods are often constrained to a single solution space when confronted with complex problems. In this paper, we propose a novel thinking pattern in RAG that integrates autonomous strategic planning with efficient reasoning actions, significantly activating intrinsic reasoning capabilities and expanding the solution space of specific tasks via Monte Carlo Tree Search (MCTS), which we refer to as AirRAG. Specifically, our approach designs five fundamental reasoning actions, which are expanded to a broad tree-based reasoning space using MCTS. The approach also incorporates self-consistency verification to explore potential reasoning paths and inference scaling law. Additionally, computationally optimal strategies are employed to allocate more inference resources to key actions, thereby enhancing overall performance. Experimental results demonstrate the effectiveness of AirRAG, showing significant performance gains on complex question-answering datasets. Furthermore, AirRAG is flexible and lightweight, making it easy to integrate with other advanced technologies and models.

Figures

Figures reproduced from arXiv: 2501.10053 by the authors.

Figure 1
Figure 1. Comparison of average performance across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The schematic diagram of our proposed AirRAG. AirRAG implements a paradigm that combines system [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Impact of the retrieved document number scaling ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Left: Performance comparison under different size of document database. A streamlined database can maintain a better performance. Right: Performance comparison in increasing the number of valid rollouts. Sampling a higher number of diverse reasoning paths consistently …
Figure 5
Figure 5. Figure 5: Overview of the distribution of query complexity over three multi-hop QA datasets. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Impact of the retrieved document number scaling and the maximum context length scaling over three [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison of different verification methods. "QwenRM" is short for reward model trained [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison on the number of different effective rollouts over three datasets. Sampling more [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Example prompt for SAY reasoning action. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Example prompt for QT reasoning action. Example Prompt for RA actions You are an expert in question answering. I am going to give you some contexts with may or may not be relevant to the question. Answer the question according to the contexts. {contexts} Question: {qu…
Figure 11
Figure 11. Figure 11: Example Prompt for RA actions. Example prompt for SA reasoning action You are an expert in question answering. Given the context, sub-queries and responses, output a correct and concise answer to User Query. <Here are some examples.> User Query: {question} {history} C…
Figure 12
Figure 12. Figure 12: Example prompt for SA reasoning action [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Incorrect reasoning path of iterative RAG methods. Ambiguity or missing information in the intermediate [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Effective reasoning rollout of AirRAG. The fundamental action space and tree-based search provides [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Rewarding each parallel reasoning path by Monte-Carlo-Shapley marginal contribution, scored by a generative reward model, lifts Pass@16 on AIME24/AIME25/AMC23 by 4-90% relative over Parallel-R1 with a fifth of the tra...

  2. Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A single LLM is trained with multi-agent distilled trajectories plus agentic RL, and the resulting Chain-of-Agents models set state-of-the-art Pass@1 scores among tool-integrated reasoning methods on GAIA, BrowseComp,...

  3. DeepRAG: Thinking to Retrieve Step by Step for Large Language Models

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A retrieval-augmented QA system that learns when to retrieve at each reasoning step outperforms adaptive RAG baselines on six benchmarks.

  4. Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A new RL framework that rewards fine-grained reasoning steps, called Atomic Thoughts, claims better agentic deep research on seven benchmarks.

  5. EvolveSearch: An Iterative Self-Evolving Search Agent

    cs.CL 2025-05 conditional novelty 5.0 of 10

    An iterative loop of RL and filtered SFT on the agent's own rollouts improves a 7B web-search agent by a few accuracy points on multi-hop QA benchmarks.

  6. R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

    cs.CL 2025-05 conditional novelty 5.0 of 10

    R1-Searcher++ uses SFT cold-start plus reinforcement learning with group and memorization rewards to teach Qwen-2.5-7B to balance internal knowledge and external retrieval, improving accuracy and reducing retrieval calls.

  7. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

Reference graph

Works this paper leans on

44 extracted references · 2 canonical work pages · cited by 7 Pith papers

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. https://aclanthology.org/D13-1160/ Semantic parsing on F reebase from question-answer pairs . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533--1544, Seattle, Washington, USA. Association for Computational Linguistics

    2013

  3. [3]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Micha Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. https://doi.org/10.1609/aaai.v38i16.29720 Graph of Thoughts: Solving Elaborate Problems with Large Language Models . Proceedings of the AAAI Conference on Artificial I...

    doi:10.1609/aaai.v38i16.29720 2024

  4. [4]

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. https://openreview.net/forum?id=VaXnxQ3UKo Alphamath almost zero: Process supervision without process . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  5. [5]

    Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025. https://arxiv.org/abs/2503.19470 Research: Learning to reason with search for llms via reinforcement learning . Preprint, arXiv:2503.19470

    arXiv 2025

  6. [6]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. https://doi.org/10.18653/v1/2023.acl-long.99 Precise zero-shot dense retrieval without relevance labels . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762--1777, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.99 2023

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, and et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  8. [8]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=7Bywt2mQsCe Measuring mathematical problem solving with the MATH dataset . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

Show all 44 references
  1. [9]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  2. [10]

    Paul I Jaffe, Russell A Poldrack, Robert J Schafer, and Patrick G Bissett. 2023. Modelling human behaviour in cognitive tasks with latent dynamical systems. Nature Human Behaviour, 7(6):986--1000

    2023

  3. [11]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. https://doi.org/10.18653/v1/2024.naacl-long.389 Adaptive- RAG : Learning to adapt retrieval-augmented large language models through question complexity . In Proceedings of the 2024 Conference of the N...

    2024 doi

  4. [12]

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.495 Active retrieval augmented generation . In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  5. [13]

    Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. 2024. https://arxiv.org/abs/2405.13576 Flashrag: A modular toolkit for efficient retrieval-augmented generation research . CoRR, abs/2405.13576

    2024 arXiv

  6. [14]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  7. [15]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. https://proceedings.mlr.press/v202/kandpal23a.html Large language models struggle to learn long-tail knowledge . In Proceedings of the 40th International Conference on Machine Learning, volume 20...

    2023

  8. [16]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  9. [17]

    Levente Kocsis and Csaba Szepesv \'a ri. 2006. Bandit based monte-carlo planning. In Machine Learning: ECML 2006, pages 282--293, Berlin, Heidelberg. Springer Berlin Heidelberg

    2006

  10. [18]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  11. [19]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. https://arxiv.org/abs/2501.05366 Search-o1: Agentic search-enhanced large reasoning models . Preprint, arXiv:2501.05366

    2025 arXiv

  12. [20]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.322 Query rewriting in retrieval-augmented large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 53...

    2023 doi

  13. [21]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  14. [22]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022...

    2022 arXiv

  15. [23]

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. https://arxiv.org/abs/2408.06195 Mutual reasoning makes smaller llms stronger problem-solvers . Preprint, arXiv:2408.06195

    2024 arXiv

  16. [24]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://openreview.net/forum?id=Yacmpz84TH Toolformer: Language models can teach themselves to use tools . In Thirty-seventh C...

    2023

  17. [25]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.620 Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy . In Findings of the Association for Comput...

    2023 doi

  18. [26]

    Chi, Nathanael Sch\" a rli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch\" a rli, and Denny Zhou. 2023. https://proceedings.mlr.press/v202/shi23a.html Large language models can be easily distracted by irrelevant context . In Proceedings of the 40th Internat...

    2023

  19. [27]

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint ar...

    2017 arXiv

  20. [28]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314

    2024 arXiv

  21. [29]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  22. [30]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/v1/2023.acl-long.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  23. [31]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

  24. [32]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferenc...

    2023

  25. [33]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884

    2024 arXiv

  26. [34]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  27. [35]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  28. [36]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. https://openreview.net/forum?id=5Xc1ecxO1h Tree of thoughts: Deliberate problem solving with large language models . In Thirty-seventh Conference on Neural Informat...

    2023

  29. [37]

    Tian Yu, Shaolei Zhang, and Yang Feng. 2024. https://arxiv.org/abs/2411.19443 Auto-rag: Autonomous retrieval-augmented generation for large language models . Preprint, arXiv:2411.19443

    2024 arXiv

  30. [38]

    Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2024. https://arxiv.org/abs/2410.04343 Inference scaling for long-context retrieval augmented generation . Preprint, arXiv:2410.04343

    2024 arXiv

  31. [39]

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, and Dongzhan Zhou. 2024. https://arxiv.org/abs/2410.02884 Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning...

    2024 arXiv

  32. [40]

    Chi, Quoc V Le, and Denny Zhou

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. https://openreview.net/forum?id=3bq3jsvcQ1 Take a step back: Evoking reasoning via abstraction in large language models . In The Twelfth International Conference on L...

    2024

  33. [41]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. https://arxiv.org/abs/2504.03160 Deepresearcher: Scaling deep research via reinforcement learning in real-world environments . Preprint, arXiv:2504.03160

    2025 arXiv

  34. [42]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. https://openreview.net/forum?id=WZH7099tgfM Least-to-most prompting enables complex reasoning in large language mode...

    2023

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.