Pith. sign in

REVIEW 4 major objections 6 minor 82 references

Contextual Information Policy Optimization for Search Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that rewarding search agents specifically for letting retrieved evidence influence their next reasoning action, rather than only for answer correctness, reduces confirmation bias and improves question-answering across…

desk verdict CIPO's EALR reward is a genuinely new dense signal for evidence grounding, but the masked-condition counterfactual is an unvalidated OOD attention pattern; the paper deserves a serious referee but needs a validity check on that assumption before the claim is solid. read the letter →

arxiv 2608.06128 v3 pith:I627NOJ7 submitted 2026-08-06 cs.AI

Xingyu Guo , Wei Chen , Linlin Yang , Baochang Zhang This is my paper · ORCID
classification cs.AI
keywords reinforcementlearningsearchagentsretrieval-augmentedgenerationevidencegroundingprior-drivenreasoningcreditassignmentlikelihoodratioquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search agents retrieve evidence into context, but standard reinforcement learning rewards final answer correctness or coarse progress, so an agent can earn full credit for reasoning that merely confirms what it already believed from memory. CIPO measures, at each post-retrieval turn, how much the observed next thinking-and-search action depends on the just-retrieved information, by comparing its likelihood with and without attention access to that information. This Evidence-Access Log-Likelihood Ratio (EALR) becomes a dense turn-level reward, combined with a normalized outcome reward to preserve accuracy. The paper reports that this reduces measured prior-driven reasoning and improves F1 across seven question-answering benchmarks, without human process annotations or a separate reward model.

What carries the argument

The Evidence-Access Log-Likelihood Ratio (EALR) is the central object: for each turn $t$, EALR is the token-averaged log of the ratio of the likelihood of the observed next-turn action when the previous information block $E_{i,t}$ is visible to its likelihood when attention to $E_{i,t}$ is masked during teacher-forced scoring (Eq.\ 5). The mask changes only the attention edges from the scored action tokens to that preceding block; positions, token identities, and all other context stay fixed, so the ratio isolates the influence of the new evidence. EALR is normalized within a rollout group and accumulated into a discounted turn-level advantage together with the outcome reward before the clipped policy update, which is what actually reshapes the policy. The paper also notes that the expectation of the unnormalized log-ratio under the evidence-visible policy equals the KL divergence between the visible and masked action distributions, giving the signal a policy-level interpretation.

What would settle it

A decisive test is to remove the information block at inference time for a frozen CIPO-trained policy and measure whether the immediate next action changes; if EALR does not predict which transitions actually reverse or alter when evidence is absent, the masked-likelihood counterfactual does not estimate no-evidence behavior, and the training signal is not measuring evidence grounding.

Watch

Extended reading notes

Core claim

The central claim is that prior-driven reasoning in search agents can be directly penalized by a local counterfactual: for each transition from a retrieved information block to the next thinking-and-search action, compute the token-averaged log-likelihood of that action with normal attention to the evidence minus the same action's log-likelihood when attention to the evidence block is masked. A positive difference means seeing the evidence made the actual action more probable; a difference near zero means the action was equally likely without the evidence. The paper asserts that using this difference as a dense reward, together with a global outcome reward, trains the policy to let retrieved facts guide or revise subsequent reasoning instead of merely confirming parametric guesses. In the paper's experiments this reduces prior-driven behavior and improves performance on most of the seven evaluated benchmarks.

Load-bearing premise

The load-bearing premise is that attention-masking the previous search-result block while scoring the same action faithfully estimates what the agent would have done without that evidence, even though the model was never trained with such masks.

Editorial extensions

If this is right

  • After training with CIPO, a search agent should be more likely to revise or abandon an initial hypothesis when later search results contradict it, because continuing to act as if the evidence were absent lowers the discounted advantage.
  • EALR adds one masked scoring pass per post-retrieval turn during training and no extra scoring at inference; the paper's efficiency numbers show the relative F1 gain exceeds the relative wall-clock increase for both tested model sizes.
  • Because the signal is computed from a tagged information block followed by policy-generated tokens, it transfers to other settings where tool outputs or database results are inserted into context, without re-annotating process rewards.
  • Combining EALR with outcome supervision raises supportive-evidence utilization while lowering irrelevant-evidence utilization, indicating the policy becomes more selective about which retrieved content it depends on.
  • Held-out benchmarks improve even though training used only in-domain data, suggesting the learned evidence-dependence behavior generalizes beyond the training distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that EALR could serve as a diagnostic grounding meter for any agentic system: by comparing rollout actions with and without attention access to a context block, one could audit whether a deployed agent actually conditions on retrieved or tool-provided information.
  • The next-turn scoring choice means evidence whose effect shows up only several actions later, such as changing the final answer without changing the immediate next search, receives no direct credit; a variant that scores a window of future actions could test whether longer-horizon grounding credit is needed.
  • A possible failure mode the paper does not fully rule out is reward hacking of the contrast: a policy could inflate EALR by generating verbose thought blocks that make the masked-visible likelihood gap large without genuinely changing its conclusions, so an independent check comparing EALR against human judgments of evidence use would be a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Contextual Information Policy Optimization (CIPO), a reinforcement learning framework for search agents that adds a dense, turn-level reward, the Evidence-Access Log-Likelihood Ratio (EALR), to a terminal outcome reward. EALR is computed by teacher-forcing the same next-turn action under evidence-visible and evidence-masked conditions through the frozen rollout policy and taking the token-averaged log-likelihood difference (Eq. 5). The paper shows that the expected value of the sequence-level ratio under the visible policy equals the KL divergence between visible and masked action distributions (Eq. 7). CIPO normalizes EALR rewards and outcome rewards within a rollout group, computes discounted advantages, and optimizes a clipped PPO objective with a KL penalty. Experiments on seven QA benchmarks (NQ, TQ, HotpotQA, 2Wiki, MuSiQue, Bamboogle, PopQA) with Qwen2.5-3B and 7B report consistent F1 improvements over outcome-only and process-supervised baselines, plus ablations on reward design and a counterfactual masking analysis. The paper also measures prior-driven vs evidence-driven reasoning with an LLM judge and reports reduced prior-driven rates.

Significance. Assuming the masked-likelihood identification is valid, CIPO is a clean and practical idea: it extracts a dense evidence-use signal from a single extra forward pass, requires no human process labels or an additional reward model, and is agnostic to the base RL algorithm. The KL derivation in Eq. (7) is mathematically correct, and the paper makes a good-faith effort to control confounds: all RL methods share the same retrieval environment, training data, rollout group size, and budget, and Appendix E explicitly tests whether the method merely rewards indiscriminate sensitivity to retrieved text. The reported gains are substantial (4.7-5.0 F1 points over the strongest baseline) and hold across two model scales and out-of-domain benchmarks; if reproducible, this is a meaningful advance for training grounded search agents. Code is released, which helps reproducibility.

major comments (4)
  1. [Section 3.1, Eq. (4)] The claim that EALR measures the dependence of the agent's action on retrieved evidence rests on interpreting πθold(a | h, M(E)) as the likelihood of the same action in a counterfactual 'no-evidence' condition. This identification is not validated. The attention mask that blocks the immediately preceding information block while keeping token identities, sequence positions, and all other context creates an attention pattern that the model has almost certainly never seen during pretraining or RL; softmax renormalization over the remaining keys means the masked likelihood can change for reasons unrelated to evidence content (e.g., due to the unusual position gap left by the masked block). The ablations in Table 9 vary which span is masked, but no experiment compares the block-mask condition with an actual deletion of E (with positions realigned) or with a different operationalization of evidence dependence (e.g., replacing E with a matched-length irrelevant passage). I request such a validation; if the masked-likelihood signal diverges from the deletion-based signal, the central conclusion that CIPO reduces prior-driven reasoning would be undermined.
  2. [Section 4, Tables 1-3] All experimental tables report a single run per configuration without error bars or significance testing. The headline improvements—4.7 F1 points average over IGPO/GiGPO in Table 1 and 5.0 points over GSPO in Table 3—could be within run-to-run variance for online RL with 16 rollouts per prompt. Please report at least three independent training runs per condition (mean±std) and a paired test across the seven benchmarks (e.g., Wilcoxon signed-rank) to support the claim of consistent superiority.
  3. [Section 4.4, Appendix F] The main behavioral evidence for the paper's thesis—that CIPO reduces prior-driven reasoning—comes from an LLM judge (GPT-5.5) that classifies trajectories as prior- or evidence-driven based on whether retrieved evidence influences reasoning. The EALR training reward is also a direct measure of the same construct (action-likelihood sensitivity to retrieved evidence). While the F1 improvements are independent of this metric, the behavioral claim is at risk of circularity because the evaluation metric is aligned with the optimized objective. The paper should include a human-evaluated subset (with inter-annotator agreement) and/or an observational measure (e.g., token-level attribution of answer spans to retrieved passages) to verify that the decrease in prior-driven rate is not an artifact of the judge being sensitive to the same surface features that EALR rewards.
  4. [Section 3.1, Eq. (5), Appendix D] EALR credits only the immediate next-turn action (the next <think> and <search> blocks). Evidence can change a later reasoning step or the final answer without changing the immediate next action, and in that case the dense evidence-use signal contributes nothing at the turn where retrieval occurred. Appendix D compares with a full-trajectory variant, but that variant is itself confounded by later retrieved evidence and apparently performs worse; the paper does not quantify how often evidence-induced revisions occur only after more than one turn. Please report this frequency on the validation set and clarify whether the outcome reward alone is expected to cover such delayed evidence use.
minor comments (6)
  1. [Figure 1] In the evidence bubble for the prior-driven example, 'V on Braun' should be 'von Braun' (the same typo appears in the figure text).
  2. [Abstract and Section 1] 'With this manner' is not idiomatic; use 'In this way'.
  3. [Section 4.1] The text says training is on the 'training splits of the in-domain datasets'—please specify whether the training data is the union of the four training splits or a subset, and how this matches the baselines' training data.
  4. [Section 3.2, Eq. (9)] γ is set to 1.0 in all experiments; no sensitivity analysis is reported for γ, β, ε, or the rollout group size G, so the robustness of the method to these free parameters is not demonstrated.
  5. [Appendix C] The retrieval environment is said to be 'consistent with the Search-R1 environment architecture', but the search API, number of results returned, and snippet length are not fully specified; please provide these details to make the environment reproducible.
  6. [References] Sheng et al. (2025) is cited in the text as the 'verl framework' but the reference title given is 'Hybridflow: A flexible and efficient rlhff framework'; the relationship should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the EALR signal is computed from the frozen policy's own likelihoods, and the behavioral evaluation uses an external LLM judge on held-out trajectories; the formal derivation is self-contained.

full rationale

The paper's central derivation is self-contained and does not reduce to its own inputs. EALR is defined as a token-averaged log-likelihood ratio between evidence-visible and evidence-masked teacher-forced scoring of the same collected next-turn action (Eqs. 3-5). This is a definition of a reward signal, not a fitted parameter renamed as a prediction. The likelihoods are computed from the frozen rollout policy, require no additional rollouts or reward models, and are not fitted to any evaluation label. The mathematical identity in Eq. 7, relating the expected EALR to a KL divergence, is a standard calculation rather than a circular reduction. The claim that CIPO reduces prior-driven reasoning is evaluated with an external LLM judge (GPT-5.5) on held-out trajectories, with evidence annotations that are manually verified and explicitly not used during training; the judge's criteria overlap semantically with the notion of evidence dependence, but the evaluation is an independent measurement, not a consequence of the EALR definition. The ablations in Table 9 compare alternative masking choices, showing that the design is empirically tested rather than forced by construction. No self-citation is load-bearing, and no uniqueness claim is imported from the authors' prior work. The skeptical concern about the masked-condition likelihood being out-of-distribution is a validity or correctness risk, not a circularity in the derivation. Therefore the circularity burden is low and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; EALR is a new scalar reward, not an invented entity. The main epistemic load is carried by the counterfactual masking assumption and the LLM-based behavioral evaluation.

free parameters (4)
  • discount factor gamma = 1.0
    Set in implementation details (Section 4.1); controls how EALR and outcome rewards are accumulated across turns in Eq. 9.
  • KL penalty coefficient beta = 0.001
    Chosen hyperparameter for the KL divergence penalty against the reference policy in Eq. 11; listed in Appendix C.
  • clipping epsilon = not reported
    Standard PPO clip range used in Eq. 10, but the exact value is not stated in the main text or appendix.
  • rollout group size G = 16
    Group size used for reward normalization in Eq. 8 and for averaged updates in Eq. 11; affects the stability of advantage estimates.
assumptions (3)
  • domain assumption Attention masking of the information block produces a valid counterfactual for the no-evidence condition.
    Eq. 3 and Eq. 4 define the masked condition via additive -inf attention logits; the paper assumes the masked likelihood faithfully represents what the policy would do without evidence, although the model was never trained with such masks.
  • domain assumption LLM judge (GPT-5.5) classifications of prior-driven versus evidence-driven reasoning are valid.
    Section 4.4 and Appendix F use a closed-source judge for the central behavioral claim; the reliability of that judge is not independently established.
  • domain assumption The local retrieval service returns relevant search results as in the Search-R1 environment.
    Section 4.1 states the use of a local retrieval service consistent with Search-R1; the correctness and coverage of that service are taken as given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextual Information Policy Optimization for Search Agents." pith.science (2026). https://pith.science/paper/I627NOJ7

@misc{pith2026260806128,
  author       = {Pith},
  title        = {Pith review of: Contextual Information Policy Optimization for Search Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I627NOJ7}},
  note         = {Machine review of arXiv:2608.06128}
}
read the original abstract

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use external evidence during multi-step reasoning. For knowledge-intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant evidence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer correctness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reasoning: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirmation bias and inefficient evidence use. To address this issue, we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning actions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to preserve answer correctness. With this manner, CIPO discourages evidence-detached guesses and promotes reasoning trajectories in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive experiments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven reasoning and achieves excellent performance on most tasks.

Figures

Figures reproduced from arXiv: 2608.06128 by the authors.

Figure 1
Figure 1. Illustrative case for the same question. Prior-driven [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of CIPO. It evaluates each post-retrieval reasoning step under evidence-visible and evidence [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Representative step-wise validation-set F1 trajectories on two in-domain and two out-of-domain benchmarks. Each [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prior-driven rates during training. CIPO maintains [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Evidence-driven rates on the test sets across seven [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Additional step-wise validation-set F1 trajectories for the benchmarks not shown in the main paper. Each panel [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 18 linked inside Pith

  1. [1]

    Ahmadian, A.; Cremer, C.; Gall \'e , M.; Fadaee, M.; Kreutzer, J.; Pietquin, O.; \"U st \"u n, A.; and Hooker, S. 2024. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12248--12267

    2024

  2. [2]

    Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, 9112--9141

    2024

  3. [4]

    Dai, Y.; Yang, S.; Wang, G.; Deng, Y.; Zhang, Z.; Yin, J.; Zeng, P.; Ying, Z.; Meng, C.; Yi, C.; et al. 2026. Careful queries, credible results: Teaching rag models advanced web search tools with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence

    2026

  4. [6]

    Feng, L.; Xue, Z.; Liu, T.; and An, B. 2026. Group-in-group policy optimization for llm agent training. NeurIPS

    2026

  5. [9]

    Goel, H.; Udathu, A.; Jabbireddy, S.; Kalkar, P.; and Parulekar, A. 2026. S ^3 -R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data. arXiv preprint arXiv:2605.01248

    arXiv 2026

  6. [10]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081): 633--638

    2025

  7. [11]

    He, B.; Hu, M.; Xu, Z.; Wang, H.; Zong, L.; Chen, Y.; Ma, C.; Liu, X.; Zhou, P.; and King, I. 2026. Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration. ICML

    2026

  8. [12]

    D.; Sugawara, S.; and Aizawa, A

    Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, 6609--6625

    2020

Show all 82 references
  1. [15]

    S.; and Zettlemoyer, L

    Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601--1611

    2017

  2. [16]

    Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics

    2019

  3. [17]

    u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K \"u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \"a schel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS

    2020

  4. [19]

    Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; and Hajishirzi, H. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL

    2023

  5. [20]

    S.; et al

    Ning, L.; Liang, Z.; Jiang, Z.; Qu, H.; Ding, Y.; Fan, W.; Wei, X.-y.; Lin, S.; Liu, H.; Yu, P. S.; et al. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conference on Knowle...

    2025

  6. [21]

    A.; and Lewis, M

    Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, 5687--5711

    2023

  7. [22]

    L.; Lai, H.; Sun, X.; Sun, J.; Yang, X.; Yang, Y.; Yao, S.; Xu, W.; et al

    Qi, Z.; Liu, X.; Iong, I. L.; Lai, H.; Sun, X.; Sun, J.; Yang, X.; Yang, Y.; Yao, S.; Xu, W.; et al. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, volume 2025, 79791--79821

    2025

  8. [23]

    Qu, C.; Dai, S.; Wei, X.; Cai, H.; Wang, S.; Yin, D.; Xu, J.; and Wen, J.-R. 2025. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8): 198343

    2025

  9. [26]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, 1279--1297

    2025

  10. [28]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539--554

    2022

  11. [29]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long p...

    2023

  12. [31]

    Wang, G.; Dai, S.; Ye, G.; Gan, Z.; Yao, W.; Deng, Y.; Wu, X.; and Ying, Z. 2026 b . Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents. ICLR

    2026

  13. [32]

    Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024 a . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024

  14. [33]

    Wang, X.; Wang, Z.; Gao, X.; Zhang, F.; Wu, Y.; Xu, Z.; Shi, T.; Wang, Z.; Li, S.; Qian, Q.; et al. 2024 b . Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 17716--17736

    2024

  15. [36]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS

    2022

  16. [38]

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, 2369--2380

    2018

  17. [39]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2022. React: Synergizing reasoning and acting in language models. ICLR

    2022

  18. [40]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 113222--113244

    2026

  19. [43]

    Zhang, W.; Li, X.; Dong, K.; Wang, Y.; Jia, P.; Li, X.; Zhang, Y.; Xu, D.; Du, Z.; Guo, H.; et al. 2026. Process vs. outcome reward: Which is better for agentic rag reinforcement learning. Advances in Neural Information Processing Systems, 38: 58701--58729

    2026

  20. [45]

    Zheng, Y.; Fu, D.; Hu, X.; Cai, X.; Ye, L.; Lu, P.; and Liu, P. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 414--431

    2025

  21. [46]

    Zheng, Y.; Sun, S.; Qiu, L.; Ru, D.; Jiayang, C.; Li, X.; Lin, J.; Wang, B.; Luo, Y.; Pan, R.; et al. 2024. OpenResearcher: unleashing AI for accelerated scientific research. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demo...

    2024

  22. [47]

    NeurIPS , year=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. NeurIPS , year=

  23. [48]

    arXiv preprint arXiv:2604.17555 , year=

    CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search , author=. arXiv preprint arXiv:2604.17555 , year=

    arXiv

  24. [49]

    Goel, Harsh and Udathu, Akhil and Jabbireddy, Susmija and Kalkar, Pradnesh and Parulekar, Atharva , journal=

  25. [50]

    arXiv preprint arXiv:2606.16409 , year=

    PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2606.16409 , year=

  26. [51]

    ICLR , year=

    React: Synergizing reasoning and acting in language models , author=. ICLR , year=

  27. [52]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

  28. [53]

    International conference on learning representations , volume=

    Self-rag: Learning to retrieve, generate, and critique through self-reflection , author=. International conference on learning representations , volume=

  29. [54]

    arXiv preprint arXiv:2503.09516 , year=

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning , author=. arXiv preprint arXiv:2503.09516 , year=

    arXiv

  30. [55]

    ICML , year=

    Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration , author=. ICML , year=

  31. [56]

    ICLR , year=

    Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents , author=. ICLR , year=

  32. [57]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  33. [58]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    arXiv

  34. [59]

    NeurIPS , year=

    Chain-of-thought prompting elicits reasoning in large language models , author=. NeurIPS , year=

  35. [60]

    EMNLP , year=

    Search-o1: Agentic search-enhanced large reasoning models , author=. EMNLP , year=

  36. [61]

    arXiv preprint arXiv:2503.05592 , year=

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2503.05592 , year=

    arXiv

  37. [62]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  38. [63]

    arXiv preprint arXiv:2505.15107 , year=

    Stepsearch: Igniting llms search ability via step-wise proximal policy optimization , author=. arXiv preprint arXiv:2505.15107 , year=

    arXiv

  39. [64]

    outcome reward: Which is better for agentic rag reinforcement learning , author=

    Process vs. outcome reward: Which is better for agentic rag reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  40. [65]

    NeurIPS , year=

    Group-in-group policy optimization for llm agent training , author=. NeurIPS , year=

  41. [66]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  42. [67]

    Buy 4 reinforce samples, get a baseline for free! , author=

  43. [68]

    arXiv e-prints , pages=

    Reinforce++: A simple and efficient approach for aligning large language models , author=. arXiv e-prints , pages=

  44. [69]

    arXiv preprint arXiv:2507.18071 , year=

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    arXiv

  45. [70]

    Transactions of the Association for Computational Linguistics , year=

    Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , year=

  46. [71]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  47. [72]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  48. [73]

    Proceedings of the 28th International Conference on Computational Linguistics , pages=

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps , author=. Proceedings of the 28th International Conference on Computational Linguistics , pages=

  49. [74]

    Transactions of the Association for Computational Linguistics , volume=

    MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  50. [75]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Measuring and narrowing the compositionality gap in language models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  51. [76]

    ACL , year=

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories , author=. ACL , year=

  52. [77]

    arXiv preprint arXiv:2412.15115 , year =

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =

    arXiv

  53. [78]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  54. [79]

    arXiv preprint arXiv:2312.10997 , year=

    Retrieval-augmented generation for large language models: A survey , author=. arXiv preprint arXiv:2312.10997 , year=

    arXiv

  55. [80]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Searching for best practices in retrieval-augmented generation , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  56. [81]

    arXiv preprint arXiv:2509.00877 , year=

    EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes , author=. arXiv preprint arXiv:2509.00877 , year=

  57. [82]

    arXiv preprint arXiv:2411.00744 , year=

    Corag: A cost-constrained retrieval optimization system for retrieval-augmented generation , author=. arXiv preprint arXiv:2411.00744 , year=

  58. [83]

    arXiv preprint arXiv:2503.00223 , year=

    Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning , author=. arXiv preprint arXiv:2503.00223 , year=

    arXiv

  59. [84]

    Advances in Neural Information Processing Systems , year=

    Learning to reason with search for llms via reinforcement learning , author=. Advances in Neural Information Processing Systems , year=

  60. [85]

    arXiv preprint arXiv:2505.17005 , year=

    R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning , author=. arXiv preprint arXiv:2505.17005 , year=

    arXiv

  61. [86]

    International Conference on Learning Representations , volume=

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning , author=. International Conference on Learning Representations , volume=

  62. [87]

    arXiv preprint arXiv:2503.23383 , year=

    Torl: Scaling tool-integrated rl , author=. arXiv preprint arXiv:2503.23383 , year=

    arXiv

  63. [88]

    Proceedings of the AAAI Conference on Artificial Intelligence , year=

    Careful queries, credible results: Teaching rag models advanced web search tools with reinforcement learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , year=

  64. [89]

    arXiv preprint arXiv:2508.12800 , year=

    Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward , author=. arXiv preprint arXiv:2508.12800 , year=

    arXiv

  65. [90]

    arXiv preprint arXiv:2506.18959 , year=

    From web search towards agentic deep research: Incentivizing search with reasoning agents , author=. arXiv preprint arXiv:2506.18959 , year=

    arXiv

  66. [91]

    arXiv preprint arXiv:2506.18096 , year=

    Deep research agents: A systematic examination and roadmap , author=. arXiv preprint arXiv:2506.18096 , year=

    arXiv

  67. [92]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V

    A survey of webagents: Towards next-generation ai agents for web automation with large foundation models , author=. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 , pages=

  68. [93]

    Frontiers of Computer Science , volume=

    Tool learning with large language models: A survey , author=. Frontiers of Computer Science , volume=. 2025 , publisher=

    2025

  69. [94]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    OpenResearcher: unleashing AI for accelerated scientific research , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

    2024

  70. [95]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  71. [96]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  72. [97]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  73. [98]

    arXiv preprint arXiv:2509.02547 , year=

    The landscape of agentic reinforcement learning for llms: A survey , author=. arXiv preprint arXiv:2509.02547 , year=

    arXiv

  74. [99]

    arXiv preprint arXiv:2501.15602 , year=

    Rethinking external slow-thinking: From snowball errors to probability of correct reasoning , author=. arXiv preprint arXiv:2501.15602 , year=

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.