pith. sign in

arxiv: 2606.00590 · v1 · pith:25D7MY63new · submitted 2026-05-30 · 💻 cs.IR · cs.AI

Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback

Md Zarif Ul Alam , Alireza Salemi , Hamed Zamani This is my paper

Pith reviewed 2026-06-28 18:21 UTC · model grok-4.3

classification 💻 cs.IR cs.AI
keywords agentic searchcritic modelretrieval feedbackquery refinementmulti-hop question answeringautomatic supervisioninstruction tuningintrospective reasoning
0
0 comments X

The pith

A critic model that assesses whether retrieved evidence supports the next reasoning step improves retrieval quality and answer accuracy in agentic search.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agentic search systems use iterative retrieval to handle complex queries, but optimizing the retriever usually demands costly annotations or co-training. The paper establishes that a separate critic model can evaluate the agent's reasoning trace to decide if the current retrieval is adequate for the next step. This judgment drives both an inference loop that refines queries and instructions and a training method that turns successful and failed refinement paths into supervision signals. Sympathetic readers would care because the approach removes the need for manual relevance labels while delivering gains on standard multi-hop question answering benchmarks.

Core claim

Critic-R closes the feedback loop by introducing a critic model that evaluates the agent's introspective reasoning trace after consuming retrieved evidence to determine whether the retrieved context sufficiently supports the next reasoning step. The framework then applies this signal in Critic-R-Zero, an inference-time query refinement loop that iteratively rewrites queries and retrieval instructions, and in Critic-Embed, an optimization approach that uses successful and failed refinement trajectories as automatic supervision for the retrieval model.

What carries the argument

The critic model, which judges the sufficiency of retrieved context for the agent's next reasoning step based on the introspective trace.

If this is right

  • Both retrieval quality and downstream answer accuracy improve on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle.
  • Retriever training no longer requires gold-standard relevance annotations.
  • The same critic signal works for both inference-time refinement and model optimization.
  • Instruction-tuned retrievers benefit from the natural language feedback.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This feedback mechanism could be applied to other iterative decision processes where an agent interacts with external tools.
  • Performance gains may depend on the critic being trained on high-quality reasoning traces from the base agent.
  • Extending the critic to evaluate partial reasoning chains rather than single steps might yield further improvements.

Load-bearing premise

The critic model produces judgments that reliably indicate whether the retrieved context supports the next reasoning step, and these judgments can be trusted as supervision.

What would settle it

A controlled experiment where the critic's outputs are replaced with random or inverted judgments and the system is rerun to check whether the reported gains in retrieval quality and answer accuracy vanish.

Figures

Figures reproduced from arXiv: 2606.00590 by Alireza Salemi, Hamed Zamani, Md Zarif Ul Alam.

Figure 1
Figure 1. Figure 1: Critic-R Overview. diately after consuming the retrieved evidence—to determine whether the retrieved context is suffi￾cient for the next reasoning step. This design uses the observation that the agent often explicitly indi￾cates whether the retrieved documents contain the information required to continue reasoning. This verification signal enables two complemen￾tary mechanisms: (1) Critic-R-Zero (Inference… view at source ↗
Figure 2
Figure 2. Figure 2: Multi-hop average F1 vs. retrieval depth [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: System prompt for the reasoning agent MR. The placeholder {question} is replaced at runtime with the input question. Satisfaction Judgment Prompt PJ You are an evaluator for a search problem. You will be given a global search query, a local sub-query, retrieved documents, and critique feedback from a reasoning model. Your ONLY task is to evaluate if the retrieved documents are satisfactory for answering th… view at source ↗
Figure 4
Figure 4. Figure 4: Satisfaction judgment prompt PJ . Given the global question, the current sub-query, the retrieved documents, and the reasoner’s introspective feedback, the critic emits a binary verdict together with a diagnostic reason. Query Refinement Prompt PR You are a search query optimizer. The previous search failed to retrieve satisfactory documents. Global Query: {og_query} Failed Sub-query: {sub_query} Failed In… view at source ↗
Figure 5
Figure 5. Figure 5: Query refinement prompt PR. Invoked only when PJ returns no, the critic uses the diagnostic reason to rewrite the failed sub-query and retrieval instruction for the next attempt. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
read the original abstract

Agentic search systems iteratively interact with retrieval models to answer complex queries. Despite substantial progress, optimizing retrievers for agentic search remains challenging, often requiring heavy co-training or gold-standard annotations that limit real-world applicability. We propose Critic-R, a framework that explicitly closes the feedback loop between the reasoning agent and the retrieval model during both inference and training. Critic-R introduces a critic model that evaluates the agent's introspective reasoning trace after consuming retrieved evidence to determine whether the retrieved context sufficiently supports the next reasoning step. Critic-R has two complementary mechanisms: Critic-R-Zero, an inference-time query refinement loop that iteratively rewrites queries and retrieval instructions, and Critic-Embed, an optimization approach for retrieval models that leverages successful and failed refinement trajectories as automatic supervision without requiring manual relevance annotation. We evaluate Critic-R on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle. Results show that Critic-R significantly improves both retrieval quality and downstream answer accuracy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes Critic-R, a framework for agentic search that introduces a critic model to evaluate whether retrieved context sufficiently supports the agent's next reasoning step based on introspective traces. This enables two mechanisms: Critic-R-Zero (inference-time iterative query refinement) and Critic-Embed (using successful/failed trajectories as automatic positive/negative supervision to optimize retrievers without manual relevance annotations). The approach is evaluated on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle, with claims of significant improvements in both retrieval quality and downstream answer accuracy.

Significance. If the central claims hold after addressing validation gaps, the work would be significant for reducing dependence on gold-standard annotations in training retrievers for multi-hop agentic systems. The closed feedback loop via natural language critic judgments offers a practical path to scalable optimization, though its value hinges on the reliability of those judgments as supervision signals.

major comments (2)
  1. [Abstract] Abstract: The claim that Critic-Embed supplies supervision 'without requiring manual relevance annotation' treats critic judgments on 'whether the retrieved context sufficiently supports the next reasoning step' as reliable ground truth for both inference refinement and trajectory labeling. No human validation, inter-annotator agreement, or correlation with human labels on this specific judgment is reported, which is load-bearing for the no-annotation claim and the reported gains.
  2. [Abstract] Abstract: No experimental details, baselines, error bars, ablation results, or critic accuracy metrics are visible, preventing assessment of whether the reported gains on HotpotQA et al. support the central claim or could arise from other components or critic artifacts.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful comments highlighting the importance of validating the critic judgments and ensuring experimental details are clear. We address each major comment below and outline planned revisions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that Critic-Embed supplies supervision 'without requiring manual relevance annotation' treats critic judgments on 'whether the retrieved context sufficiently supports the next reasoning step' as reliable ground truth for both inference refinement and trajectory labeling. No human validation, inter-annotator agreement, or correlation with human labels on this specific judgment is reported, which is load-bearing for the no-annotation claim and the reported gains.

    Authors: We agree that the absence of human validation for the critic's judgments is a substantive gap, as these judgments serve as the automatic supervision signal. The manuscript correctly states that no manual relevance annotations are collected for training the retriever, but it does not demonstrate that the critic judgments correlate with human assessments of support for the next reasoning step. In the revision we will add a human study on a held-out set of trajectories, reporting agreement rates and inter-annotator agreement to substantiate the reliability of the labels. revision: yes

  2. Referee: [Abstract] Abstract: No experimental details, baselines, error bars, ablation results, or critic accuracy metrics are visible, preventing assessment of whether the reported gains on HotpotQA et al. support the central claim or could arise from other components or critic artifacts.

    Authors: The abstract is intentionally concise. The full manuscript (Section 4 and Appendix) reports the full experimental protocol, baselines (including DPR, Contriever, and prior agentic retrievers), results with standard error bars across three seeds, ablations isolating the critic and refinement loop, and critic accuracy metrics (precision/recall of positive/negative trajectory labels). We will add one sentence to the abstract summarizing the scale of gains and the presence of these controls if the editor permits. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical framework with external benchmarks

full rationale

The paper describes an empirical framework (Critic-R with Critic-R-Zero and Critic-Embed) that uses a critic model to generate automatic supervision signals from reasoning traces. Performance is measured on standard QA benchmarks (HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle) that supply independent gold answers and relevance labels. No mathematical derivation chain, fitted parameters renamed as predictions, or self-citation load-bearing steps appear in the abstract or description. The use of critic judgments for labeling is a methodological choice whose validity is tested externally rather than reducing to the inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities can be extracted or audited.

pith-pipeline@v0.9.1-grok · 5709 in / 1008 out tokens · 19575 ms · 2026-06-28T18:21:21.840301+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages · 8 internal anchors

  1. [1]

    A Asai, W Zeqiu, W Yizhong, and 1 others. 2023a. Self-rag: learning to retrieve, generate, and cri- tique through self-reflection. arxiv.arXiv preprint arXiv:2310.11511. Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023b. Retrieval-based language models and applications. InProceedings of the 61st Annual Meet- ing of the Association for Computation...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Retrieval- augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2(1):32. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Atlas: Few-shot learning with retrieval augmented language models.Journal of Machine Learning Research, 24(251):1–43. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025a. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Hai...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781

    Dense passage retrieval for open- domain question answering. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, and 1 others

    2020

  5. [5]

    Let's Verify Step by Step

    Let’s verify step by step, 2023.URL https://arxiv. org/abs/2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi

    Agentic- r: Learning to retrieve for agentic search.arXiv preprint arXiv:2601.11888. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi

    work page arXiv

  7. [7]

    InFindings of the Association for Computa- tional Linguistics: EMNLP 2023, pages 5687–5711

    Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computa- tional Linguistics: EMNLP 2023, pages 5687–5711. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham

    2023

  8. [8]

    InFind- ings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274

    En- hancing retrieval-augmented large language models with iterative retrieval-generation synergy. InFind- ings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274. Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Richard James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih

    2023

  9. [9]

    Replug: Retrieval- augmented black-box language models. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 8371–8384. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Rong Wen

    2024

  10. [10]

    R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 629–648

    Training a utility-based retriever through shared con- text attribution for retrieval-augmented language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 629–648. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning

    2025

  13. [13]

    InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 2369–2380

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 2369–2380. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao

    2018

  14. [14]

    ReAct: Synergizing Reasoning and Acting in Language Models

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629. Hamed Zamani and Michael Bendersky

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search

    Cosearch: Joint training of reasoning and document ranking via reinforce- ment learning for agentic search.arXiv preprint arXiv:2604.17555. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025a. Jasper and stella: distillation of sota embedding models.Preprint, arXiv:2412.19048. Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    InPro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 414–431

    10 Deepresearcher: Scaling deep research via reinforce- ment learning in real-world environments. InPro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 414–431. A General-Domain QA Results Table 5 reports the inference-time scaling results forCritic-R-Zeroon the three general-domain QA benchmarks (NQ, TriviaQA...

    2025

  17. [17]

    B Implementation Details Reasoning Agent ( MR):The reasoning agent is an instruction-tuned LLM operating under the ReAct paradigm (Yao et al., 2022)

    The same trends observed on the multi-hop suite hold here: any critic reliably improves over theno-criticab- lation across all reasoner scales, and the largest critic (Qwen2.5-72B) typically yields the strongest average performance. B Implementation Details Reasoning Agent ( MR):The reasoning agent is an instruction-tuned LLM operating under the ReAct par...

    2022

  18. [18]

    D Critic-Embed Training Details Critic-Embed is initialized from Stella-400M em- bedding model (Zhang et al., 2025a) 7 and fine- tuned with InfoNCE (temperature τ= 0.02 )

    test 14,267 Table 6: Evaluation set sizes for the QA datasets used in our experiments. D Critic-Embed Training Details Critic-Embed is initialized from Stella-400M em- bedding model (Zhang et al., 2025a) 7 and fine- tuned with InfoNCE (temperature τ= 0.02 ). The effective batch size is 128 (per-device 32 with 4- step gradient accumulation), trained for 5 ...

    work page arXiv