Pith. sign in

REVIEW 4 major objections 5 minor 93 references

Adding turn-level rewards for using retrieved evidence — not just landing on the right answer — lifts a search agent's reasoning faithfulness and keeps or improves its accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A turn-level faithfulness reward improves a Search-R1-style agent's Information-Think and Think-Answer faithfulness as judged by the same reward model used for training, while task accuracy is roughly unchanged.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The faithfulness taxonomy is a useful contribution, but the headline gains are measured with the same reward model that generated the training rewards, so the empirical claim is not yet established. the 4 major comments →

arxiv 2510.13272 v3 pith:3Z5HKBSH submitted 2025-10-15 cs.CL

Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

Zhichao Xu , Zongyu Wu , Yun Zhou , Aosong Feng , Kang Zhou , Sangmin Woo , Kiran Ramnath , Yijun Tian
show 4 more authors
Xuan Qi Weikang Qiu Lin Lee Cheong Haibo Ding
This is my paper
classification cs.CL
keywords retrieval-augmented generationagentic searchreinforcement learningchain-of-thought faithfulnessprocess rewardsreward modelquestion answeringRLVR
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search agents trained to maximize final-answer correctness can get the right answer while ignoring or distorting the evidence they retrieved in the middle of their reasoning. The paper formalizes this failure as three traceability checks — do the thoughts justify the search queries, do the thoughts use the retrieved information, and is the final answer grounded in the thoughts — and shows that a strong outcome-trained agent (Search-R1) fails primarily on the information-to-thought link. It then introduces VERITAS, which turns the two problematic checks into fine-grained turn-level rewards added to the exact-match reward in reinforcement learning. On seven QA benchmarks, VERITAS-R1 improves Information-Think faithfulness by about 15% relative to Search-R1 and Think-Answer faithfulness by about 3%, while matching or slightly improving exact-match accuracy. A sympathetic reader should care because this suggests process supervision can make retrieval-augmented agents more trustworthy without trading away performance.

Core claim

The central claim is that unfaithful chain-of-thought in RL-trained search agents is not an inevitable byproduct of tool use but a misspecified reward: outcome-only training never asks whether the reasoning steps are true to the retrieved evidence. VERITAS operationalizes that by rewarding, at each turn, (1) whether the thought block actually engages the retrieved information (Information-Think faithfulness) and (2) whether the final answer is entailed by the last thought (Think-Answer faithfulness), with weights 0.9 on exact match and smaller weights on the two faithfulness signals. Using the same PPO pipeline, retriever, and corpus as Search-R1, the resulting policy's average Information-T

What carries the argument

The load-bearing object is the turn-level faithfulness reward computed over the agent's structured trajectory — the <think>, <search>, <information>, <answer> blocks. A distilled reward model (a Qwen2.5-14B fine-tuned with LoRA on labels from a stronger judge) scores each transition; in particular, the Information-Think score asks whether the new think block is a genuine synthesis of the just-retrieved information. This score is added as a dense per-turn reward to the sparse exact-match outcome reward with a small weight, creating an optimization signal at the exact point where unfaithfulness occurs. The Think-Search dimension is measured but, because outcome-only models already score high o

Load-bearing premise

The paper's central claim depends on its trained reward model being a truthful judge of whether the agent's thinking uses the retrieved evidence, and the same judge is used for both training and evaluation — if that judge rewards format rather than substance, the reported faithfulness gains could be an artifact.

What would settle it

Score held-out VERITAS-R1 trajectories with a faithfulness judge that was not used anywhere in training (fresh human annotation, or a strong judge excluded from the reward-model distillation) and compare the Information-Think score against Search-R1; the paper's reported gains should survive this re-scoring. A complementary counterfactual: take a trajectory, replace a retrieved passage with contradictory content, and check whether the following think block and final answer change — if they do not, the model is not actually using the evidence.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, process-based faithfulness rewards are a viable training objective: models can be made to reason from evidence rather than merely to the answer, and the effect shows up on held-out QA datasets without an accuracy penalty.
  • The Information-Think reward does most of the work, so future agentic-search training can focus on the evidence-to-thought link and spend less effort on answer-to-thought alignment.
  • Because Search-R1 already had high Think-Search faithfulness, outcome-only RL appears to teach agents to search deliberately; the gap is in using what they find and grounding the final answer.
  • The evaluation framework (three traceability checks over tagged trajectories) can serve as a general diagnostic for any retrieval or tool-using agent with structured outputs.
  • A small Think-Answer reward improves answer grounding on target datasets, but too large a weight overfits to training domains — so the practical recipe is asymmetric weights across faithfulness dimensions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same evidence-to-thought reward could be applied to any RL agent that alternates observations and reasoning — web automation, code interpreters, database querying — since the Information-Think check does not depend on search syntax, only on whether the next reasoning step uses the most recent observation.
  • Editorial inference: because the reward model used for training is the same one used to measure the results, an independent evaluation — fresh human labels on final VERITAS-R1 trajectories, or a judge model excluded from distillation — is the key untested check on whether the gains reflect genuine evidence use rather than optimization against the training judge.
  • Editorial inference: the regex-based Think-Answer metric likely undercounts faithful paraphrases, so the true answer-grounding improvement could be larger than reported; a semantic-entailment version of the metric would sharpen the measurement.
  • Editorial inference: the asymmetry in reward weights implies a practical rule of thumb — moderate weight on evidence-use, small weight on answer-grounding — which could be tuned dynamically by domain difficulty in future systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies faithfulness of RL-trained agentic search systems. It defines three faithfulness dimensions (Think-Search, Information-Think, Think-Answer), evaluates Search-R1, and proposes VERITAS, which adds turn-level faithfulness rewards R_info-think and R_think-answer to the exact-match reward (Eq. 1), using a distilled Qwen2.5-14B reward model. Experiments on seven QA benchmarks report that VERITAS-R1 improves Information-Think faithfulness substantially and Think-Answer faithfulness modestly while maintaining task accuracy. The manuscript includes ablations, hyperparameter sensitivity studies, and reward-model scaling experiments.

Significance. If the empirical claims held, the paper would provide a practical demonstration that process supervision on faithfulness can improve reasoning reliability without sacrificing accuracy, and its trajectory-level faithfulness metrics could become a reusable evaluation framework. Strengths include a clear formalization of the three faithfulness dimensions, reported reward-model agreement statistics, and ablation over reward components and weights. However, the headline faithfulness gains are measured with the same reward model used to train the policy, and the human validation does not cover final policy trajectories; the significance is therefore conditional on independent verification. The inconsistent headline numbers in the abstract versus Table 6 further reduce the current reliability of the reported effects.

major comments (4)
  1. [Section 4, Eq. (1), Figure 3/Table 6] The central faithfulness improvement is evaluated with the same reward model used as the RL reward. Section 4 states 'we use this fine-tuned RM for all subsequent RL training and evaluation.' The RM is distilled from Claude-3.7 labels on Search-R1 trajectories; the 50-sample human check (Table 9) is on the RM test split, again Search-R1 trajectories, not rollouts of the trained VERITAS-R1 policy. Because R_info-think in Eq. (1) directly rewards the RM's judgment, the Table 6 gains (Info-Think 0.607 to 0.849 for EM+Info-Think; 0.607 to 0.700 for full VERITAS) may reflect optimizing surface cues of that judge rather than genuine faithfulness. Please evaluate final VERITAS-R1 rollouts with an independent judge (e.g., Claude-3.7-Sonnet or human annotators) and report agreement and per-trajectory examples.
  2. [Section 3.3, Think-Answer metric and Eq. (1)] The regex metric counts an answer as faithful if its content appears in the preceding <think> block. Since R_think-answer directly rewards exactly this substring match, the reported Think-Answer increase (Table 6: 0.805 to 0.831 for full VERITAS; 0.805 to 0.853 for EM+Think-Ans) may simply reflect the policy copying the answer into the thought block. This is metric gaming, not evidence of entailed reasoning. Please report qualitative examples and/or evaluate Think-Answer with a semantic entailment judge on final trajectories.
  3. [Abstract/Introduction vs Table 6] The paper claims 'improves information-think faithfulness by 15.3% and think-answer faithfulness by 3.2% over Search-R1.' Table 6 reports absolute scores: baseline Info-Think 0.607 vs EM+Info-Think 0.849 (+24.2pp) and full VERITAS 0.700 (+9.3pp); Think-Answer baseline 0.805 vs full 0.831 (+2.6pp) and EM+Think-Ans 0.853 (+4.8pp). The 15.3%/3.2% figures do not match any row or variant. Specify which model variant and whether the percentages are absolute or relative, and align the abstract with the table.
  4. [Section 5.2 and Table 6] The text states 'VERITAS-R1 consistently outperforms the strong Search-R1 baseline across the majority of datasets' and the Conclusion says it 'achieves higher task accuracy.' Table 6 shows EM+Info-Think exceeds baseline on 4/7 datasets (NQ, TriviaQA, HotpotQA, 2wiki) and is lower on PopQA, Musique, Bamboogle; EM+Info-Think+Think-Ans exceeds baseline on only 3/7. Average EM differences are +0.006 and +0.003. No confidence intervals or significance tests are reported. The data support 'maintains accuracy' at best, not 'higher accuracy.' Please revise the claim and add statistical evaluation.
minor comments (5)
  1. [Section 2] Typo: 'consistenting' should be 'consisting'; 'RL to d-based reasoning' should be 'RL-based reasoning.'
  2. [Table 6 note] The note says EM results may differ from Table 1 due to vLLM randomness. If sampling is stochastic, average over multiple inference seeds or fix the seed so that the comparisons are not confounded by randomness.
  3. [Section G, Table 9] The human evaluation is performed by a single author. Please provide annotation instructions, report whether the annotator was blind to the model condition, and ideally use multiple independent annotators with agreement statistics.
  4. [Abstract vs Conclusion] The abstract says 'comparable task performance' while the Conclusion says 'higher task accuracy'; these are inconsistent and should be aligned.
  5. [Section 5.2] The word 'skyrockets' is subjective; replace with the precise numerical change.

Circularity Check

2 steps flagged

Headline faithfulness gains are measured by the same reward model and regex used as training rewards; no independent judge on final VERITAS-R1 policy, so the gains are partly forced by construction.

specific steps
  1. fitted input called prediction [Section 4 'Practical implementation via a distilled reward model'; Section 5.2 Figure 3 and Table 6]
    "The final reward function is a weighted sum: R=w EM · REM +w info-think · Rinfo-think +wthink-answer · Rthink-answer (1) ... Given its strong performance and efficiency, we use this fine-tuned RM for all subsequent RL training and evaluation."

    The Info-Think component of Eq. (1) is scored by the fine-tuned Qwen2.5-14B reward model, and the Info-Think faithfulness numbers in Figure 3/Table 6 are scored by that same RM ('use this fine-tuned RM for all subsequent RL training and evaluation'). The policy is therefore trained to maximize the very function that is later reported as the faithfulness improvement; a rise in that number is the training objective itself, not an independent measurement. The 50-sample human check (Table 9) was run on the RM's evaluation split of Search-R1 trajectories, not on final VERITAS-R1 rollouts, so it does not validate RM judgments on the trained policy's distribution. The headline 15.3pp Info-Think gain is thus partly forced by construction.

  2. fitted input called prediction [Section 3.3 'Regex-Based metrics for answer faithfulness'; Eq. (1); Section 5.2 Table 6]
    "We assess whether the content of the <answer></answer> block is present in its preceding thinking content, <think></think>. ... The answer is considered faithful if a match is found, confirming that the final output was directly extracted from the last reasoning process before giving the answer."

    This regex match is used both as Rthink-answer in Eq. (1) and as the reported Think-Answer faithfulness metric (Table 6). The RL policy is directly rewarded to make answer text appear in the preceding think block; copying the answer into the thought block satisfies the metric by construction. Consequently the reported Think-Answer gains (e.g., 0.805 to 0.831 for the combined model, 0.853 for +Think-Ans) reflect optimization of the regex, not independent evidence of entailment or grounding. The paper's own limitation notes the regex 'may lack recall' and cannot recognize paraphrase/inference, so it is not a neutral validator of the training claim.

full rationale

VERITAS's contribution is a training objective; the paper's own derivation (Eq. 1) is explicit. The circularity is not in the math but in the measurement: R_info-think is produced by a distilled RM trained on Claude-3.7 labels of Search-R1 trajectories, and the same RM is used to report Info-Think faithfulness on VERITAS-R1. Since RL maximizes this RM score, the reported improvement is the training signal itself. The human validation (50 samples, Table 9) was on the RM's evaluation split of Search-R1 trajectories, so it does not certify RM behavior on the final policy's outputs. Similarly, Think-Answer faithfulness is a regex match that is also R_think-answer; it can be gamed by copying. Task accuracy (EM) results are independent and not circular, and no load-bearing self-citation or imported uniqueness theorem appears. But the central faithfulness claim, as measured, is not independent of the training reward; score 7 reflects partial circularity rather than a fully tautological derivation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on one tuned weighted reward function and on the assumption that a distilled judge trained on Claude labels is a valid, transferable measure of faithfulness. The judge itself is the main unexamined load-bearing component.

free parameters (3)
  • w_EM = 0.9
    Weight on exact-match reward in Eq. (1); selected from the hyperparameter sensitivity study in Figure 4.
  • w_info-think = 0.05
    Weight on Information-Think faithfulness reward; reported as the optimal balance after tuning.
  • w_think-ans = 0.02
    Weight on Think-Answer reward; set lower because 0.05 'leads to overfit on the trainset' (Section 5.2).
axioms (4)
  • domain assumption LLM-as-a-judge (Claude-3.7-Sonnet) faithfulness labels are valid ground truth for Information-Think and Think-Search.
    Used to build the reward-model training set; no independent benchmark, only a 50-sample human agreement check on RM test examples.
  • domain assumption A regex match between the final answer and the preceding thinking block measures Think-Answer faithfulness.
    The paper itself notes this metric lacks recall and fails on paraphrasing (Limitations section).
  • ad hoc to paper The distilled RM's judgments transfer from Search-R1 trajectories to the final VERITAS-R1 policy trajectories.
    The RM was trained and labeled on Search-R1 trajectories but used to evaluate the new policy; no validation on final policy outputs is provided.
  • ad hoc to paper Information-Think faithfulness is the key optimization target, while Think-Search is dropped because it is already high.
    Section 3.4: only Information-Think and Think-Answer rewards are used in training because Think-Search is already high; this post-hoc selection shapes the headline claim.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/3Z5HKBSH

@misc{pith2026251013272,
  author       = {Pith},
  title        = {Pith review of: Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Z5HKBSH}},
  note         = {Machine review of arXiv:2510.13272}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent work has begun training LLMs to dynamically plan, query, and reason with search engines as tools -- a paradigm increasingly referred to as agentic search. Although these methods achieve performance improvement across popular short-form QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for agentic search, covering three distinct faithfulness metrics: Think-Search faithfulness, Information-Think faithfulness, and Think-Answer faithfulness. Our evaluations reveal that canonical agentic search systems trained through Reinforcement Learning from Verifiable Reward (RLVR) using episode-level outcome-based reward -- including Search-R1 and ReSearch -- have significant room for improvement on these faithfulness dimensions. To foster faithful reasoning in agentic search, we introduce VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained turn-level faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with \ours not only significantly improve reasoning faithfulness, but also achieve better task performance compared to baselines trained against episode-level outcome-based reward.

Figures

Figures reproduced from arXiv: 2510.13272 by Aosong Feng, Haibo Ding, Kang Zhou, Kiran Ramnath, Lin Lee Cheong, Sangmin Woo, Weikang Qiu, Xuan Qi, Yijun Tian, Yun Zhou, Zhichao Xu, Zongyu Wu.

Figure 1
Figure 1. Figure 1: Evaluation of Search-R1. ∗ and † denotes LLMaaJ and trained reward model, respectively. We can see models trained with outcome-based reward, their good task performance does not automatically guarantee fully faithful reasoning chains, which may be suboptimal for practical deployment of such systems. traceability: Information-Think faithfulness (do the agent’s thoughts consider the retrieved infor￾mation?),… view at source ↗
Figure 2
Figure 2. Figure 2: The pipeline of VERITAS-R1 in the RL framework with PPO RL algorithm. Applying process supervision (VERITAS Reward Model) improves the faithfulness of policy model’s intermediate reasoning steps. Outcome in this figure refers to the rollout trajectory used for subsequent reward calculation and advantage estimation. we define their corresponding rewards as Rinfo-think and Rthink-answer. The final reward fun… view at source ↗
Figure 3
Figure 3. Figure 3: Faithfulness evaluation results comparing two variants of Search-R1 and our methods. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Hyperparameter sensitivity study. models optimized for final answers often produce untrustworthy intermediate steps. We introduced a formal evaluation framework with three faithful￾ness metrics and proposed VERITAS, a training paradigm that integrates these metrics as process￾based rewards. Our central finding is that reward￾ing the reasoning process does not trade off with performance; instead, it creates… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 33 linked inside Pith

  1. [1]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 225--237. Association for Computational Linguistics

    2024

  2. [2]

    Anthropic . 2025. Claude 3.7 sonnet and claude code

    2025

  3. [3]

    Iv \'a n Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679

    Pith/arXiv arXiv 2025

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG : Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    Pith/arXiv arXiv 2023

  5. [5]

    Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness tests for natural language explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 283--294

    2023

  6. [6]

    Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926

    Pith/arXiv arXiv 2025

  7. [7]

    Sriram Balasubramanian, Samyadeep Basu, and Soheil Feizi. 2025. A closer look at bias and chain-of-thought faithfulness of large (vision) language models. arXiv preprint arXiv:2505.23945

    arXiv 2025

  8. [8]

    Guangsheng Bao, Hongbo Zhang, Cunxiang Wang, Linyi Yang, and Yue Zhang. 2025. How likely do LLM s with C o T mimic human reasoning? In Proceedings of the 31st International Conference on Computational Linguistics, pages 7831--7850, Abu Dhabi, UAE. Association for Computational Linguistics

    2025

  9. [9]

    Oliver Bentham, Nathan Stringham, and Ana Marasovic. 2024. Chain-of-thought unfaithfulness as disguised accuracy. Transactions on Machine Learning Research. Reproducibility Certification

    2024

  10. [10]

    Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, and 1 others. 2024. Context-dpo: Aligning language models for context-faithfulness. arXiv preprint arXiv:2412.15280

    Pith/arXiv arXiv 2024

  11. [11]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  12. [12]

    Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. RQ - RAG : Learning to refine queries for retrieval augmented generation. In First Conference on Language Modeling

    2024

  13. [13]

    Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, and Na Zou. 2025. MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2607...

    2025

  14. [14]

    Sherman Chann. 2023. Non-determinism in gpt-4 is caused by sparse moe. Accessed on August, 5:2023

    2023

  15. [15]

    Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, and 1 others. 2025. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470

    Pith/arXiv arXiv 2025

  16. [16]

    Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. 2023. Do models explain themselves? counterfactual simulatability of natural language explanations. arXiv preprint arXiv:2307.08678

    Pith/arXiv arXiv 2023

  17. [17]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2024

  18. [18]

    James Chua and Owain Evans. 2025. Are deepseek r1 and other reasoning models more faithful? arXiv preprint arXiv:2501.08156

    Pith/arXiv arXiv 2025

  19. [19]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  20. [20]

    Ponti, and Siva Reddy

    Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022 a . F aith D ial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473--1490

    2022

  21. [21]

    Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022 b . On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5271--5285. Association for Computation...

    2022

  22. [22]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491--6501

    2024

  23. [23]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    Pith/arXiv arXiv 2023

  24. [24]

    Gemini, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    Pith/arXiv arXiv 2024

  25. [25]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346--361

    2021

  26. [26]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  27. [27]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  28. [28]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  29. [29]

    Horace He and Thinking Machines Lab. 2025. Defeating nondeterminism in llm inference. Thinking Machines Lab: Connectionism. Https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

    2025

  30. [30]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625, Barcelona, Spain (Online). International Committee on Computational Linguistics

    2020

  31. [31]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst

    2025

  32. [32]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    Pith/arXiv arXiv 2024

  33. [33]

    Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Xin Zhao, Yang Song, and Tao Zhang. 2025. RAG -star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ...

    2025

  34. [34]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969--7992

    2023

  35. [35]

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. 2025 a . An empirical study on reinforcement learning for reasoning-search interleaved llm agents. arXiv preprint arXiv:2505.15117

    Pith/arXiv arXiv 2025

  36. [36]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025 b . Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    Pith/arXiv arXiv 2025

  37. [37]

    Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025 c . Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, page 737–740, New York, NY, USA. Association for Computing Machinery

    2025

  38. [38]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    Pith/arXiv arXiv 2017

  39. [39]

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237--285

    1996

  40. [40]

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke H \"u llermeier. 2025. A survey of reinforcement learning from human feedback. Transactions on Machine Learning Research. Survey Certification

    2025

  41. [41]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  42. [42]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. T...

    2019

  43. [43]

    Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang, Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, and 1 others. 2025. Bridging offline and online reinforcement learning for llms. arXiv preprint arXiv:2506.21495

    Pith/arXiv arXiv 2025

  44. [44]

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, and 11 others. 2023. Measuring faithfulness in chain-of-tho...

    Pith/arXiv arXiv 2023

  45. [45]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, pages 9459--9474. Curran Associates, Inc

    2020

  46. [46]

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. H alu E val: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449--6464. Association for Computational Linguistics

    2023

  47. [47]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366

    Pith/arXiv arXiv 2025

  48. [48]

    Jintao Liang, Gang Su, Huifeng Lin, You Wu, Rui Zhao, and Ziyue Li. 2025. Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval-augmented generation for industry challenges. arXiv preprint arXiv:2506.10408

    Pith/arXiv arXiv 2025

  49. [49]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. In The Twelfth International Conference on Learning Representations

    2023

  50. [50]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303--5315, Singapore. Association for Computational Linguistics

    2023

  51. [51]

    Chaitanya Malaviya, Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, and Kyle Lo. 2025. Contextualized evaluations: Judging language model responses to underspecified queries. Transactions of the Association for Computational Linguistics, 13:878--900

    2025

  52. [52]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802--9822, Toronto, Canada....

    2023

  53. [53]

    Gary Marchionini. 2006. Exploratory search: from finding to understanding. Communications of the ACM, 49(4):41--46

    2006

  54. [54]

    Katie Matton, Robert Ness, John Guttag, and Emre Kiciman. 2025. Walk the talk? measuring the faithfulness of large language model explanations. In The Thirteenth International Conference on Learning Representations

    2025

  55. [55]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics

    2020

  56. [56]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  57. [57]

    Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. Faitheval: Can your language model stay faithful to context, even if ''the moon is made of marshmallows''. In The Thirteenth International Conference on Learning Representations

    2025

  58. [58]

    Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Dyknow: dynamically verifying time-sensitive factual knowledge in llms. arXiv preprint arXiv:2404.08700

    Pith/arXiv arXiv 2024

  59. [59]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730--27744

    2022

  60. [60]

    Letitia Parcalabescu and Anette Frank. 2024. On measuring faithfulness or self-consistency of natural language explanations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6048--6089

    2024

  61. [61]

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15012--15032

    2024

  62. [62]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687--5711, Singapore. Association for Computational Linguistics

    2023

  63. [63]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728--53741

    2023

  64. [64]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  65. [65]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438

    Pith/arXiv arXiv 2015

  66. [66]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  67. [67]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  68. [68]

    Yaorui Shi, Sihang Li, Chang Wu, Zhiyuan Liu, Junfeng Fang, Hengxing Cai, An Zhang, and Xiang Wang. 2025. Search and refine during think: Autonomous retrieval-augmented reasoning of llms. arXiv preprint arXiv:2505.11277

    arXiv 2025

  69. [69]

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic retrieval-augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136

    Pith/arXiv arXiv 2025

  70. [70]

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592

    Pith/arXiv arXiv 2025

  71. [71]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008--3021

    2020

  72. [72]

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. 2025. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588

    Pith/arXiv arXiv 2025

  73. [73]

    Richard S Sutton, Andrew G Barto, and 1 others. 1998. Reinforcement learning: An introduction. MIT press Cambridge

    1998

  74. [74]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539--554

    2022

  75. [75]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014--10037, Toronto, Canada. Association for Compu...

    2023

  76. [76]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952--74965

    2023

  77. [77]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    Pith/arXiv arXiv 2022

  78. [78]

    Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. 2025. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization. arXiv preprint arXiv:2505.15107

    Pith/arXiv arXiv 2025

  79. [79]

    Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning, 8(3):279--292

    1992

  80. [80]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.