Pith. sign in

REVIEW 3 major objections 3 minor 33 references

Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper tries to establish that jointly training one LLM to rewrite both user queries and corpus documents—with rewards computed from the entangled pair—improves retrieval by 5–7 NDCG@10 points in sparse settings, and that document…

desk verdict Genuinely new bidirectional RL idea for query+document augmentation, but the key ablation gap is a single-seed 1.5-point NDCG difference with no error bars, so the central claim is not yet established. read the letter →

arxiv 2506.18670 v1 pith:HFLO4U5R submitted 2025-06-23 cs.IR

Jingming Liu , Yumeng Li , Wei Shi , Yao-Xiang Ding , Hui Su , Kun Zhou This is my paper
classification cs.IR
keywords informationretrievalqueryrewritingdocumentaugmentationbidirectionalreinforcementlearningrewardsamplingNDCGsparsedense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the bottleneck in LLM-based retrieval is not only the query but also the corpus, and tries to establish that jointly training one LLM to rewrite both queries and documents—with rewards computed from the entangled pair—yields retrieval gains that no query-only, document-only, or separately trained policy achieves. A sympathetic reader would care because it suggests a way to improve hard, compact-domain retrieval without changing the retriever, and because it claims that the gain depends on a specific bidirectional RL procedure rather than on document rewriting per se. The paper reports 5–7 point improvements in NDCG@10 (a ranked-retrieval quality metric) on three BEIR datasets in sparse retrieval, plus generalization to unseen domains. It also reports that standard RL normalizations, GRPO's group normalization and REINFORCE++'s batch normalization, fail in this setting, and proposes a centralization-only advantage with differential scaling instead.

What carries the argument

The engine of the method is a bidirectional RL loop built on three components: query–document composite sampling, which packs a batch with queries, relevant documents, and irrelevant documents so that rewards can be computed inside the batch without scanning the whole corpus; within-batch reward sampling, which estimates the entangled query–document reward by randomly drawing one document rollout per iteration and averaging the resulting NDCG scores over $N$ iterations; and a centralization-only advantage computation with differential scaling for queries, relevant documents, and irrelevant documents, which replaces the group normalization of GRPO and batch normalization of REINFORCE++ to keep sampling noise from dominating the policy gradient.

What would settle it

Run the same bidirectional training on NFCorpus with the reward-sampling count $N$ set to 1 versus a larger $N$, and compare NDCG@10 and training stability; if the two settings converge to the same performance, the claim that sampling accuracy matters at the 1e-2 level is false. A second check is to compute the exact reward exhaustively for a small batch and measure the actual estimation error against the claimed <1e-2.

Watch

Extended reading notes

Core claim

The paper's central claim is that letting the LLM modify documents is only useful when the query-rewriting and document-rewriting policies are learned together in one reward-entangled RL loop: the reward for a query rollout depends on which document rollouts are simultaneously drawn, and vice versa. Because the full combinatorial reward over all query–document rollout pairs is intractable, the paper introduces within-batch reward sampling that randomly selects one document rollout per sampling iteration and averages NDCG rewards across iterations, then uses a centralization-only advantage computation to prevent sampling noise from being amplified by normalization. Trained this way, the shared policy learns to pull augmented queries and documents into a common word or semantic space, as shown by lower cross-entropy between the two augmented distributions, and improves NDCG@10 in both BM25 and dense BGE retrieval settings. The paper claims this collaborative training, not document augmentation itself, produces the observed 5–7 point sparse-retrieval gains.

Load-bearing premise

The method depends on the sampled reward used to train both policies being accurate enough to guide learning; the paper asserts the sampling error is below 1e-2 and the variance can be cut to near 1e-5, but provides no derivation, measurement, or value of the sampling count $N$ to support that.

Editorial extensions

If this is right

  • Because document augmentations can be precomputed once per corpus, the trained policy adds no per-query retrieval cost beyond standard query rewriting.
  • In sparse retrieval, the trained policy improves NDCG@10 by 5–7 points over the base retriever on all three in-domain datasets, and generalizes to unseen domains.
  • The ablation shows that neither document augmentation alone nor query/document augmentation trained separately reproduces the gain, so the benefit is attributable to the joint bidirectional training procedure.
  • The centralization-only advantage computation, without group or batch normalization, is what keeps the entangled rewards usable for policy learning; standard GRPO/REINFORCE++ normalization either amplifies sampling variance or biases advantages toward easy queries.
  • The method works with both sparse (BM25) and dense (BGE) retrievers, so it is orthogonal to the retrieval architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reward-entangled two-sided training is a small instance of credit assignment in shared-policy multi-agent systems, so the sampling-based reward estimator may transfer to other paired-generation tasks (translation, summarization, or evaluation) where a downstream scorer couples two outputs.
  • Editorial inference: the paper's cross-entropy analysis suggests a direct alignment objective—training the policy to minimize the divergence between augmented query and document distributions—could be tested as a cheaper, RL-free surrogate for the entangled reward.
  • Editorial inference: because training samples only a small mini-collection per batch, scaling to very large corpora will require approximate retrieval or per-step document subsampling; a testable extension is to train with documents resampled from a fast index each gradient step.
  • Editorial inference: the claim that document-only RL 'yields little benefit' is established on three BEIR datasets with a 7B model; a stronger test would sweep corpus size, domain difficulty, and model scale to map where the bidirectional advantage actually appears.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes CoAugRetriever, an LLM-based information retriever that augments both queries and documents, trained jointly via reinforcement learning. The method introduces query-document composite sampling, within-batch reward sampling to handle the entangled rewards, and a centralization-only advantage computation that deviates from GRPO and REINFORCE++. Evaluation on BEIR datasets reports gains over query-only and document-only RL training in both sparse (BM25) and dense (BGE) settings, plus cross-benchmark transfer experiments. The central claim is that the benefit comes specifically from joint bidirectional RL, not from document augmentation or query RL alone.

Significance. If substantiated, the idea of jointly learning query and document augmentation policies is a meaningful contribution to LLM-based retrieval, and the method's orthogonality to the underlying retriever makes it practically appealing. The paper provides a thoughtful analysis of why standard RL advantage normalizations fail in this setting, and the code is released, which supports reproducibility. However, the load-bearing comparisons that motivate the method are reported as single runs without uncertainty quantification, and the reward-sampling estimator that underpins the training is asserted rather than verified. The current evidence is suggestive but not conclusive enough for the paper's strong causal claims.

major comments (3)
  1. [§4.3, Table 2] The central claim that collaborative bidirectional training is essential rests on a single-run NDCG@10 difference of 0.015 in the sparse setting (RL-QD 0.403 vs. RL-Q+RL-D 0.388) and 0.012 in the dense setting (0.384 vs. 0.372). No seeds, standard deviations, or significance tests are reported, so the term 'significant performance gap' in §4.3 is not supported. Given the typical seed-to-seed variability of BEIR NDCG@10, which is often on the order of 0.01–0.02, this gap could easily be noise. The same issue affects the normalization ablation in Table 4, where the decisive comparison between 'Ours' and 'w/o adv scale' is also a single run. Please report multiple seeds with confidence intervals or significance tests for at least the central ablations, or explicitly temper the claim.
  2. [§3.3 and §3.4] The reward-sampling estimator is load-bearing for the entire training method, but its claimed accuracy is not established. Section 3.3 asserts that averaging over N sampling iterations 'accurately estimating the rewards for each query and document rollout (error < 1e-2)', and Section 3.4 asserts that the variance 'can easily reduce this variance to the level of 1e-5', yet no derivation, empirical measurement, or value of N is provided. If the estimator is biased or higher-variance than claimed, the policy gradients will be noisy and the joint training may fail for reasons unrelated to the method's conceptual contribution. Please specify N, provide a derivation or a direct empirical comparison of the sampled reward against the exact reward, and report how the ablation results vary with N.
  3. [Abstract and §4.2] The abstract's claim of 'strong cross-benchmark generalization' is contradicted by the dense retrieval results on FiQA-2018: Ours-NF (0.371) and Ours-SF (0.376) both underperform Qwen2.5-7B (0.383), as the text itself acknowledges in §4.2. The generalization behavior is therefore uneven and particularly weak in the dense setting. Please make the generalization claim more precise, separating sparse and dense retrieval, and avoid overstating the result in the abstract.
minor comments (3)
  1. [§4.1] The phrase 'improvements of 5%–7% in the sparse retrieval scenario' is ambiguous: it should be stated as absolute NDCG@10 point gains rather than percentages, since NDCG@10 is already a ratio-scale metric and the relative improvements are much larger.
  2. [Appendix A.3, Table 7] The definition of the 'Amplified Variance' statistic is not fully clear: the text says it subtracts the percentage of low-variance groups after normalization from the original percentage, but the table does not explain which direction corresponds to a problem. Please state explicitly whether a positive value indicates harmful amplification and how the threshold was chosen.
  3. [Throughout] Several hyperparameter choices are reported in Appendix B.1 but without sensitivity analysis, especially the advantage scale coefficients (1.0, 0.2, 0.1). A brief discussion of how sensitive the results are to these values would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL reward is a training target, while all claimed gains are measured on held-out BEIR benchmarks.

full rationale

Auditing the paper's derivation chain, I find no step in which an output is defined in terms of the result it is supposed to establish, no fitted parameter relabeled as a prediction, and no load-bearing self-citation. The learned augmentation policies are optimized against NDCG rewards computed from query and document rollouts; this reward is a training objective, not a held-out result. The reported gains in Tables 1, 2, 4-6 are measured on BEIR test collections using independent retrievers (BM25 and BGE), and cross-benchmark numbers are on datasets not used for training. The central claim that document augmentation is only useful under joint bidirectional RL is an empirical ablation from Table 2 rather than a consequence of the method's definitions; whether the 0.015 NDCG@10 gap is statistically reliable is a legitimate experimental-rigor concern but not a circularity one. The 'error < 1e-2' and variance '1e-5' statements in Sections 3.3 and 3.4 are unsubstantiated estimator claims, but they are not circular reductions. The reference to 'our concurrent work [Jiang et al., 2025]' is not a same-author citation and plays no load-bearing role. The advantage-scale coefficients are hand-tuned hyperparameters, not predictions, and the method is evaluated against external baselines rather than against its own fitting targets. Hence no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several unproved assumptions about reward estimation and advantage normalization, plus a small set of hand-set coefficients. No new physical or conceptual entities are introduced.

free parameters (3)
  • Advantage scale coefficients for query, relevant document, and irrelevant document = 1.0, 0.2, 0.1
    Chosen by hand in Section B.1 to balance gradient contributions; no sensitivity analysis is reported.
  • Number of reward sampling iterations N = Not reported in the paper.
    Section 3.3 describes reward sampling repeated N times, but the paper does not state N or show how the error bound depends on it.
  • Reward sampling error and variance bounds = error < 1e-2, variance < 1e-5
    Asserted in Sections 3.3 and 3.4 without derivation or empirical measurement, yet training quality depends on these values.
assumptions (5)
  • standard math Monte Carlo reward sampling converges to the true expected reward and has bounded error.
    Section 3.3 relies on sampling one rollout per document and averaging across iterations; no convergence proof or error analysis is given.
  • domain assumption NDCG@10 is an appropriate reward signal for retrieval policy learning.
    NDCG@10 is used throughout for reward computation and evaluation; the paper assumes it aligns with retrieval quality.
  • domain assumption A fixed non-differentiable retriever can serve as the RL environment.
    The policy gradient treats BM25 and BGE as black-box scoring functions; this is standard in the line of work but still an assumption about reward signal quality.
  • ad hoc to paper Group-wise centralization without normalization preserves meaningful advantages.
    Justified by ablations in Table 4 rather than a theoretical argument; this is central to the claimed success.
  • ad hoc to paper Cross-entropy H(Q,D) between augmented query and document distributions measures semantic alignment.
    Table 3 uses cross-entropy as evidence of alignment, but the connection between lower cross-entropy and better retrieval is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation." pith.science (2026). https://pith.science/paper/HFLO4U5R

@misc{pith2026250618670,
  author       = {Pith},
  title        = {Pith review of: Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFLO4U5R}},
  note         = {Machine review of arXiv:2506.18670}
}
read the original abstract

Recent studies have proposed leveraging Large Language Models (LLMs) as information retrievers through query rewriting. However, for challenging corpora, we argue that enhancing queries alone is insufficient for robust semantic matching; the LLM should also have sufficient understanding of the corpus by directly handling and augmenting the documents themselves. To this end, we present an LLM-based retriever empowered to augment both user queries and corpus documents, with its policy fully explored via reinforcement learning (RL) and minimal human inductive bias. Notably, we find that simply allowing the LLM to modify documents yields little benefit unless paired with our carefully designed bidirectional RL framework, which enables the LLM to simultaneously learn and collaborate on both query and document augmentation policies. A key technical challenge in realizing such a framework lies in jointly updating both policies during training, where the rewards for the two directions depend on each other, making their entangled reward intractable. Our approach addresses this by introducing a reward sampling strategy and a specifically designed RL algorithm that enables effective training with these sampled rewards. Experimental results demonstrate that our approach significantly enhances LLM-based retrieval performance in both sparse and dense settings, particularly in difficult retrieval domains, and achieves strong cross-benchmark generalization. Our code is released at https://github.com/liujm2001/CoAugRetriever.

Figures

Figures reproduced from arXiv: 2506.18670 by the authors.

Figure 1
Figure 1. Alongside ordinary test-time query rewriting, we allow the LLM to pre-compute lightweight [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the bidirectional RL training process. Left: Overview of the training pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the problems identified in our mission when adopting state-of-the-art [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Implementation of our RL training pipeline. The entanglement is introduced by our [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative demonstration of RL behaviors that boost performance. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 9 canonical work pages

  1. [1]

    Umass at trec 2004: Novelty and hard

    Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. Umass at trec 2004: Novelty and hard. InProceedings of TREC-13, pages 715–725,

    work page 2004

  2. [4]

    The snowflake elastic data warehouse

    Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, et al. The snowflake elastic data warehouse. InProceedings of the 2016 International Conference on Management of Data, pages 215–226,

    work page 2016

  3. [12]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November

    work page 2020

  4. [13]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

    arXiv preprint arXiv:2212.14024. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,

    arXiv

  5. [14]

    Internet- augmented language models through few-shot prompting for open-domain question answering

    Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115,

    arXiv

  6. [16]

    Query rewriting in retrieval- augmented large language models

    11 Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315,

    work page 2023

  7. [17]

    Rafe: ranking feedback improves query rewriting for rag.arXiv preprint arXiv:2405.14431,

    Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Rafe: ranking feedback improves query rewriting for rag.arXiv preprint arXiv:2405.14431,

    arXiv

  8. [19]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis

    Accessed: 2025-01-24. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models.arXiv preprint arXiv:2210.03350,

    arXiv 2025

Show all 33 references
  1. [22]

    Large language models are strong zero-shot retriever.arXiv preprint arXiv:2304.14233,

    12 Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou, and Daxin Jiang. Large language models are strong zero-shot retriever.arXiv preprint arXiv:2304.14233,

    arXiv

  2. [23]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    arXiv

  3. [24]

    Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652,

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652,

    arXiv

  4. [28]

    Query2doc: Query expansion with large language models

    Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678,

    arXiv

  5. [29]

    Multilin- gual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672,

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilin- gual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672,

    arXiv

  6. [30]

    Query expansion with freebase.Proceedings of the 2015 International Conference on The Theory of Information Retrieval,

    Chenyan Xiong and Jamie Callan. Query expansion with freebase.Proceedings of the 2015 International Conference on The Theory of Information Retrieval,

    2015

  7. [31]

    Enhancing conversational search: Large language model-aided informative query rewriting.arXiv preprint arXiv:2310.09716,

    Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. Enhancing conversational search: Large language model-aided informative query rewriting.arXiv preprint arXiv:2310.09716,

    arXiv

  8. [33]

    The best results are bolded, and other top-three results are underlined

    14 Table 5: NDCG@10 Performance of models trained from Qwen2.5-7B on additional test-only datasets. The best results are bolded, and other top-three results are underlined. Settings SCIDOCS TREC-COVID Base Retriever BM25 0.165 0.688 BGE-base-en-v1.5 0.214 0.672 Qwen2.5-7B BM25...

    2018

  9. [34]

    The selection of the threshold does not affect our conclusion, as it clearly shows that many rewards have very small variances, which are greatly amplified. We believe the abundance of small-variance groups is due to the fact that augmentation does not necessarily alter the si...

    2025

  10. [1982]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  11. [1988]

    Proximal policy optimization algorithms.ArXiv, abs/1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.ArXiv, abs/1707.06347,

    arXiv

  12. [2001]

    Ziegler, Ryan J

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback.ArXiv, abs/2009.01325,

    2009 arXiv

  13. [2008]

    Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K

    European Language Resources Association (ELRA). Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, and Lili Qiu. Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your llms use external data more wisely.ArXiv, abs/2409.14924,

    arXiv

  14. [2009]

    Hamilton, Chris Dyer, and Dani Yogatama

    Devendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer, and Dani Yogatama. End- to-end training of multi-document reader and retriever for open-domain question answering. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman...

    2021

  15. [2010]

    Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663,

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663,

    arXiv

  16. [2014]

    The faiss library.arXiv preprint arXiv:2401.08281,

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library.arXiv preprint arXiv:2401.08281,

    arXiv

  17. [2017]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.ArXiv, abs/1706.03741,

    arXiv

  18. [2018]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...

    arXiv

  19. [2019]

    You only need one model for open-domain question answering

    Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher Manning, and Kyoung-Gu Woo. You only need one model for open-domain question answering. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3047–3060, Abu Dhabi, Uni...

    2022

  20. [2020]

    Grounding by trying: Llms with reinforcement learning-enhanced retrieval.arXiv preprint arXiv:2410.23214,

    Sheryl Hsu, Omar Khattab, Chelsea Finn, and Archit Sharma. Grounding by trying: Llms with reinforcement learning-enhanced retrieval.arXiv preprint arXiv:2410.23214,

    arXiv

  21. [2021]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.ArXiv, abs/2307.09288,

    arXiv

  22. [2022]

    Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXiv preprint arXiv:2503.00223,

    Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXiv preprint arXiv:2503.00223,

    arXiv

  23. [2023]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    10 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  24. [2024]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2:1,

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2:1,

    arXiv

  25. [2025]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.