REVIEW 3 major objections 6 minor 1 cited by
Multi-Turn On-Policy Distillation with Prefix Replay
T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Replacing live agent rollouts with replayed teacher prefixes — weighted toward early, reliable steps — makes multi-turn on-policy distillation offline, at least 4× faster per rollout, and just as accurate.
desk verdict Practical offline distillation recipe with a suggestive but under-specified theoretical justification; worth engaging for the method and experiments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the "effective prefix distribution" ρ_t = w_t P_t, the distribution over interaction histories at step t on which training mass is placed. A bound decomposes the gap to ideal interactive OPD into a student-occupancy term and a teacher-reliability term; the paper's bridge objective minimizes a weighted combination of KL divergences to the student and teacher occupancies, whose exact solution is a geometric bridge. Because the exact importance weight is a high-variance prefix-level likelihood ratio that empirically decays with depth, ReOPD replaces it with the one-parameter step-decay schedule ω(t)=κ^t, applied by sampling positions proportionally. This schedule is t
What would settle it
On a fixed teacher pool, compute the exact per-position weights br_t = ∏_{s<t} π_θold(a_s)/π_T(a_s) and regress log(br_t) against step index t; if the log-ratio is not approximately linear in t, or if depth explains only a small fraction of its variance, the step-decay proxy ω(t)=κ^t is not a faithful replacement for the density-ratio weight. Alternatively, run ReOPD against OPD on a task where the teacher is unreliable on student histories but the teacher–student gap is not monotone in depth; if ReOPD loses, the reliability-aware explanation is wrong.
Extended reading notes
Core claim
The central discovery is that multi-turn OPD should be treated as reliability-aware prefix distribution design rather than "maximally on-policy" distillation. The paper proves a bound on the gap between a replayed-prefix objective and the ideal interactive objective: the gap is at most a student-occupancy mismatch term plus a teacher-reliability term, so pushing prefixes fully onto the student is not automatically optimal. ReOPD implements the resulting balance by replaying teacher prefixes verbatim, letting the student generate only the action at the evaluated step, and weighting positions by ω(t)=κ^t so early, low-shift prefixes carry more training mass. Empirically, the method matches OPD
Load-bearing premise
The position-only step-decay schedule ω(t)=κ^t is a valid replacement for the exact likelihood-ratio weight, which assumes the average per-step teacher–student gap is roughly constant along the trajectory, while κ=0.6 is set empirically rather than derived from that gap; if depth does not actually track the reliability ratio, the method's advantage over online OPD is not predicted by the theory.
Editorial extensions
If this is right
- ReOPD preserves or improves OPD-level accuracy across teacher and student scales in mathematical-reasoning-with-Python and search environments.
- ReOPD uses zero tool calls during student training and is at least 4× faster per rollout than OPD; with teacher prefix collection cost included, it remains over 2× faster.
- A single student can be distilled jointly across heterogeneous environments from one merged offline pool, without deploying all environments during student training.
- The teacher's own RL rollouts can be reused as the prefix pool at no extra collection cost; a stationary pool drawn from the final teacher gives nearly identical students.
- Fully student-on-policy training is not uniformly optimal: when the teacher is unreliable on student-induced histories, teacher-anchored prefixes improve distillation.
- At a fixed distillation target, prefixes from the teacher itself outperform prefixes from larger or stronger generators, showing that teacher reliability, not raw capability, governs prefix quality.
Reading between the lines
- Beyond the paper: the bridge-to-schedule map suggests κ could be set a priori as exp(−γ_t c̄) once the per-step teacher–student gap c̄ is measured on the pool; the paper fixes κ=0.6 empirically, so a gap-adaptive schedule is a direct testable extension.
- Beyond the paper: the step-decay surrogate uses only depth as a proxy for reliability, but a data-dependent weight derived from the student's own sampling probabilities at the evaluated step could capture per-history variation; the paper mentions this as future work.
- Beyond the paper: the two-regime result implies a natural curriculum — start with a steep decay when the student is far below the teacher and flatten it toward uniform as the gap closes, which would make the schedule adapt to training progress.
- Beyond the paper: if the prefix pool is free whenever the teacher is RL-trained, multi-teacher distillation to a single generalist student becomes a logistics problem (merging offline pools) rather than a compute-cost problem, which could materially change how agent teams are trained.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Replayed-Prefix On-Policy Distillation (ReOPD), an offline distillation method for multi-turn agentic tasks. Instead of rolling the student through the environment and querying the teacher on student-sampled histories (OPD), ReOPD reuses a fixed pool of teacher trajectories: it replays a teacher-forced prefix, lets the student act at the supervised step, and uses the teacher conditional as the target. The authors identify a 'prefix trap' with a two-sided shift (student occupancy vs. teacher reliability) and derive a bound (Prop. 1) decomposing the gap to an ideal interactive objective. They propose a geometric bridge between student and teacher occupancies and approximate the resulting weight by a step-decay schedule ω(t)=κ^t, which reallocates mass to early, low-shift steps. Experiments in math-with-Python and search across multiple teacher/student scales show ReOPD matches or improves OPD accuracy while being at least 4× faster per rollout and using zero tool calls during student training.
Significance. The empirical contribution is practically significant: ReOPD turns expensive online interaction into a reusable offline pool, and the experimental design is thorough (two environments, multiple teacher/student scales, ablations, multi-environment joint training). Proposition 1 is correct, and the two-sided decomposition is a useful conceptual framing. However, the connection between the geometric bridge and the implemented step-decay schedule is asserted rather than established: κ is fixed empirically, and Figure 4 does not provide the supporting evidence claimed. The method itself remains useful even if the theoretical narrative is weakened; the paper's operational speedups and the observation that teacher-anchored prefixes help in a large-gap regime are valuable independent of the derivation.
major comments (3)
- [§4, Eq. (8) and §5, implementation details] The paper claims κ is pinned by the bridge-to-schedule map κ=exp(−γ_t c̄) in Eq. (8), but the implementation section states "steepness is set empirically" and uses κ=0.6 without reporting γ_t or c̄. The map is therefore underdetermined: γ_t is never specified as a function of t, and c̄ (the average per-step teacher–student gap) is not measured in either environment. As written, the schedule is a tuned heuristic, which conflicts with the claim that ReOPD "realizes" the reliability-aware prefix distribution. To restore the link, the authors should report c̄ on the math and search pools and show whether κ=0.6 matches exp(−γ_t c̄) for a plausible γ_t, or reframe the contribution as an empirical simplification with the exact weight (7) as the principled target and the step-decay as a separately validated approximation.
- [Figure 4 and surrounding text] Figure 4 is the only empirical evidence for "step index is a strong proxy for the likelihood-ratio weight." As plotted, the log-ratio values are all in [0.36, 0.41]—a span of about 0.05—whereas the implemented κ=0.6^t decay corresponds to a per-step log decrease of log(1/0.6)≈0.51. The trend in the figure is far too shallow to justify the steep schedule used in the algorithm. Additionally, the text states that the student-to-teacher ratio is typically below one (so its log should be negative), but the y-axis is positive; this inconsistency suggests the figure is plotting an absolute value or inverse ratio, which should be stated. The figure, as presented, does not support the claim that depth explains most of the variation, and it does not justify the specific steepness of κ=0.6.
- [§4, "From the bridge to a step-decaying weight"] The bridge objective in Eq. (5) defines a per-history effective distribution ρ⋆_t, and the exact weight (7) varies within each step. The implemented surrogate ω(t)=κ^t is constant within a step and is normalized across positions, so it does not solve the optimization (5). The paper asserts that both forms "point the same way" because br_t decays with depth, but this is precisely the claim that Figure 4 fails to establish (see previous comment). Moreover, no experiment compares the step-decay surrogate against the exact weight (7) or against a per-history weight derived from the bridge. Without such a comparison, the assertion that ReOPD is a "practical implementation of the resulting effective distribution" remains unsupported. The authors should either provide a direct comparison (e.g., using the exact weight on a subset of steps) or explicitly downgrade the step-decay to a heuristic th
minor comments (6)
- [Title page] The rendered URL "/gl⌢beProject Page: ..." appears to be a formatting artifact and should be cleaned up.
- [§4, Eq. (8)] Since log br_t ≈ −(t−1)c̄, the exponent in Eq. (8) should be κ^{t−1} rather than κ^t; the "up to a constant" clause can absorb this, but the subsequent identification with ω(t)=κ^t should be stated more carefully.
- [§5.2, Teacher's prefix resampling] The text references "Table 8" for the time used in teacher prefix resampling, but only Figure 8 appears in the submission. The cross-reference should be corrected.
- [Table 1] The Search max generation tokens value is listed as "4196"; this is likely a typo (perhaps 4096 or 8192) and should be fixed.
- [Figure 4] The y-axis label should clarify whether the plotted quantity is log br_t, −log br_t, or some other transformation, so that it is consistent with the text's statement that the ratio is typically below one.
- [Table 4 header] The column header "MA TH500" has a spacing typo and should read "MATH500."
Circularity Check
No significant circularity: the two-regime prediction is qualitative and independently tested; the tuned κ is an openly acknowledged implementation knob, not a disguised input.
full rationale
The paper's derivation chain is self-contained rather than circular. Proposition 1 is a genuine decomposition of the objective gap into occupancy mismatch and teacher-reliability terms; the geometric bridge in Eq. (6) is the explicit solution of the KL-weighted objective in Eq. (5); Eq. (7) is a well-defined importance weight; and Eq. (8) is an explicitly stated approximation, not an equality forced by construction. The central two-regime prediction is qualitative—ReOPD should help when the teacher–student gap is large and match OPD when the teacher is reliable on student histories—and it is tested on held-out benchmarks (Tables 4 and 5). The step-decay steepness κ=0.6 is admittedly set empirically in Section 5 rather than computed from the Eq. (8) map, and the limitations section concedes that the surrogate 'captures how deep a prefix is rather than directly measuring how far a given prefix lies from the overlap.' This weakens the quantitative authority of the bridge-to-schedule map and is a legitimate correctness/risk concern, but it does not make any prediction equivalent to its input: the two-regime outcome could have failed, and the fitted knob is openly disclosed rather than renamed as a derived parameter. Self-citations (e.g., Liao et al. 2026, Ye et al. 2026a/b) appear only in related work and are not load-bearing. No circular step is exhibited.
Assumptions & free parameters
free parameters (1)
- κ (step-decay base) =
0.6
assumptions (4)
- domain assumption A1: The loss against the ideal target is uniformly bounded by B (Assumption 1).
- ad hoc to paper Teacher support is a valid surrogate for unobserved teacher reliability.
- ad hoc to paper The per-step teacher–student gap c̄ is approximately stationary, so log br_t ≈ −(t−1)c̄, and step depth is a strong proxy for the likelihood-ratio weight.
- domain assumption The teacher's RL rollouts (a mixed-policy pool spanning checkpoints) are within the final teacher's reliable support.
Cite this review
Pith. "Pith review of Multi-Turn On-Policy Distillation with Prefix Replay." pith.science (2026). https://pith.science/paper/PCUFW5E7
@misc{pith2026260704763,
author = {Pith},
title = {Pith review of: Multi-Turn On-Policy Distillation with Prefix Replay},
year = {2026},
howpublished = {\url{https://pith.science/paper/PCUFW5E7}},
note = {Machine review of arXiv:2607.04763}
}
abstract
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
EvoReason: Self-Evolving Reasoning Primitive-Guided On-Policy Distillation for Latent Reasoning in Generative Recommendation
A self-evolving, primitive-guided distillation method for latent reasoning in generative recommendation reports consistent ranking gains over existing latent-reasoning baselines.
Reference graph
Works this paper leans on
-
[1]
2025 , howpublished =
Zilin Zhu and Chengxing Xie and Xin Lv and slime Contributors , title =. 2025 , howpublished =
2025
-
[2]
arXiv preprint arXiv:2212.03533 , year=
Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year=
-
[3]
Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=
Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=
2020
-
[4]
Proceedings of the 2018 conference on empirical methods in natural language processing , pages=
HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=
2018
-
[5]
Transactions of the Association for Computational Linguistics , volume=
Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=
2019
-
[6]
arXiv preprint arXiv:2103.03874 , year=
Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=
-
[7]
Advances in Neural Information Processing Systems , volume=
Solving quantitative reasoning problems with language models , author=. Advances in Neural Information Processing Systems , volume=
-
[8]
Hugging Face repository , volume=
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions , author=. Hugging Face repository , volume=
Show all 300 references
-
[9]
arXiv preprint arXiv:2504.11536 , year=
Retool: Reinforcement learning for strategic tool use in llms , author=. arXiv preprint arXiv:2504.11536 , year=
-
[10]
arXiv preprint arXiv:2505.09388 , year=
Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=
-
[11]
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=
-
[12]
Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , author=. Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=
-
[13]
Transactions of the Association for Computational Linguistics (TACL) , volume=
MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics (TACL) , volume=
-
[14]
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=
-
[15]
Findings of the Association for Computational Linguistics: EMNLP 2023 , year=
Measuring and Narrowing the Compositionality Gap in Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , year=
2023
-
[16]
Langley , title =
P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =
2000
-
[17]
2025 , eprint=
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=
2025
-
[18]
T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980
1980
-
[19]
M. J. Kearns , title =
-
[20]
Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983
1983
-
[21]
R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000
2000
-
[22]
Suppressed for Anonymity , author=
-
[23]
Newell and P
A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981
1981
-
[24]
arXiv preprint arXiv:2501.07301 , year=
The lessons of developing process reward models in mathematical reasoning , author=. arXiv preprint arXiv:2501.07301 , year=
-
[25]
GitHub repository , howpublished =
Wei Xiong and Hanning Zhang and Nan Jiang and Tong Zhang , title =. GitHub repository , howpublished =. 2024 , publisher =
2024
-
[26]
arXiv preprint arXiv:2412.06559 , year=
Processbench: Identifying process errors in mathematical reasoning , author=. arXiv preprint arXiv:2412.06559 , year=
-
[27]
A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959
1959
-
[28]
Scaling Learning Algorithms Towards
Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards
-
[29]
and Osindero, Simon and Teh, Yee Whye , journal =
Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =
-
[30]
Advances in Neural Information Processing Systems , volume=
Self-refine: Iterative refinement with self-feedback , author=. Advances in Neural Information Processing Systems , volume=
-
[31]
Advances in Neural Information Processing Systems , volume=
Reflexion: Language agents with verbal reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=
-
[32]
Advances in Neural Information Processing Systems , volume=
Language models can solve computer tasks , author=. Advances in Neural Information Processing Systems , volume=
-
[33]
Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
Li, Ming and Chen, Lichang and Chen, Jiuhai and He, Shwai and Gu, Jiuxiang and Zhou, Tianyi. Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning. Findings of the Association for Computational Linguistics ACL 2024. 2024
2024
-
[34]
arXiv preprint arXiv:2211.00053 , year=
Generating sequences by learning to self-correct , author=. arXiv preprint arXiv:2211.00053 , year=
-
[35]
arXiv preprint arXiv:2203.11171 , year=
Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=
-
[36]
arXiv preprint arXiv:2407.18219 , year=
Recursive introspection: Teaching language model agents how to self-improve , author=. arXiv preprint arXiv:2407.18219 , year=
-
[37]
arXiv preprint arXiv:2409.13156 , year=
Rrm: Robust reward model training mitigates reward hacking , author=. arXiv preprint arXiv:2409.13156 , year=
-
[38]
arXiv preprint arXiv:2409.12917 , year=
Training language models to self-correct via reinforcement learning , author=. arXiv preprint arXiv:2409.12917 , year=
-
[39]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[40]
arXiv preprint arXiv:2310.01798 , year=
Large language models cannot self-correct reasoning yet , author=. arXiv preprint arXiv:2310.01798 , year=
-
[41]
arXiv preprint arXiv:2310.03708 , year=
Beyond one-preference-for-all: Multi-objective direct preference optimization , author=. arXiv preprint arXiv:2310.03708 , year=
-
[42]
2016 , publisher=
Deep learning , author=. 2016 , publisher=
2016
-
[43]
arXiv preprint arXiv:2403.19159 , year=
Disentangling length from quality in direct preference optimization , author=. arXiv preprint arXiv:2403.19159 , year=
-
[44]
arXiv preprint arXiv:2406.11817 , year=
Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level , author=. arXiv preprint arXiv:2406.11817 , year=
-
[45]
arXiv preprint arXiv:2403.04642 , year=
Teaching large language models to reason with reinforcement learning , author=. arXiv preprint arXiv:2403.04642 , year=
-
[46]
Hashimoto , title =
Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , month =
2023
-
[47]
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=
Robust fine-tuning of zero-shot models , author=. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=
2022
-
[48]
arXiv preprint arXiv:2012.09816 , year=
Towards understanding ensemble, knowledge distillation and self-distillation in deep learning , author=. arXiv preprint arXiv:2012.09816 , year=
2012 arXiv
-
[49]
arXiv preprint arXiv:1806.03822 , year=
Know what you don't know: Unanswerable questions for SQuAD , author=. arXiv preprint arXiv:1806.03822 , year=
-
[50]
Advances in Neural Information Processing Systems , volume=
Experience replay for continual learning , author=. Advances in Neural Information Processing Systems , volume=
-
[51]
International Conference on Machine Learning , pages=
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[52]
2024 , publisher =
Entropy-Regularized Process Reward Model , author=. 2024 , publisher =
2024
-
[53]
ACM Transactions on Information Systems (TOIS) , volume=
Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2007 , publisher=
2007
-
[54]
doi:10.57967/hf/0513 , publisher =
Edward Beeching and Younes Belkada and Kashif Rasul and Lewis Tunstall and Leandro von Werra and Nazneen Rajani and Nathan Lambert , title =. doi:10.57967/hf/0513 , publisher =
-
[55]
arXiv preprint arXiv:2408.15240 , year=
Generative verifiers: Reward modeling as next-token prediction , author=. arXiv preprint arXiv:2408.15240 , year=
-
[56]
arXiv preprint arXiv:2410.12832 , year=
Generative Reward Models , author=. arXiv preprint arXiv:2410.12832 , year=
-
[57]
arXiv preprint arXiv:2401.12187 , year=
Warm: On the benefits of weight averaged reward models , author=. arXiv preprint arXiv:2401.12187 , year=
-
[58]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Robust fine-tuning of zero-shot models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[59]
arXiv preprint arXiv:2407.13690 , year=
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving , author=. arXiv preprint arXiv:2407.13690 , year=
-
[60]
arXiv preprint arXiv:2407.14622 , year=
BOND: Aligning LLMs with Best-of-N Distillation , author=. arXiv preprint arXiv:2407.14622 , year=
-
[61]
2023 , eprint=
UltraFeedback: Boosting Language Models with High-quality Feedback , author=. 2023 , eprint=
2023
-
[62]
arXiv preprint arXiv:2402.07319 , year=
Odin: Disentangled reward mitigates hacking in rlhf , author=. arXiv preprint arXiv:2402.07319 , year=
-
[63]
arXiv preprint arXiv:2407.21783 , year=
The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=
-
[64]
2023 , publisher=
Mathematical analysis of machine learning algorithms , author=. 2023 , publisher=
2023
-
[65]
2024 , journal =
"Alignment Guidebook , author=. 2024 , journal =
2024
-
[66]
arXiv preprint arXiv:2305.11206 , year=
Lima: Less is more for alignment , author=. arXiv preprint arXiv:2305.11206 , year=
-
[67]
arXiv preprint arXiv:2309.17452 , year=
Tora: A tool-integrated reasoning agent for mathematical problem solving , author=. arXiv preprint arXiv:2309.17452 , year=
-
[68]
arXiv preprint arXiv:2404.02078 , year=
Advancing llm reasoning generalists with preference trees , author=. arXiv preprint arXiv:2404.02078 , year=
-
[69]
arXiv preprint arXiv:2308.09583 , year=
Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct , author=. arXiv preprint arXiv:2308.09583 , year=
-
[70]
arXiv preprint arXiv:2404.19733 , year=
Iterative reasoning preference optimization , author=. arXiv preprint arXiv:2404.19733 , year=
-
[71]
arXiv preprint arXiv:2305.20050 , year=
Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=
-
[72]
arXiv preprint arXiv:2405.03548 , year=
Mammoth2: Scaling instructions from the web , author=. arXiv preprint arXiv:2405.03548 , year=
-
[73]
arXiv preprint arXiv:2404.16767 , year=
REBEL: Reinforcement Learning via Regressing Relative Rewards , author=. arXiv preprint arXiv:2404.16767 , year=
-
[74]
arXiv preprint arXiv:2405.00451 , year=
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning , author=. arXiv preprint arXiv:2405.00451 , year=
-
[75]
Advances in Neural Information Processing Systems , volume=
Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in Neural Information Processing Systems , volume=
-
[76]
arXiv preprint arXiv:2406.10858 , year=
Step-level Value Preference Optimization for Mathematical Reasoning , author=. arXiv preprint arXiv:2406.10858 , year=
-
[77]
nature , volume=
Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=
2016
-
[78]
arXiv preprint arXiv:1312.5602 , year=
Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=
-
[79]
arXiv preprint arXiv:2312.10003 , year=
Rest meets react: Self-improvement for multi-step reasoning llm agent , author=. arXiv preprint arXiv:2312.10003 , year=
-
[80]
arXiv preprint arXiv:2402.10176 , year=
OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset , author=. arXiv preprint arXiv:2402.10176 , year=
-
[81]
arXiv preprint arXiv:2410.01560 , year=
Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data , author=. arXiv preprint arXiv:2410.01560 , year=
-
[82]
arXiv preprint arXiv:2405.21046 , year=
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF , author=. arXiv preprint arXiv:2405.21046 , year=
-
[83]
Connection Science , volume=
Function optimization using connectionist reinforcement learning algorithms , author=. Connection Science , volume=. 1991 , publisher=
1991
-
[84]
2010 , publisher=
Modeling purposeful adaptive behavior with the principle of maximum causal entropy , author=. 2010 , publisher=
2010
-
[85]
arXiv preprint arXiv:2110.14168 , year=
Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=
-
[86]
2023 , eprint=
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework , author=. 2023 , eprint=
2023
-
[87]
arXiv preprint arXiv:2404.12358 , year=
From r to Q*: Your Language Model is Secretly a Q-Function , author=. arXiv preprint arXiv:2404.12358 , year=
-
[88]
arXiv preprint arXiv:2405.16436 , year=
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer , author=. arXiv preprint arXiv:2405.16436 , year=
-
[89]
arXiv preprint arXiv:2402.01878 , year=
LiPO: Listwise Preference Optimization through Learning-to-Rank , author=. arXiv preprint arXiv:2402.01878 , year=
-
[90]
arXiv preprint arXiv:2404.10719 , year=
Is dpo superior to ppo for llm alignment? a comprehensive study , author=. arXiv preprint arXiv:2404.10719 , year=
-
[91]
arXiv preprint arXiv:2308.01825 , year=
Scaling relationship on learning mathematical reasoning with large language models , author=. arXiv preprint arXiv:2308.01825 , year=
-
[92]
CoRR, abs/2312.08935 , year=
Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. CoRR, abs/2312.08935 , year=
-
[93]
arXiv preprint arXiv:2402.00658 , year=
Learning planning-based reasoning by trajectories collection and process reward synthesizing , author=. arXiv preprint arXiv:2402.00658 , year=
-
[94]
European conference on machine learning , pages=
Bandit based monte-carlo planning , author=. European conference on machine learning , pages=. 2006 , organization=
2006
-
[95]
arXiv preprint arXiv:2411.04109 , year=
Self-Consistency Preference Optimization , author=. arXiv preprint arXiv:2411.04109 , year=
-
[96]
2019 , publisher=
High-dimensional statistics: A non-asymptotic viewpoint , author=. 2019 , publisher=
2019
-
[97]
arXiv preprint arXiv:2406.18629 , year=
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs , author=. arXiv preprint arXiv:2406.18629 , year=
-
[98]
Machine learning , volume=
Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=
1992
-
[99]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[100]
arXiv preprint arXiv:2402.14830 , year=
Orca-math: Unlocking the potential of slms in grade school math , author=. arXiv preprint arXiv:2402.14830 , year=
-
[101]
arXiv preprint arXiv:2210.03629 , year=
React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year=
-
[102]
Meta AI Blog , year=
Introducing Meta Llama 3: The most capable openly available LLM to date , author=. Meta AI Blog , year=
-
[103]
, Gonzalez and Ion Stoica , month =
Tianle ,Li* and Wei-Lin, Chiang and Evan, Frick nad Lisa ,Dunlap and Banghua , Zhu and Joseph E. , Gonzalez and Ion Stoica , month =. From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline , url =
-
[104]
Advances in Neural Information Processing Systems , volume=
Implicit generation and modeling with energy based models , author=. Advances in Neural Information Processing Systems , volume=
-
[105]
Notes by GE Forsythe , pages=
Various techniques used in connection with random digits , author=. Notes by GE Forsythe , pages=
-
[106]
2024 , eprint=
WARP: On the Benefits of Weight Averaged Rewarded Policies , author=. 2024 , eprint=
2024
-
[107]
2024 , eprint=
OffsetBias: Leveraging Debiased Data for Tuning Evaluators , author=. 2024 , eprint=
2024
-
[108]
arXiv preprint arXiv:2406.11827 , year=
WPO: Enhancing RLHF with Weighted Preference Optimization , author=. arXiv preprint arXiv:2406.11827 , year=
-
[109]
arXiv preprint arXiv:2409.02392 , year=
Building Math Agents with Multi-Turn Iterative Preference Learning , author=. arXiv preprint arXiv:2409.02392 , year=
-
[110]
Liu , year =
Peter J. Liu , year =
-
[111]
2024 , eprint=
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models , author=. 2024 , eprint=
2024
-
[112]
Geirhos, Robert and Jacobsen, Jörn-Henrik and Michaelis, Claudio and Zemel, Richard and Brendel, Wieland and Bethge, Matthias and Wichmann, Felix A. , year=. Shortcut learning in deep neural networks , volume=. Nature Machine Intelligence , publisher=. doi:10.1038/s42256-020-0...
-
[113]
arXiv preprint arXiv:1912.03263 , year=
Your classifier is secretly an energy based model and you should treat it like one , author=. arXiv preprint arXiv:1912.03263 , year=
1912 arXiv
-
[114]
arXiv preprint arXiv:2405.19107 , year=
Offline Regularised Reinforcement Learning for Large Language Models Alignment , author=. arXiv preprint arXiv:2405.19107 , year=
-
[115]
arXiv preprint arXiv:2306.12420 , year=
Lmflow: An extensible toolkit for finetuning and inference of large foundation models , author=. arXiv preprint arXiv:2306.12420 , year=
-
[116]
arXiv preprint arXiv:2403.07691 , volume=
Orpo: Monolithic preference optimization without reference model , author=. arXiv preprint arXiv:2403.07691 , volume=
-
[117]
5-math technical report: Toward mathematical expert model via self-improvement , author=
Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement , author=. arXiv preprint arXiv:2409.12122 , year=
-
[118]
Hugging Face repository , howpublished =
Edward Beeching and Shengyi Costa Huang and Albert Jiang and Jia Li and Benjamin Lipkin and Zihan Qina and Kashif Rasul and Ziju Shen and Roman Soletskyi and Lewis Tunstall , title =. Hugging Face repository , howpublished =. 2024 , publisher =
2024
-
[119]
arXiv preprint arXiv:2402.14008 , year=
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems , author=. arXiv preprint arXiv:2402.14008 , year=
-
[120]
2024 , journal =
HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =
2024
-
[121]
2025 , howpublished=
7B Model and 8K Examples: Emerging Reasoning with Reinforcement Learning is Both Effective and Efficient , author=. 2025 , howpublished=
2025
-
[122]
2025 , howpublished=
Stop Gamma Decay: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning , author=. 2025 , howpublished=
2025
-
[123]
2025 , howpublished=
Online-DPO-R1: Unlocking Effective Reasoning Without the PPO Overhead , author=. 2025 , howpublished=
2025
-
[124]
2025 , howpublished=
There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study , author=. 2025 , howpublished=
2025
-
[125]
2025 , eprint=
Process Reinforcement through Implicit Rewards , author=. 2025 , eprint=
2025
-
[126]
The method of paired comparisons , author=
Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=
1952
-
[127]
5 technical report , author=
Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=
-
[128]
Proceedings of the 10th conference on pattern languages of programs (plop2003) , pages=
Abstract syntax tree implementation idioms , author=. Proceedings of the 10th conference on pattern languages of programs (plop2003) , pages=
-
[129]
arXiv e-prints , pages=
Improve Mathematical Reasoning in Language Models by Automated Process Supervision , author=. arXiv e-prints , pages=
-
[130]
arXiv preprint arXiv:2402.03300 , year=
Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=
-
[131]
Advances in neural information processing systems , volume=
Generative adversarial nets , author=. Advances in neural information processing systems , volume=
-
[132]
arXiv preprint arXiv:1810.04805 , year=
Bert: Pre-training of deep bidirectional transformers for language understanding , author=. arXiv preprint arXiv:1810.04805 , year=
-
[133]
IEEE transactions on pattern analysis and machine intelligence , volume=
A framework of composite functional gradient methods for generative adversarial models , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2019 , publisher=
2019
-
[134]
arXiv preprint arXiv:2307.09288 , year=
Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=
-
[135]
International Conference on Machine Learning , pages=
Bilinear classes: A structural framework for provable generalization in rl , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[136]
Stochastic linear optimization under bandit feedback , author=
-
[137]
Advances in neural information processing systems , volume=
Improved algorithms for linear stochastic bandits , author=. Advances in neural information processing systems , volume=
-
[138]
ArXiv , year=
GPT-4 Technical Report , author=. ArXiv , year=
-
[139]
Blog , year=
Preference Tuning LLMs with Direct Preference Optimization Methods , author=. Blog , year=
-
[140]
arXiv preprint arXiv:2403.08295 , year=
Gemma: Open models based on gemini research and technology , author=. arXiv preprint arXiv:2403.08295 , year=
-
[141]
arXiv preprint arXiv:2312.09390 , year=
Weak-to-strong generalization: Eliciting strong capabilities with weak supervision , author=. arXiv preprint arXiv:2312.09390 , year=
-
[142]
Understanding Dataset Difficulty with
Ethayarajh, Kawin and Choi, Yejin and Swayamdipta, Swabha , booktitle =. Understanding Dataset Difficulty with. 2022 , editor =
2022
-
[143]
arXiv preprint arXiv:2401.01879 , year=
Theoretical guarantees on the best-of-n alignment policy , author=. arXiv preprint arXiv:2401.01879 , year=
-
[144]
arXiv preprint arXiv:2403.13787 , year=
RewardBench: Evaluating Reward Models for Language Modeling , author=. arXiv preprint arXiv:2403.13787 , year=
-
[145]
arXiv e-prints , pages=
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models , author=. arXiv e-prints , pages=
-
[146]
arXiv preprint arXiv:2303.14420 , year=
Better Aligning Text-to-Image Models with Human Preference , author=. arXiv preprint arXiv:2303.14420 , year=
-
[147]
arXiv preprint arXiv:2302.12192 , year=
Aligning text-to-image models using human feedback , author=. arXiv preprint arXiv:2302.12192 , year=
-
[148]
arXiv preprint arXiv:2005.12729 , year=
Implementation matters in deep policy gradients: A case study on ppo and trpo , author=. arXiv preprint arXiv:2005.12729 , year=
2005 arXiv
-
[149]
arXiv preprint arXiv:2402.18571 , year=
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards , author=. arXiv preprint arXiv:2402.18571 , year=
-
[150]
Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=
Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=
-
[151]
arXiv preprint arXiv:2312.16682 , year=
Some things are more cringe than others: Preference optimization with the pairwise cringe loss , author=. arXiv preprint arXiv:2312.16682 , year=
-
[152]
arXiv preprint arXiv:2311.08516 , year=
LLMs cannot find reasoning errors, but can correct them! , author=. arXiv preprint arXiv:2311.08516 , year=
-
[153]
arXiv preprint arXiv:2412.21187 , year=
Do NOT Think That Much for 2+ 3=? On the Overthinking of o1-Like LLMs , author=. arXiv preprint arXiv:2412.21187 , year=
-
[154]
arXiv preprint arXiv:2406.04520 , year=
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning , author=. arXiv preprint arXiv:2406.04520 , year=
-
[155]
arXiv preprint arXiv:2402.14740 , year=
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms , author=. arXiv preprint arXiv:2402.14740 , year=
-
[156]
arXiv preprint arXiv:2409.06957 , year=
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation , author=. arXiv preprint arXiv:2409.06957 , year=
-
[157]
arXiv preprint arXiv:2402.04792 , year=
Direct language model alignment from online ai feedback , author=. arXiv preprint arXiv:2402.04792 , year=
-
[158]
arXiv preprint arXiv:2304.05302 , year=
RRHF: Rank Responses to Align Language Models with Human Feedback without tears , author=. arXiv preprint arXiv:2304.05302 , year=
-
[159]
arXiv preprint arXiv:2309.16240 , year=
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints , author=. arXiv preprint arXiv:2309.16240 , year=
-
[160]
Advances in Neural Information Processing Systems , volume=
Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension , author=. Advances in Neural Information Processing Systems , volume=
-
[161]
Advances in Neural Information Processing Systems , volume=
Preference-based reinforcement learning with finite-time guarantees , author=. Advances in Neural Information Processing Systems , volume=
-
[162]
Conference on Uncertainty in Artificial Intelligence , pages=
Dueling posterior sampling for preference-based reinforcement learning , author=. Conference on Uncertainty in Artificial Intelligence , pages=. 2020 , organization=
2020
-
[163]
Advances in neural information processing systems , volume=
The epoch-greedy algorithm for multi-armed bandits with side information , author=. Advances in neural information processing systems , volume=
-
[164]
arXiv preprint arXiv:2310.12036 , year=
A general theoretical paradigm to understand learning from human preferences , author=. arXiv preprint arXiv:2310.12036 , year=
-
[165]
arXiv preprint arXiv:2306.02231 , year=
Fine-Tuning Language Models with Advantage-Induced Policy Alignment , author=. arXiv preprint arXiv:2306.02231 , year=
-
[166]
arXiv preprint arXiv:2112.09332 , year=
Webgpt: Browser-assisted question-answering with human feedback , author=. arXiv preprint arXiv:2112.09332 , year=
-
[167]
arXiv preprint arXiv:2309.10202 , year=
Stabilizing RLHF through Advantage Model and Selective Rehearsal , author=. arXiv preprint arXiv:2309.10202 , year=
-
[168]
Agarwal, Alekh and Jin, Yujia and Zhang, Tong , booktitle=. V. 2023 , organization=
2023
-
[169]
Advances in neural information processing systems , volume=
Policy finetuning: Bridging sample-efficient offline and online reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[170]
International Conference on Machine Learning , pages=
Nearly minimax optimal reinforcement learning with linear function approximation , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[171]
arXiv preprint arXiv:2308.08998 , year=
Reinforced self-training (rest) for language modeling , author=. arXiv preprint arXiv:2308.08998 , year=
-
[172]
arXiv preprint arXiv:1907.01752 , year=
On the weaknesses of reinforcement learning for neural machine translation , author=. arXiv preprint arXiv:1907.01752 , year=
1907 arXiv
-
[173]
arXiv preprint arXiv:2212.09611 , year=
Optimizing Prompts for Text-to-Image Generation , author=. arXiv preprint arXiv:2212.09611 , year=
-
[174]
arXiv preprint arXiv:2204.06601 , year=
Causal confusion and reward misidentification in preference-based reward learning , author=. arXiv preprint arXiv:2204.06601 , year=
-
[175]
arXiv preprint arXiv:2012.05862 , year=
Understanding learned reward functions , author=. arXiv preprint arXiv:2012.05862 , year=
2012 arXiv
-
[176]
2004 , publisher=
Convex optimization , author=. 2004 , publisher=
2004
-
[177]
arXiv preprint arXiv:2105.08140 , year=
Uncertainty weighted actor-critic for offline reinforcement learning , author=. arXiv preprint arXiv:2105.08140 , year=
-
[178]
arXiv preprint arXiv:2112.00861 , year=
A general language assistant as a laboratory for alignment , author=. arXiv preprint arXiv:2112.00861 , year=
-
[179]
arXiv preprint arXiv:1707.06347 , year=
Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=
-
[180]
SIAM Journal on Mathematics of Data Science , volume=
Feel-good thompson sampling for contextual bandits and reinforcement learning , author=. SIAM Journal on Mathematics of Data Science , volume=. 2022 , publisher=
2022
-
[181]
Advances in Neural Information Processing Systems , volume=
Better exploration with optimistic actor critic , author=. Advances in Neural Information Processing Systems , volume=
-
[182]
Optimistic Exploration with Backward Bootstrapped Bonus for Deep Reinforcement Learning , author=
-
[183]
arXiv preprint arXiv:2002.12174 , year=
Optimistic exploration even with a pessimistic initialisation , author=. arXiv preprint arXiv:2002.12174 , year=
2002 arXiv
-
[184]
arXiv preprint arXiv:2212.08073 , year=
Constitutional ai: Harmlessness from ai feedback , author=. arXiv preprint arXiv:2212.08073 , year=
-
[185]
Econometrica: Journal of the Econometric Society , pages=
Intransitivity, utility, and the aggregation of preference patterns , author=. Econometrica: Journal of the Econometric Society , pages=. 1954 , publisher=
1954
-
[186]
, author=
Intransitivity of preferences. , author=. Psychological review , volume=. 1969 , publisher=
1969
-
[187]
arXiv preprint arXiv:2312.00886 , year=
Nash Learning from Human Feedback , author=. arXiv preprint arXiv:2312.00886 , year=
-
[188]
arXiv preprint arXiv:2401.04056 , year=
A Minimaximalist Approach to Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2401.04056 , year=
-
[189]
Journal of computer and system sciences , volume=
A decision-theoretic generalization of on-line learning and an application to boosting , author=. Journal of computer and system sciences , volume=. 1997 , publisher=
1997
-
[190]
2023 , publisher =
OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants , author =. 2023 , publisher =
2023
-
[191]
arXiv preprint arXiv:2402.05749 , year=
Generalized Preference Optimization: A Unified Approach to Offline Alignment , author=. arXiv preprint arXiv:2402.05749 , year=
-
[192]
2024 , eprint=
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , author=. 2024 , eprint=
2024
-
[193]
arXiv preprint arXiv:2409.11704 , year=
From lists to emojis: How format bias affects model alignment , author=. arXiv preprint arXiv:2409.11704 , year=
-
[194]
SIAM Journal on Optimization , volume=
Convergence analysis of a proximal-like minimization algorithm using Bregman functions , author=. SIAM Journal on Optimization , volume=. 1993 , publisher=
1993
-
[195]
arXiv preprint arXiv:2404.16792 , year=
Weak-to-strong extrapolation expedites alignment , author=. arXiv preprint arXiv:2404.16792 , year=
-
[196]
arXiv preprint arXiv:2310.17303 , year=
Regularized rl , author=. arXiv preprint arXiv:2310.17303 , year=
-
[197]
arXiv preprint arXiv:2309.12284 , year=
Metamath: Bootstrap your own mathematical questions for large language models , author=. arXiv preprint arXiv:2309.12284 , year=
-
[198]
HuggingFace repository , howpublished =
OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces , author =. HuggingFace repository , howpublished =. 2023 , publisher =
2023
-
[199]
, author=
Amplify-Instruct: Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training. , author=. arXiv preprint arXiv:(coming soon) , url=
-
[200]
HuggingFace repository , howpublished =
Snorkel-Mistral-PairRM-DPO , author=. HuggingFace repository , howpublished =. 2024 , publisher =
2024
-
[201]
Advances in Neural Information Processing Systems , volume=
Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume=
-
[202]
Advances in Neural Information Processing Systems , volume=
Star: Bootstrapping reasoning with reasoning , author=. Advances in Neural Information Processing Systems , volume=
-
[203]
Advances in Neural Information Processing Systems , volume=
Evaluating and improving tool-augmented computation-intensive math reasoning , author=. Advances in Neural Information Processing Systems , volume=
-
[204]
arXiv preprint arXiv:2211.14275 , year=
Solving math word problems with process-and outcome-based feedback , author=. arXiv preprint arXiv:2211.14275 , year=
-
[205]
arXiv preprint arXiv:2405.14734 , year=
Simpo: Simple preference optimization with a reference-free reward , author=. arXiv preprint arXiv:2405.14734 , year=
-
[206]
arXiv preprint arXiv:2312.06585 , year=
Beyond human data: Scaling self-training for problem-solving with language models , author=. arXiv preprint arXiv:2312.06585 , year=
-
[207]
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving , author=
-
[208]
arXiv preprint arXiv:2305.11738 , year=
Critic: Large language models can self-correct with tool-interactive critiquing , author=. arXiv preprint arXiv:2305.11738 , year=
-
[209]
arXiv preprint arXiv:2211.12588 , year=
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks , author=. arXiv preprint arXiv:2211.12588 , year=
-
[210]
International Conference on Machine Learning , pages=
Pal: Program-aided language models , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[211]
arXiv preprint arXiv:2401.10020 , year=
Self-Rewarding Language Models , author=. arXiv preprint arXiv:2401.10020 , year=
-
[212]
arXiv preprint arXiv:2210.17517 , year=
Lila: A unified benchmark for mathematical reasoning , author=. arXiv preprint arXiv:2210.17517 , year=
-
[213]
arXiv preprint arXiv:2211.16482 , year=
Chaining simultaneous thoughts for numerical reasoning , author=. arXiv preprint arXiv:2211.16482 , year=
-
[214]
arXiv preprint arXiv:2210.16257 , year=
Solving math word problems via cooperative reasoning induced language models , author=. arXiv preprint arXiv:2210.16257 , year=
-
[215]
arXiv preprint arXiv:2205.10625 , year=
Least-to-most prompting enables complex reasoning in large language models , author=. arXiv preprint arXiv:2205.10625 , year=
-
[216]
arXiv preprint arXiv:2109.00110 , year=
Minif2f: a cross-system benchmark for formal olympiad-level mathematics , author=. arXiv preprint arXiv:2109.00110 , year=
-
[217]
2008 7th IEEE international conference on development and learning , pages=
Tamer: Training an agent manually via evaluative reinforcement , author=. 2008 7th IEEE international conference on development and learning , pages=. 2008 , organization=
2008
-
[218]
Proceedings of KDD cup and workshop , volume=
The netflix prize , author=. Proceedings of KDD cup and workshop , volume=. 2007 , organization=
2007
-
[219]
arXiv preprint arXiv:2310.06825 , year=
Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=
-
[220]
arXiv preprint arXiv:1803.04585 , year=
Categorizing variants of Goodhart's Law , author=. arXiv preprint arXiv:1803.04585 , year=
-
[221]
Thirty-seventh Conference on Neural Information Processing Systems , year=
Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=
-
[222]
CoRR, abs/2209.13085, 2022
Defining and characterizing reward hacking. CoRR, abs/2209.13085, 2022. doi: 10.48550 , author=. arXiv preprint arXiv.2209.13085 , year=
2022 arXiv
-
[223]
arXiv preprint arXiv:2307.15217 , year=
Open problems and fundamental limitations of reinforcement learning from human feedback , author=. arXiv preprint arXiv:2307.15217 , year=
-
[224]
arXiv preprint arXiv:2308.15812 , year=
Peering through preferences: Unraveling feedback acquisition for aligning large language models , author=. arXiv preprint arXiv:2308.15812 , year=
-
[225]
arXiv preprint arXiv:2310.00898 , year=
Enable Language Models to Implicitly Learn Self-Improvement From Data , author=. arXiv preprint arXiv:2310.00898 , year=
-
[226]
arXiv preprint arXiv:2310.02263 , year=
Contrastive Post-training Large Language Models on Data Curriculum , author=. arXiv preprint arXiv:2310.02263 , year=
-
[227]
Advances in Neural Information Processing Systems , volume=
Eluder dimension and the sample complexity of optimistic exploration , author=. Advances in Neural Information Processing Systems , volume=
-
[228]
arXiv preprint arXiv:2305.17608 , year=
Reward Collapse in Aligning Large Language Models , author=. arXiv preprint arXiv:2305.17608 , year=
-
[229]
arXiv preprint arXiv:2305.14816 , year=
Provable Offline Reinforcement Learning with Human Feedback , author=. arXiv preprint arXiv:2305.14816 , year=
-
[230]
Machine learning , volume=
Finite-time analysis of the multiarmed bandit problem , author=. Machine learning , volume=. 2002 , publisher=
2002
-
[231]
arXiv preprint arXiv:2202.05448 , year=
Fast rates in pool-based batch active learning , author=. arXiv preprint arXiv:2202.05448 , year=
-
[232]
Advances in neural information processing systems , volume=
Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=
-
[233]
2023 , eprint=
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM , author=. 2023 , eprint=
2023
-
[234]
CS Dept., UW Seattle, Seattle, WA, USA, Tech
Reinforcement learning: Theory and algorithms , author=. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , volume=
-
[235]
arXiv preprint arXiv:2204.05862 , year=
Training a helpful and harmless assistant with reinforcement learning from human feedback , author=. arXiv preprint arXiv:2204.05862 , year=
-
[236]
International Conference on Machine Learning , pages=
Improved optimistic algorithms for logistic bandits , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[237]
2023 , url=
Hanze Dong and Wei Xiong and Deepanshu Goyal and Yihan Zhang and Winnie Chow and Rui Pan and Shizhe Diao and Jipeng Zhang and KaShun SHUM and Tong Zhang , journal=. 2023 , url=
2023
-
[238]
arXiv preprint arXiv:1909.08593 , year=
Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=
1909 arXiv
-
[239]
arXiv preprint arXiv:2109.10862 , year=
Recursively summarizing books with human feedback , author=. arXiv preprint arXiv:2109.10862 , year=
-
[240]
Introducing claude , author=
-
[241]
2020 , publisher=
Bandit algorithms , author=. 2020 , publisher=
2020
-
[242]
Mathematics of Operations Research , volume=
Linearly parameterized bandits , author=. Mathematics of Operations Research , volume=. 2010 , publisher=
2010
-
[243]
International Conference on Machine Learning , pages=
Scaling laws for reward model overoptimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[244]
Textbooks Are All You Need II:
Li, Yuanzhi and Bubeck, S. Textbooks Are All You Need II:. arXiv preprint arXiv:2309.05463 , year=
-
[245]
Geng, Xinyang and Liu, Hao , title =
-
[246]
arXiv preprint arXiv:2306.05685 , year=
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena , author=. arXiv preprint arXiv:2306.05685 , year=
-
[247]
International Conference on Machine Learning , pages=
A self-play posterior sampling algorithm for zero-sum Markov games , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[248]
arXiv preprint arXiv:2205.11275 , year=
RL with KL penalties is better viewed as Bayesian inference , author=. arXiv preprint arXiv:2205.11275 , year=
-
[249]
, author=
Maximum entropy inverse reinforcement learning. , author=. Aaai , volume=. 2008 , organization=
2008
-
[250]
arXiv preprint arXiv:2309.06256 , year=
Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models , author=. arXiv preprint arXiv:2309.06256 , year=
-
[251]
Advances in neural information processing systems , volume=
Thinking fast and slow with deep learning and tree search , author=. Advances in neural information processing systems , volume=
-
[252]
2023 , school=
A Sufficient Condition of Sample-Efficient Reinforcement Learning with General Function Approximation , author=. 2023 , school=
2023
-
[253]
arXiv preprint arXiv:2405.17931 , year=
Online merging optimizers for boosting rewards and mitigating tax in alignment , author=. arXiv preprint arXiv:2405.17931 , year=
-
[254]
arXiv preprint arXiv:2406.16768 , year=
WARP: On the Benefits of Weight Averaged Rewarded Policies , author=. arXiv preprint arXiv:2406.16768 , year=
-
[255]
arXiv preprint arXiv:2305.10425 , year=
Slic-hf: Sequence likelihood calibration with human feedback , author=. arXiv preprint arXiv:2305.10425 , year=
-
[256]
arXiv preprint arXiv:2106.03352 , year=
The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces , author=. arXiv preprint arXiv:2106.03352 , year=
-
[257]
arXiv preprint arXiv:2401.09003 , year=
Augmenting math word problems via iterative question composing , author=. arXiv preprint arXiv:2401.09003 , year=
-
[258]
arXiv preprint arXiv:2107.14702 , year=
Towards general function approximation in zero-sum markov games , author=. arXiv preprint arXiv:2107.14702 , year=
-
[259]
GitHub repository , howpublished =
Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang , title =. GitHub repository , howpublished =. 2020 , publisher =
2020
-
[260]
arXiv preprint arXiv:2309.05653 , year=
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning , author=. arXiv preprint arXiv:2309.05653 , year=
-
[261]
2023 , publisher =
SlimOrca: An Open Dataset of GPT-4 Augmented FLAN Reasoning Traces, with Verification , author =. 2023 , publisher =
2023
-
[262]
arXiv preprint arXiv:2309.06657 , year=
Statistical rejection sampling improves preference optimization , author=. arXiv preprint arXiv:2309.06657 , year=
-
[263]
arXiv preprint arXiv:2305.18290 , year=
Direct preference optimization: Your language model is secretly a reward model , author=. arXiv preprint arXiv:2305.18290 , year=
-
[264]
Advances in Neural Information Processing Systems , volume=
Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=
-
[265]
arXiv preprint arXiv:2111.04850 , year=
Dueling rl: reinforcement learning with trajectory preferences , author=. arXiv preprint arXiv:2111.04850 , year=
-
[266]
arXiv preprint arXiv:2301.11270 , year=
Principled Reinforcement Learning with Human Feedback from Pairwise or K -wise Comparisons , author=. arXiv preprint arXiv:2301.11270 , year=
-
[267]
Advances in neural information processing systems , volume=
Provable benefits of actor-critic methods for offline reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[268]
Advances in Neural Information Processing Systems , volume=
Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning , author=. Advances in Neural Information Processing Systems , volume=
-
[269]
arXiv preprint arXiv:2305.14387 , year=
Alpacafarm: A simulation framework for methods that learn from human feedback , author=. arXiv preprint arXiv:2305.14387 , year=
-
[270]
Journal of Machine Learning Research , volume=
A survey of preference-based reinforcement learning methods , author=. Journal of Machine Learning Research , volume=. 2017 , publisher=
2017
-
[271]
arXiv preprint arXiv:2302.11048 , year=
Adversarial model for offline reinforcement learning , author=. arXiv preprint arXiv:2302.11048 , year=
-
[272]
Advances in neural information processing systems , volume=
Bellman-consistent pessimism for offline reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[273]
Advances in Neural Information Processing Systems , volume=
Bridging offline reinforcement learning and imitation learning: A tale of pessimism , author=. Advances in Neural Information Processing Systems , volume=
-
[274]
The Journal of Machine Learning Research , volume=
On the theory of policy gradient methods: Optimality, approximation, and distribution shift , author=. The Journal of Machine Learning Research , volume=. 2021 , publisher=
2021
-
[275]
International Conference on Machine Learning , pages=
Is pessimism provably efficient for offline rl? , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[276]
arXiv preprint arXiv:2306.17563 , year=
Large language models are effective text rankers with pairwise ranking prompting , author=. arXiv preprint arXiv:2306.17563 , year=
-
[277]
International Conference on Artificial Intelligence and Statistics , pages=
Corruption-robust offline reinforcement learning , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=
2022
-
[278]
arXiv preprint arXiv:2306.02561 , year=
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion , author=. arXiv preprint arXiv:2306.02561 , year=
-
[279]
arXiv preprint arXiv:2212.04717 , year=
On the Sensitivity of Reward Inference to Misspecified Human Models , author=. arXiv preprint arXiv:2212.04717 , year=
-
[280]
arXiv preprint arXiv:2211.01962 , year=
Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond , author=. arXiv preprint arXiv:2211.01962 , year=
-
[281]
arXiv preprint arXiv:2310.16944 , year=
Zephyr: Direct distillation of lm alignment , author=. arXiv preprint arXiv:2310.16944 , year=
-
[282]
arXiv preprint arXiv:2301.01392 , year=
Benchmarks and algorithms for offline preference-based reward learning , author=. arXiv preprint arXiv:2301.01392 , year=
-
[283]
arXiv preprint arXiv:2210.06718 , year=
Hybrid rl: Using both offline and online data can make rl efficient , author=. arXiv preprint arXiv:2210.06718 , year=
-
[284]
arXiv preprint arXiv:2310.02743 , year=
Reward Model Ensembles Help Mitigate Overoptimization , author=. arXiv preprint arXiv:2310.02743 , year=
-
[285]
International Conference on Machine Learning , pages=
Provably efficient exploration in policy optimization , author=. International Conference on Machine Learning , pages=. 2020 , organization=
2020
-
[286]
Conference on Learning Theory , pages=
Cautiously optimistic policy optimization and exploration with linear function approximation , author=. Conference on Learning Theory , pages=. 2021 , organization=
2021
-
[287]
International Conference on Machine Learning , pages=
Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[288]
arXiv preprint arXiv:2305.08841 , year=
A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes , author=. arXiv preprint arXiv:2305.08841 , year=
-
[289]
arXiv preprint arXiv:2305.18438 , year=
Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism , author=. arXiv preprint arXiv:2305.18438 , year=
-
[290]
arXiv preprint arXiv:2210.01241 , year=
Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization , author=. arXiv preprint arXiv:2210.01241 , year=
-
[291]
arXiv preprint arXiv:2402.00782 , year=
Dense Reward for Free in Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2402.00782 , year=
-
[292]
arXiv preprint arXiv:2404.18922 , year=
DPO Meets PPO: Reinforced Token Optimization for RLHF , author=. arXiv preprint arXiv:2404.18922 , year=
-
[293]
arXiv preprint arXiv:2404.08495 , year=
Dataset Reset Policy Optimization for RLHF , author=. arXiv preprint arXiv:2404.08495 , year=
-
[294]
arXiv preprint arXiv:2402.05369 , year=
Noise Contrastive Alignment of Language Models with Explicit Rewards , author=. arXiv preprint arXiv:2402.05369 , year=
-
[295]
arXiv preprint arXiv:2305.00633 , volume=
Decomposition enhances reasoning via self-evaluation guided decoding , author=. arXiv preprint arXiv:2305.00633 , volume=
-
[296]
arXiv preprint arXiv:2210.14215 , year=
In-context reinforcement learning with algorithm distillation , author=. arXiv preprint arXiv:2210.14215 , year=
-
[297]
arXiv preprint arXiv:2405.00675 , year=
Self-Play Preference Optimization for Language Model Alignment , author=. arXiv preprint arXiv:2405.00675 , year=
-
[298]
arXiv preprint arXiv:2212.09561 , year=
Large language models are better reasoners with self-verification , author=. arXiv preprint arXiv:2212.09561 , year=
-
[299]
arXiv preprint arXiv:2308.03188 , year=
Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies , author=. arXiv preprint arXiv:2308.03188 , year=
-
[300]
An implementation of generative prm , author=
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.