Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Replacing live agent rollouts with replayed teacher prefixes — weighted toward early, reliable steps — makes multi-turn on-policy distillation offline, at least 4× faster per rollout, and just as accurate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:33 UTC pith:PCUFW5E7

load-bearing objection Practical offline distillation recipe with a suggestive but under-specified theoretical justification; worth engaging for the method and experiments. the 3 major comments →

arxiv 2607.04763 v3 pith:PCUFW5E7 submitted 2026-07-06 cs.LG cs.AIcs.CLstat.ML

Multi-Turn On-Policy Distillation with Prefix Replay

Baohao Liao , Hanze Dong , Christof Monz , Xinxing Xu , Li Dong , Furu Wei This is my paper
classification cs.LG cs.AIcs.CLstat.ML
keywords multi-turn distillationon-policy distillationprefix replayLLM agentsreliability-aware prefix distributiontwo-sided distribution shiftstep-decay scheduleoffline distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that multi-turn on-policy distillation for LLM agents can be run entirely offline by replaying pre-recorded teacher trajectories, and that the right way to do it is to design a reliability-aware prefix distribution rather than maximize student-on-policy training. It identifies a "prefix trap": histories that are more relevant to the student can place the teacher on prefixes where its supervision is unreliable. A bound splits the gap to an ideal interactive objective into a student-occupancy term and a teacher-reliability term, and ReOPD realizes the resulting balance with the step-decay schedule κ^t, applied by sampling early, low-shift prefixes more often. Empirically the method preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than fully online OPD. If correct, this turns expensive live environment interaction into a reusable offline resource for distillation across tools and environments.

Core claim

The central discovery is that multi-turn OPD should be treated as reliability-aware prefix distribution design rather than "maximally on-policy" distillation. The paper proves a bound on the gap between a replayed-prefix objective and the ideal interactive objective: the gap is at most a student-occupancy mismatch term plus a teacher-reliability term, so pushing prefixes fully onto the student is not automatically optimal. ReOPD implements the resulting balance by replaying teacher prefixes verbatim, letting the student generate only the action at the evaluated step, and weighting positions by ω(t)=κ^t so early, low-shift prefixes carry more training mass. Empirically, the method matches OPD

What carries the argument

The load-bearing object is the "effective prefix distribution" ρ_t = w_t P_t, the distribution over interaction histories at step t on which training mass is placed. A bound decomposes the gap to ideal interactive OPD into a student-occupancy term and a teacher-reliability term; the paper's bridge objective minimizes a weighted combination of KL divergences to the student and teacher occupancies, whose exact solution is a geometric bridge. Because the exact importance weight is a high-variance prefix-level likelihood ratio that empirically decays with depth, ReOPD replaces it with the one-parameter step-decay schedule ω(t)=κ^t, applied by sampling positions proportionally. This schedule is t

Load-bearing premise

The position-only step-decay schedule ω(t)=κ^t is a valid replacement for the exact likelihood-ratio weight, which assumes the average per-step teacher–student gap is roughly constant along the trajectory, while κ=0.6 is set empirically rather than derived from that gap; if depth does not actually track the reliability ratio, the method's advantage over online OPD is not predicted by the theory.

What would settle it

On a fixed teacher pool, compute the exact per-position weights br_t = ∏_{s<t} π_θold(a_s)/π_T(a_s) and regress log(br_t) against step index t; if the log-ratio is not approximately linear in t, or if depth explains only a small fraction of its variance, the step-decay proxy ω(t)=κ^t is not a faithful replacement for the density-ratio weight. Alternatively, run ReOPD against OPD on a task where the teacher is unreliable on student histories but the teacher–student gap is not monotone in depth; if ReOPD loses, the reliability-aware explanation is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • ReOPD preserves or improves OPD-level accuracy across teacher and student scales in mathematical-reasoning-with-Python and search environments.
  • ReOPD uses zero tool calls during student training and is at least 4× faster per rollout than OPD; with teacher prefix collection cost included, it remains over 2× faster.
  • A single student can be distilled jointly across heterogeneous environments from one merged offline pool, without deploying all environments during student training.
  • The teacher's own RL rollouts can be reused as the prefix pool at no extra collection cost; a stationary pool drawn from the final teacher gives nearly identical students.
  • Fully student-on-policy training is not uniformly optimal: when the teacher is unreliable on student-induced histories, teacher-anchored prefixes improve distillation.
  • At a fixed distillation target, prefixes from the teacher itself outperform prefixes from larger or stronger generators, showing that teacher reliability, not raw capability, governs prefix quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the bridge-to-schedule map suggests κ could be set a priori as exp(−γ_t c̄) once the per-step teacher–student gap c̄ is measured on the pool; the paper fixes κ=0.6 empirically, so a gap-adaptive schedule is a direct testable extension.
  • Beyond the paper: the step-decay surrogate uses only depth as a proxy for reliability, but a data-dependent weight derived from the student's own sampling probabilities at the evaluated step could capture per-history variation; the paper mentions this as future work.
  • Beyond the paper: the two-regime result implies a natural curriculum — start with a steep decay when the student is far below the teacher and flatten it toward uniform as the gap closes, which would make the schedule adapt to training progress.
  • Beyond the paper: if the prefix pool is free whenever the teacher is RL-trained, multi-teacher distillation to a single generalist student becomes a logistics problem (merging offline pools) rather than a compute-cost problem, which could materially change how agent teams are trained.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces Replayed-Prefix On-Policy Distillation (ReOPD), an offline distillation method for multi-turn agentic tasks. Instead of rolling the student through the environment and querying the teacher on student-sampled histories (OPD), ReOPD reuses a fixed pool of teacher trajectories: it replays a teacher-forced prefix, lets the student act at the supervised step, and uses the teacher conditional as the target. The authors identify a 'prefix trap' with a two-sided shift (student occupancy vs. teacher reliability) and derive a bound (Prop. 1) decomposing the gap to an ideal interactive objective. They propose a geometric bridge between student and teacher occupancies and approximate the resulting weight by a step-decay schedule ω(t)=κ^t, which reallocates mass to early, low-shift steps. Experiments in math-with-Python and search across multiple teacher/student scales show ReOPD matches or improves OPD accuracy while being at least 4× faster per rollout and using zero tool calls during student training.

Significance. The empirical contribution is practically significant: ReOPD turns expensive online interaction into a reusable offline pool, and the experimental design is thorough (two environments, multiple teacher/student scales, ablations, multi-environment joint training). Proposition 1 is correct, and the two-sided decomposition is a useful conceptual framing. However, the connection between the geometric bridge and the implemented step-decay schedule is asserted rather than established: κ is fixed empirically, and Figure 4 does not provide the supporting evidence claimed. The method itself remains useful even if the theoretical narrative is weakened; the paper's operational speedups and the observation that teacher-anchored prefixes help in a large-gap regime are valuable independent of the derivation.

major comments (3)
  1. [§4, Eq. (8) and §5, implementation details] The paper claims κ is pinned by the bridge-to-schedule map κ=exp(−γ_t c̄) in Eq. (8), but the implementation section states "steepness is set empirically" and uses κ=0.6 without reporting γ_t or c̄. The map is therefore underdetermined: γ_t is never specified as a function of t, and c̄ (the average per-step teacher–student gap) is not measured in either environment. As written, the schedule is a tuned heuristic, which conflicts with the claim that ReOPD "realizes" the reliability-aware prefix distribution. To restore the link, the authors should report c̄ on the math and search pools and show whether κ=0.6 matches exp(−γ_t c̄) for a plausible γ_t, or reframe the contribution as an empirical simplification with the exact weight (7) as the principled target and the step-decay as a separately validated approximation.
  2. [Figure 4 and surrounding text] Figure 4 is the only empirical evidence for "step index is a strong proxy for the likelihood-ratio weight." As plotted, the log-ratio values are all in [0.36, 0.41]—a span of about 0.05—whereas the implemented κ=0.6^t decay corresponds to a per-step log decrease of log(1/0.6)≈0.51. The trend in the figure is far too shallow to justify the steep schedule used in the algorithm. Additionally, the text states that the student-to-teacher ratio is typically below one (so its log should be negative), but the y-axis is positive; this inconsistency suggests the figure is plotting an absolute value or inverse ratio, which should be stated. The figure, as presented, does not support the claim that depth explains most of the variation, and it does not justify the specific steepness of κ=0.6.
  3. [§4, "From the bridge to a step-decaying weight"] The bridge objective in Eq. (5) defines a per-history effective distribution ρ⋆_t, and the exact weight (7) varies within each step. The implemented surrogate ω(t)=κ^t is constant within a step and is normalized across positions, so it does not solve the optimization (5). The paper asserts that both forms "point the same way" because br_t decays with depth, but this is precisely the claim that Figure 4 fails to establish (see previous comment). Moreover, no experiment compares the step-decay surrogate against the exact weight (7) or against a per-history weight derived from the bridge. Without such a comparison, the assertion that ReOPD is a "practical implementation of the resulting effective distribution" remains unsupported. The authors should either provide a direct comparison (e.g., using the exact weight on a subset of steps) or explicitly downgrade the step-decay to a heuristic th
minor comments (6)
  1. [Title page] The rendered URL "/gl⌢beProject Page: ..." appears to be a formatting artifact and should be cleaned up.
  2. [§4, Eq. (8)] Since log br_t ≈ −(t−1)c̄, the exponent in Eq. (8) should be κ^{t−1} rather than κ^t; the "up to a constant" clause can absorb this, but the subsequent identification with ω(t)=κ^t should be stated more carefully.
  3. [§5.2, Teacher's prefix resampling] The text references "Table 8" for the time used in teacher prefix resampling, but only Figure 8 appears in the submission. The cross-reference should be corrected.
  4. [Table 1] The Search max generation tokens value is listed as "4196"; this is likely a typo (perhaps 4096 or 8192) and should be fixed.
  5. [Figure 4] The y-axis label should clarify whether the plotted quantity is log br_t, −log br_t, or some other transformation, so that it is consistent with the text's statement that the ratio is typically below one.
  6. [Table 4 header] The column header "MA TH500" has a spacing typo and should read "MATH500."

Circularity Check

0 steps flagged

No significant circularity: the two-regime prediction is qualitative and independently tested; the tuned κ is an openly acknowledged implementation knob, not a disguised input.

full rationale

The paper's derivation chain is self-contained rather than circular. Proposition 1 is a genuine decomposition of the objective gap into occupancy mismatch and teacher-reliability terms; the geometric bridge in Eq. (6) is the explicit solution of the KL-weighted objective in Eq. (5); Eq. (7) is a well-defined importance weight; and Eq. (8) is an explicitly stated approximation, not an equality forced by construction. The central two-regime prediction is qualitative—ReOPD should help when the teacher–student gap is large and match OPD when the teacher is reliable on student histories—and it is tested on held-out benchmarks (Tables 4 and 5). The step-decay steepness κ=0.6 is admittedly set empirically in Section 5 rather than computed from the Eq. (8) map, and the limitations section concedes that the surrogate 'captures how deep a prefix is rather than directly measuring how far a given prefix lies from the overlap.' This weakens the quantitative authority of the bridge-to-schedule map and is a legitimate correctness/risk concern, but it does not make any prediction equivalent to its input: the two-regime outcome could have failed, and the fitted knob is openly disclosed rather than renamed as a derived parameter. Self-citations (e.g., Liao et al. 2026, Ye et al. 2026a/b) appear only in related work and are not load-bearing. No circular step is exhibited.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or architectural entities. Its conceptual objects (prefix trap, two-sided shift) are named phenomena rather than postulates. The main input the reader 'pays for' is the fitted schedule steepness κ and the unverified surrogate assumptions relating teacher support, step depth, and reliability.

free parameters (1)
  • κ (step-decay base) = 0.6
    Single steepness knob of the step-decay schedule. Section 5 says 'unless noted, ReOPD uses ... κ=0.6' and 'steepness is set empirically'; the stated map κ=exp(−γ_t c̄) in Eq. (8) is not used to compute it.
axioms (4)
  • domain assumption A1: The loss against the ideal target is uniformly bounded by B (Assumption 1).
    Needed for the TV bound in Prop 1; holds for bounded divergences, but the ideal target q*_t is unobserved.
  • ad hoc to paper Teacher support is a valid surrogate for unobserved teacher reliability.
    Sec 3: 'Since the true reliability error is generally unobserved, we use teacher support as a practical surrogate'; this turns the unmeasurable reliability term into a tractable KL to the teacher occupancy.
  • ad hoc to paper The per-step teacher–student gap c̄ is approximately stationary, so log br_t ≈ −(t−1)c̄, and step depth is a strong proxy for the likelihood-ratio weight.
    Sec 4, Eq. (8) and Figure 4: the step-decay schedule replaces the exact weight (7) with κ^t. If depth is not a good proxy, the schedule's justification collapses.
  • domain assumption The teacher's RL rollouts (a mixed-policy pool spanning checkpoints) are within the final teacher's reliable support.
    Sec 5, Table 7 checks this for the tested setting; the method's 'free by-product' claim depends on it.

pith-pipeline@v1.3.0-alltime-deepseek · 25452 in / 16789 out tokens · 193459 ms · 2026-08-02T08:33:38.597602+00:00 · methodology

0 comments
read the original abstract

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.

Figures

Figures reproduced from arXiv: 2607.04763 by Baohao Liao, Christof Monz, Furu Wei, HanZe Dong, Li Dong, Xinxing Xu.

Figure 1
Figure 1. Figure 1: ReOPD keeps the benefits of on-policy distillation while removing environment in￾teraction. ReOPD matches or improves OPD accuracy, but trains much faster per step and eliminates tool calls during training by replaying teacher-recorded prefixes instead of executing fresh environment rollouts. The student/teacher models are Qwen3-4B-Instruct-2507 and Qwen3-8B, respectively. ∗Equal Contribution. †Work done d… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison between OPD and ReOPD for agentic tasks. Up: OPD with online environ￾ment. The environment is always alive during training. And all steps equally contribute to the loss. Down: ReOPD with offline environment. The environment is only alive for collecting the teacher’s trajectories, which can happen during the training of the teacher agent by using RL, like GRPO. Afterwards, the envi￾ronment is not… view at source ↗
Figure 3
Figure 3. Figure 3: Training a shared student agent on multiple heterogeneous environments with OPD and ReOPD. Left: With an increasing number of environments, the operational complexity grows for OPD due to the heavy deployment of the environments. Right: ReOPD doesn’t require the deployment of all environments at the same time. The teacher’s trajectories can be collected separately for different environments, and then be me… view at source ↗
Figure 4
Figure 4. Figure 4: Step index is a strong proxy for the likelihood-ratio weight. For each supervised position we compute the per-position weight rbt = Q s<t πθold (as)/πT (as) – the student-to-teacher likelihood ratio along the teacher prefix – and plot it against the step index t. The empirical weight decays with depth, because each factor compares the student’s and teacher’s probability of the teacher’s own recorded action… view at source ↗
Figure 5
Figure 5. Figure 5: Favoring early, low-shift steps improves distillation. Both panels use ReOPD’s sampling implementation and put more training mass on early interaction steps, where the student stays close to the teacher and the two-sided shift is small. (a) Sampling by chunk: drawing supervised positions from early chunks yields the best student, and pushing mass to later chunks degrades it. (b) Sampling with decay κ: appl… view at source ↗
Figure 6
Figure 6. Figure 6: The resources used by different environments. For math, the processes are utilized for the execution of Python code in parallel. For search, the GPU memory is uti￾lized for deploying the embedding model and the storage of the Wikipedia embeddings. On search, ReOPD essentially matches OPD. With a Qwen3-4B teacher, OPD and ReOPD obtain 40.6 and 40.5 average accuracy, respectively; with a Qwen3-8B teacher, th… view at source ↗
Figure 6
Figure 6. Figure 6: Prefix source: the teacher’s own model gives the best prefixes, not a stronger gener￾ator. Teacher Qwen3-8B, student Qwen3-4B-Instruct-2507; we fix the distillation target to the teacher and vary only the prefix generator. Performance peaks when prefixes come from the teacher itself and degrades with larger or stronger generators – a direct test of the teacher-reliability view: the teacher’s conditional is… view at source ↗
Figure 7
Figure 7. Figure 7: Prefix source: the teacher’s own model gives the best prefixes, not a stronger gener￾ator. Teacher Qwen3-8B, student Qwen3-4B-Instruct-2507; we fix the distillation target to the teacher and vary only the prefix generator. Performance peaks when prefixes come from the teacher itself and degrades with larger or stronger generators – a direct test of the teacher-reliability view: the teacher’s conditional is… view at source ↗
Figure 8
Figure 8. Figure 8: Rollout time when taking the teacher’s rollout into account. When the teacher’s trajectories from RL are not avail￾able, one could use the teacher to sample the trajectories for all prompts at once. We take this time into account for the comparison. For this setting, the environment is only online for the teacher’s rollout. We studied Replayed-Prefix On-Policy Distillation in an efficient, off-environment … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EvoReason: Self-Evolving Reasoning Primitive-Guided On-Policy Distillation for Latent Reasoning in Generative Recommendation

    cs.IR 2026-07 conditional novelty 5.0

    A self-evolving, primitive-guided distillation method for latent reasoning in generative recommendation reports consistent ranking gains over existing latent-reasoning baselines.

Reference graph

Works this paper leans on

300 extracted references · 162 linked inside Pith · cited by 1 Pith paper

  1. [1]

    2025 , howpublished =

    Zilin Zhu and Chengxing Xie and Xin Lv and slime Contributors , title =. 2025 , howpublished =

    2025

  2. [2]

    arXiv preprint arXiv:2212.03533 , year=

    Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year=

    Pith/arXiv arXiv

  3. [3]

    Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

    Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

    2020

  4. [4]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  5. [5]

    Transactions of the Association for Computational Linguistics , volume=

    Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  6. [6]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Solving quantitative reasoning problems with language models , author=. Advances in Neural Information Processing Systems , volume=

  8. [8]

    Hugging Face repository , volume=

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions , author=. Hugging Face repository , volume=

  9. [9]

    arXiv preprint arXiv:2504.11536 , year=

    Retool: Reinforcement learning for strategic tool use in llms , author=. arXiv preprint arXiv:2504.11536 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  11. [11]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  12. [12]

    Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=

    Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , author=. Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=

  13. [13]

    Transactions of the Association for Computational Linguistics (TACL) , volume=

    MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics (TACL) , volume=

  14. [14]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  15. [15]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , year=

    Measuring and Narrowing the Compositionality Gap in Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , year=

    2023

  16. [16]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  17. [17]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  18. [18]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  19. [19]

    M. J. Kearns , title =

  20. [20]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  21. [21]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  22. [22]

    Suppressed for Anonymity , author=

  23. [23]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  24. [24]

    arXiv preprint arXiv:2501.07301 , year=

    The lessons of developing process reward models in mathematical reasoning , author=. arXiv preprint arXiv:2501.07301 , year=

    Pith/arXiv arXiv

  25. [25]

    GitHub repository , howpublished =

    Wei Xiong and Hanning Zhang and Nan Jiang and Tong Zhang , title =. GitHub repository , howpublished =. 2024 , publisher =

    2024

  26. [26]

    arXiv preprint arXiv:2412.06559 , year=

    Processbench: Identifying process errors in mathematical reasoning , author=. arXiv preprint arXiv:2412.06559 , year=

    Pith/arXiv arXiv

  27. [27]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  28. [28]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  29. [29]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    Self-refine: Iterative refinement with self-feedback , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Language models can solve computer tasks , author=. Advances in Neural Information Processing Systems , volume=

  33. [33]

    Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning

    Li, Ming and Chen, Lichang and Chen, Jiuhai and He, Shwai and Gu, Jiuxiang and Zhou, Tianyi. Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning. Findings of the Association for Computational Linguistics ACL 2024. 2024

    2024

  34. [34]

    arXiv preprint arXiv:2211.00053 , year=

    Generating sequences by learning to self-correct , author=. arXiv preprint arXiv:2211.00053 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2203.11171 , year=

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2407.18219 , year=

    Recursive introspection: Teaching language model agents how to self-improve , author=. arXiv preprint arXiv:2407.18219 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2409.13156 , year=

    Rrm: Robust reward model training mitigates reward hacking , author=. arXiv preprint arXiv:2409.13156 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2409.12917 , year=

    Training language models to self-correct via reinforcement learning , author=. arXiv preprint arXiv:2409.12917 , year=

    Pith/arXiv arXiv

  39. [39]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  40. [40]

    arXiv preprint arXiv:2310.01798 , year=

    Large language models cannot self-correct reasoning yet , author=. arXiv preprint arXiv:2310.01798 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2310.03708 , year=

    Beyond one-preference-for-all: Multi-objective direct preference optimization , author=. arXiv preprint arXiv:2310.03708 , year=

    Pith/arXiv arXiv

  42. [42]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  43. [43]

    arXiv preprint arXiv:2403.19159 , year=

    Disentangling length from quality in direct preference optimization , author=. arXiv preprint arXiv:2403.19159 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2406.11817 , year=

    Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level , author=. arXiv preprint arXiv:2406.11817 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2403.04642 , year=

    Teaching large language models to reason with reinforcement learning , author=. arXiv preprint arXiv:2403.04642 , year=

    Pith/arXiv arXiv

  46. [46]

    Hashimoto , title =

    Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , month =

    2023

  47. [47]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Robust fine-tuning of zero-shot models , author=. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2022

  48. [48]

    arXiv preprint arXiv:2012.09816 , year=

    Towards understanding ensemble, knowledge distillation and self-distillation in deep learning , author=. arXiv preprint arXiv:2012.09816 , year=

    Pith/arXiv arXiv 2012

  49. [49]

    arXiv preprint arXiv:1806.03822 , year=

    Know what you don't know: Unanswerable questions for SQuAD , author=. arXiv preprint arXiv:1806.03822 , year=

    Pith/arXiv arXiv

  50. [50]

    Advances in Neural Information Processing Systems , volume=

    Experience replay for continual learning , author=. Advances in Neural Information Processing Systems , volume=

  51. [51]

    International Conference on Machine Learning , pages=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  52. [52]

    2024 , publisher =

    Entropy-Regularized Process Reward Model , author=. 2024 , publisher =

    2024

  53. [53]

    ACM Transactions on Information Systems (TOIS) , volume=

    Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2007 , publisher=

    2007

  54. [54]

    doi:10.57967/hf/0513 , publisher =

    Edward Beeching and Younes Belkada and Kashif Rasul and Lewis Tunstall and Leandro von Werra and Nazneen Rajani and Nathan Lambert , title =. doi:10.57967/hf/0513 , publisher =

    doi:10.57967/hf/0513

  55. [55]

    arXiv preprint arXiv:2408.15240 , year=

    Generative verifiers: Reward modeling as next-token prediction , author=. arXiv preprint arXiv:2408.15240 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2410.12832 , year=

    Generative Reward Models , author=. arXiv preprint arXiv:2410.12832 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2401.12187 , year=

    Warm: On the benefits of weight averaged reward models , author=. arXiv preprint arXiv:2401.12187 , year=

    Pith/arXiv arXiv

  58. [58]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Robust fine-tuning of zero-shot models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  59. [59]

    arXiv preprint arXiv:2407.13690 , year=

    DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving , author=. arXiv preprint arXiv:2407.13690 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2407.14622 , year=

    BOND: Aligning LLMs with Best-of-N Distillation , author=. arXiv preprint arXiv:2407.14622 , year=

    Pith/arXiv arXiv

  61. [61]

    2023 , eprint=

    UltraFeedback: Boosting Language Models with High-quality Feedback , author=. 2023 , eprint=

    2023

  62. [62]

    arXiv preprint arXiv:2402.07319 , year=

    Odin: Disentangled reward mitigates hacking in rlhf , author=. arXiv preprint arXiv:2402.07319 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  64. [64]

    2023 , publisher=

    Mathematical analysis of machine learning algorithms , author=. 2023 , publisher=

    2023

  65. [65]

    2024 , journal =

    "Alignment Guidebook , author=. 2024 , journal =

    2024

  66. [66]

    arXiv preprint arXiv:2305.11206 , year=

    Lima: Less is more for alignment , author=. arXiv preprint arXiv:2305.11206 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2309.17452 , year=

    Tora: A tool-integrated reasoning agent for mathematical problem solving , author=. arXiv preprint arXiv:2309.17452 , year=

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint arXiv:2404.02078 , year=

    Advancing llm reasoning generalists with preference trees , author=. arXiv preprint arXiv:2404.02078 , year=

    Pith/arXiv arXiv

  69. [69]

    arXiv preprint arXiv:2308.09583 , year=

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct , author=. arXiv preprint arXiv:2308.09583 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:2404.19733 , year=

    Iterative reasoning preference optimization , author=. arXiv preprint arXiv:2404.19733 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2405.03548 , year=

    Mammoth2: Scaling instructions from the web , author=. arXiv preprint arXiv:2405.03548 , year=

    Pith/arXiv arXiv

  73. [73]

    arXiv preprint arXiv:2404.16767 , year=

    REBEL: Reinforcement Learning via Regressing Relative Rewards , author=. arXiv preprint arXiv:2404.16767 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2405.00451 , year=

    Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning , author=. arXiv preprint arXiv:2405.00451 , year=

    Pith/arXiv arXiv

  75. [75]

    Advances in Neural Information Processing Systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in Neural Information Processing Systems , volume=

  76. [76]

    arXiv preprint arXiv:2406.10858 , year=

    Step-level Value Preference Optimization for Mathematical Reasoning , author=. arXiv preprint arXiv:2406.10858 , year=

    Pith/arXiv arXiv

  77. [77]

    nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=

    2016

  78. [78]

    arXiv preprint arXiv:1312.5602 , year=

    Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=

    Pith/arXiv arXiv

  79. [79]

    arXiv preprint arXiv:2312.10003 , year=

    Rest meets react: Self-improvement for multi-step reasoning llm agent , author=. arXiv preprint arXiv:2312.10003 , year=

    Pith/arXiv arXiv

  80. [80]

    arXiv preprint arXiv:2402.10176 , year=

    OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset , author=. arXiv preprint arXiv:2402.10176 , year=

    Pith/arXiv arXiv

Showing first 80 references.