Pith. sign in

REVIEW 2 major objections 5 minor 300 references

Multi-turn on-policy distillation can match online accuracy without live environment calls by replaying teacher prefixes under a reliability-aware step-decay schedule.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 13:52 UTC pith:PCUFW5E7

load-bearing objection Clean offline multi-turn OPD recipe with a real two-sided shift analysis; fixed-κ is a soft spot the paper already owns, not a load-bearing flaw. the 2 major comments →

arxiv 2607.04763 v1 pith:PCUFW5E7 submitted 2026-07-06 cs.LG cs.AIcs.CLstat.ML

Multi-Turn On-Policy Distillation with Prefix Replay

Baohao Liao , Hanze Dong , Christof Monz , Xinxing Xu , Li Dong , Furu Wei This is my paper
classification cs.LG cs.AIcs.CLstat.ML
keywords on-policy distillationmulti-turn agentsprefix replayknowledge distillationLLM tool usedistribution shiftagent training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tackles how to distill a stronger teacher LLM agent into a student when the agent must interact with tools over many turns. Fully online on-policy distillation is expensive: every update needs fresh student rollouts through the environment plus teacher queries at those histories. The authors identify a prefix trap in this multi-turn setting: making histories more student-like improves relevance, but can query the teacher where its target is unreliable, creating a two-sided shift between student occupancy and teacher reliability. Their method, ReOPD, reuses pre-collected teacher trajectories as fixed prefixes, lets the student act only at the supervised step, and samples those steps with a simple decay that favors early, lower-shift prefixes. Across math-with-Python and search environments and several model scales, ReOPD preserves or improves online OPD accuracy, uses zero tool calls during student training, and is at least four times faster per step, turning expensive agent-environment interaction into a reusable offline resource.

Core claim

Multi-turn on-policy distillation is not automatically best when histories are fully student-generated. The gap to an ideal interactive objective decomposes into a student-occupancy mismatch and a teacher-reliability error, so the right training distribution is a reliability-aware bridge between student and teacher occupancies. ReOPD realizes that bridge off-environment by replaying teacher-forced prefixes and a step-decay schedule that emphasizes early prefixes, matching or beating online OPD while removing environment interaction during student training.

What carries the argument

The two-sided decomposition bound and the geometric bridge between student and teacher history occupancies, implemented by the one-parameter step-decay schedule that reallocates sampling mass toward early, low-shift prefixes.

Load-bearing premise

A single fixed step-decay schedule is a good enough stand-in for the exact per-history student-to-teacher likelihood ratio that balances relevance against teacher reliability.

What would settle it

On the same math and search setups, if fully online student-on-policy OPD clearly beat ReOPD even under a large teacher-student gap, or if uniform sampling over all prefix depths matched ReOPD's gains, the claim that reliability-aware early-prefix emphasis is the decisive mechanism would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Teacher RL rollouts become a free, reusable offline pool for student distillation with no extra collection cost.
  • A single student can be trained jointly across heterogeneous environments without keeping all tools online.
  • When the teacher-student gap is large, teacher-anchored prefixes with step decay outperform fully student-on-policy roll-ins.
  • Student training cost no longer scales with live environment interaction or tool calls.
  • Distillation pipelines can collect per-environment teacher traces separately and merge them into one offline pool.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reliability-aware prefix design could transfer to other multi-turn imitation settings, such as process supervision or critique models, not only teacher distillation.
  • Adaptive per-trajectory reliability scores, rather than a fixed step index, may help when trajectories vary widely in length or difficulty.
  • Shared offline multi-environment agent pools could become a standard resource analogous to static supervised fine-tuning corpora.
  • Step index as a proxy may break in environments where early mistakes are rare but late branching is extreme; measuring actual likelihood ratios online would test that limit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies multi-turn on-policy distillation (OPD) for agentic LLM tasks and proposes Replayed-Prefix On-Policy Distillation (ReOPD). Instead of rolling the student through a live environment at every update, ReOPD reuses a fixed pool of teacher trajectories as prefixes: the student acts only at selected steps while the teacher supplies dense per-step targets, with no environment calls during student training. The authors identify a “prefix trap” arising from a two-sided distribution shift (student occupancy vs. teacher reliability), formalize it via a TV-based bound (Proposition 1), and motivate a geometric bridge between student and teacher occupancies. In practice they implement reliability-aware design with a simple step-decaying sampling schedule ω(t)=κ^t that favors early, lower-shift prefixes. Across math-with-Python and search environments, multiple teacher/student scales, and a joint multi-environment setting, ReOPD matches or improves OPD accuracy, uses zero tool calls in student training, and is reported ≥4× faster per step.

Significance. If the results hold, the contribution is both practical and conceptual. Practically, ReOPD turns expensive multi-turn agent–environment interaction into a reusable offline resource (including free reuse of the teacher’s own RL rollouts), which matters for multi-tool and multi-environment distillation where keeping every environment online is operationally costly. Conceptually, framing multi-turn OPD as reliability-aware prefix distribution design—rather than “always make it fully student-on-policy”—is a useful corrective, and the two-regime empirical pattern (gains when the teacher–student gap is large; ties when the teacher stays reliable) is consistent with that view. Strengths include a clean decomposition (Prop. 1 under a mild bounded-loss assumption), controlled ablations (chunk index, κ, prefix source, RL vs. stationary pool), and evaluation against a strong online OPD baseline with the same teacher target on standard external benchmarks.

major comments (2)
  1. §4, Eqs. (7)–(8) and Fig. 4: The central algorithmic claim is that a position-only step-decay is a faithful, practical realization of the geometric bridge weight. Fig. 4 shows that log br_t decays with depth, but the manuscript does not quantify residual within-step variance of the exact ratio (e.g., R² of step index vs. log br_t, or an ablation of exact per-history weight (7) vs. ω(t)=κ^t). Without that, it remains unclear how much of the reliability term in Prop. 1 is controlled by reallocating mass only across steps. A short quantification or one ablation comparing exact vs. step-decay would make the “reliability-aware design” claim load-bearing rather than motivational.
  2. §4 (bridge-to-schedule map) and §5.2 / Fig. 5b: The theory predicts that the preferred decay steepness should track the per-step teacher–student gap ¯c (and thus the teacher–student capability gap), yet all main tables use a single fixed κ=0.6. Fig. 5b varies κ for one setting and supports moderate decay, but does not test whether the map (8) is predictive across the math vs. search and 4B/8B/30B regimes already reported in Tables 3–4. Reporting estimated ¯c (or predicted κ) per setting, or a small gap-adaptive schedule experiment, would substantially strengthen the two-regime interpretation rather than leaving κ as an empirically fixed free parameter.
minor comments (5)
  1. Fig. 1 and the efficiency claims: state hardware, batching, and whether teacher-query cost is included in the OPD timing, so the ≥4× figure is reproducible.
  2. Notation: α_t is introduced as uniform and then largely sidelined in favor of w_t; a one-sentence reminder in §4 that all reliability signal is carried by w_t would reduce confusion with the ideal objective (1).
  3. Table 5 caption is dense; a short note that math and search GRPO columns are different domain-specific teachers (already in text) would help when reading the table alone.
  4. Related work: the dual-exposure discussion citing Wang et al. (2026) is apt; a sentence distinguishing ReOPD’s offline prefix design from online dual-bias mitigations would clarify novelty.
  5. Typos / polish: occasional long sentences in §3–4 (e.g., the paragraph after Eq. (8)) could be split; “condition” vs. “conditional” is used inconsistently for π_T(·|x,h_t).

Circularity Check

0 steps flagged

No significant circularity: the two-sided bound, geometric bridge, and step-decay surrogate are independent of the reported accuracy numbers, which are measured on external benchmarks against a live OPD baseline.

full rationale

The paper's derivation chain is self-contained and does not reduce claimed results to their inputs by construction. Proposition 1 is a standard total-variation plus triangle-inequality decomposition of the gap between the ideal interactive objective R* and the replayed objective L_ρ; it does not encode the final accuracy claims. The geometric bridge (Eqs. 5–6) is the usual KL-interpolation solution between student and teacher occupancies. The exact density-ratio weight (Eq. 7) is then replaced by a one-parameter step-decay surrogate ω(t)=κ^t motivated by the empirical depth decay of the likelihood ratio (Fig. 4 and Eq. 8); κ is a free hyperparameter held fixed at 0.6 across tasks, not fitted to the evaluation metrics and then re-presented as a prediction. Algorithm 1 and the sampling implementation simply realize that schedule. Empirical claims (Tables 3–5, Fig. 1) compare ReOPD to online OPD, SFT, and cold-start on external competition and QA benchmarks (AIME, MATH500, NQ, HotpotQA, etc.) with the same teacher target; ablations (Figs. 5–6, Table 6) further isolate prefix source, decay steepness, and pool stationarity. Related-work self-citations (e.g., Liao et al. 2026, Dong et al.) are not load-bearing for the bound, the bridge, or the accuracy results. The paper itself flags the step-decay as a coarse first-order proxy (Section 6), which is a limitation, not circularity. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or ansatz-via-self-citation was found.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on standard KL distillation, a mild bounded-loss assumption for the TV bound, the modeling choice that step index proxies the student–teacher likelihood ratio, and one free schedule parameter κ. No new physical entities are postulated; the 'prefix trap' and 'two-sided shift' are named analyses of existing occupancy notions.

free parameters (1)
  • step-decay base κ = 0.6
    Single steepness knob of the sampling schedule ω(t)=κ^t; fixed at 0.6 for all main experiments and shown to matter in the κ sweep (Fig. 5b).
axioms (3)
  • domain assumption Bounded improvement loss: |ℓ(πθ, q⋆)| ≤ B for some finite B (Assumption 1).
    Used to convert occupancy TV distance into the first term of the two-sided bound (Prop. 1); holds for TV/JS and for KL under a positive lower bound on the ideal target.
  • ad hoc to paper Teacher support (or depth) is a practical surrogate for the unobserved ideal-target reliability error εθ_T,t.
    Stated in §3 after Prop. 1; justifies replacing the reliability term by a geometric bridge toward d_T and then by step decay.
  • ad hoc to paper Per-step teacher–student gap is approximately stationary across positions, so the accumulated ratio is geometric in t.
    Eq. (8) and the paragraph preceding it; licenses collapsing the exact weight into the one-parameter schedule κ^t.
invented entities (2)
  • prefix trap (temporal + two-sided distributional layers) independent evidence
    purpose: Name the failure mode of multi-turn OPD that motivates reliability-aware prefix design.
    Introduced in the introduction and §3; analytical framing rather than a new physical object; independent evidence is the empirical regime split (math vs search).
  • ReOPD (replayed-prefix on-policy distillation) independent evidence
    purpose: Concrete offline algorithm realizing the reliability-aware schedule.
    The method itself; falsifiable via the reported accuracy/speed/tool-call metrics.

pith-pipeline@v1.1.0-grok45 · 30615 in / 2919 out tokens · 27881 ms · 2026-07-11T13:52:30.034096+00:00 · methodology

0 comments
read the original abstract

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4$\times$ faster per training step than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.

Figures

Figures reproduced from arXiv: 2607.04763 by Baohao Liao, Christof Monz, Furu Wei, HanZe Dong, Li Dong, Xinxing Xu.

Figure 1
Figure 1. Figure 1: ReOPD keeps the benefits of on-policy distillation while removing environment in￾teraction. ReOPD matches or improves OPD accuracy, but trains much faster per step and eliminates tool calls during training by replaying teacher-recorded prefixes instead of executing fresh environment rollouts. The student/teacher models are Qwen3-4B-Instruct-2507 and Qwen3-8B, respectively. ∗Equal Contribution. †Work done d… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison between OPD and ReOPD for agentic tasks. Up: OPD with online environ￾ment. The environment is always alive during training. And all steps equally contribute to the loss. Down: ReOPD with offline environment. The environment is only alive for collecting the teacher’s trajectories, which can happen during the training of the teacher agent by using RL, like GRPO. Afterwards, the envi￾ronment is not… view at source ↗
Figure 3
Figure 3. Figure 3: Training a shared student agent on multiple heterogeneous environments with OPD and ReOPD. Left: With an increasing number of environments, the operational complexity grows for OPD due to the heavy deployment of the environments. Right: ReOPD doesn’t require the deployment of all environments at the same time. The teacher’s trajectories can be collected separately for different environments, and then be me… view at source ↗
Figure 4
Figure 4. Figure 4: Step index is a strong proxy for the likelihood-ratio weight. For each supervised position we compute the per-position weight rbt = Q s<t πθold (as)/πT (as) – the student-to-teacher likelihood ratio along the teacher prefix – and plot it against the step index t. The empirical weight decays with depth, because each factor compares the student’s and teacher’s probability of the teacher’s own recorded action… view at source ↗
Figure 5
Figure 5. Figure 5: Favoring early, low-shift steps improves distillation. Both panels use ReOPD’s sampling implementation and put more training mass on early interaction steps, where the student stays close to the teacher and the two-sided shift is small. (a) Sampling by chunk: drawing supervised positions from early chunks yields the best student, and pushing mass to later chunks degrades it. (b) Sampling with decay κ: appl… view at source ↗
Figure 6
Figure 6. Figure 6: Prefix source: the teacher’s own model gives the best prefixes, not a stronger gener￾ator. Teacher Qwen3-8B, student Qwen3-4B-Instruct-2507; we fix the distillation target to the teacher and vary only the prefix generator. Performance peaks when prefixes come from the teacher itself and degrades with larger or stronger generators – a direct test of the teacher-reliability view: the teacher’s conditional is… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

300 extracted references · 162 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2212.03533 , year=

    Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year=

    Pith/arXiv arXiv

  2. [2]

    Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

    Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages=

    2020

  3. [3]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  4. [4]

    Transactions of the Association for Computational Linguistics , volume=

    Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  5. [5]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Solving quantitative reasoning problems with language models , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Hugging Face repository , volume=

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions , author=. Hugging Face repository , volume=

  8. [8]

    arXiv preprint arXiv:2504.11536 , year=

    Retool: Reinforcement learning for strategic tool use in llms , author=. arXiv preprint arXiv:2504.11536 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  10. [10]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  11. [11]

    Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=

    Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps , author=. Proceedings of the 28th International Conference on Computational Linguistics (COLING) , year=

  12. [12]

    Transactions of the Association for Computational Linguistics (TACL) , volume=

    MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics (TACL) , volume=

  13. [13]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  14. [14]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , year=

    Measuring and Narrowing the Compositionality Gap in Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , year=

    2023

  15. [15]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  16. [16]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  17. [17]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  18. [18]

    M. J. Kearns , title =

  19. [19]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  20. [20]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  21. [21]

    Suppressed for Anonymity , author=

  22. [22]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  23. [23]

    arXiv preprint arXiv:2501.07301 , year=

    The lessons of developing process reward models in mathematical reasoning , author=. arXiv preprint arXiv:2501.07301 , year=

    Pith/arXiv arXiv

  24. [24]

    GitHub repository , howpublished =

    Wei Xiong and Hanning Zhang and Nan Jiang and Tong Zhang , title =. GitHub repository , howpublished =. 2024 , publisher =

    2024

  25. [25]

    arXiv preprint arXiv:2412.06559 , year=

    Processbench: Identifying process errors in mathematical reasoning , author=. arXiv preprint arXiv:2412.06559 , year=

    Pith/arXiv arXiv

  26. [26]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  27. [27]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  28. [28]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  29. [29]

    Advances in Neural Information Processing Systems , volume=

    Self-refine: Iterative refinement with self-feedback , author=. Advances in Neural Information Processing Systems , volume=

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    Advances in Neural Information Processing Systems , volume=

    Language models can solve computer tasks , author=. Advances in Neural Information Processing Systems , volume=

  32. [32]

    Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning

    Li, Ming and Chen, Lichang and Chen, Jiuhai and He, Shwai and Gu, Jiuxiang and Zhou, Tianyi. Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning. Findings of the Association for Computational Linguistics ACL 2024. 2024

    2024

  33. [33]

    arXiv preprint arXiv:2211.00053 , year=

    Generating sequences by learning to self-correct , author=. arXiv preprint arXiv:2211.00053 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2203.11171 , year=

    Self-consistency improves chain of thought reasoning in language models , author=. arXiv preprint arXiv:2203.11171 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2407.18219 , year=

    Recursive introspection: Teaching language model agents how to self-improve , author=. arXiv preprint arXiv:2407.18219 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2409.13156 , year=

    Rrm: Robust reward model training mitigates reward hacking , author=. arXiv preprint arXiv:2409.13156 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2409.12917 , year=

    Training language models to self-correct via reinforcement learning , author=. arXiv preprint arXiv:2409.12917 , year=

    Pith/arXiv arXiv

  38. [38]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  39. [39]

    arXiv preprint arXiv:2310.01798 , year=

    Large language models cannot self-correct reasoning yet , author=. arXiv preprint arXiv:2310.01798 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2310.03708 , year=

    Beyond one-preference-for-all: Multi-objective direct preference optimization , author=. arXiv preprint arXiv:2310.03708 , year=

    Pith/arXiv arXiv

  41. [41]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  42. [42]

    arXiv preprint arXiv:2403.19159 , year=

    Disentangling length from quality in direct preference optimization , author=. arXiv preprint arXiv:2403.19159 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2406.11817 , year=

    Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level , author=. arXiv preprint arXiv:2406.11817 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2403.04642 , year=

    Teaching large language models to reason with reinforcement learning , author=. arXiv preprint arXiv:2403.04642 , year=

    Pith/arXiv arXiv

  45. [45]

    Hashimoto , title =

    Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , month =

    2023

  46. [46]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Robust fine-tuning of zero-shot models , author=. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2022

  47. [47]

    arXiv preprint arXiv:2012.09816 , year=

    Towards understanding ensemble, knowledge distillation and self-distillation in deep learning , author=. arXiv preprint arXiv:2012.09816 , year=

    Pith/arXiv arXiv 2012

  48. [48]

    arXiv preprint arXiv:1806.03822 , year=

    Know what you don't know: Unanswerable questions for SQuAD , author=. arXiv preprint arXiv:1806.03822 , year=

    Pith/arXiv arXiv

  49. [49]

    Advances in Neural Information Processing Systems , volume=

    Experience replay for continual learning , author=. Advances in Neural Information Processing Systems , volume=

  50. [50]

    International Conference on Machine Learning , pages=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  51. [51]

    2024 , publisher =

    Entropy-Regularized Process Reward Model , author=. 2024 , publisher =

    2024

  52. [52]

    ACM Transactions on Information Systems (TOIS) , volume=

    Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search , author=. ACM Transactions on Information Systems (TOIS) , volume=. 2007 , publisher=

    2007

  53. [53]

    doi:10.57967/hf/0513 , publisher =

    Edward Beeching and Younes Belkada and Kashif Rasul and Lewis Tunstall and Leandro von Werra and Nazneen Rajani and Nathan Lambert , title =. doi:10.57967/hf/0513 , publisher =

    doi:10.57967/hf/0513

  54. [54]

    arXiv preprint arXiv:2408.15240 , year=

    Generative verifiers: Reward modeling as next-token prediction , author=. arXiv preprint arXiv:2408.15240 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2410.12832 , year=

    Generative Reward Models , author=. arXiv preprint arXiv:2410.12832 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2401.12187 , year=

    Warm: On the benefits of weight averaged reward models , author=. arXiv preprint arXiv:2401.12187 , year=

    Pith/arXiv arXiv

  57. [57]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Robust fine-tuning of zero-shot models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  58. [58]

    arXiv preprint arXiv:2407.13690 , year=

    DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving , author=. arXiv preprint arXiv:2407.13690 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2407.14622 , year=

    BOND: Aligning LLMs with Best-of-N Distillation , author=. arXiv preprint arXiv:2407.14622 , year=

    Pith/arXiv arXiv

  60. [60]

    2023 , eprint=

    UltraFeedback: Boosting Language Models with High-quality Feedback , author=. 2023 , eprint=

    2023

  61. [61]

    arXiv preprint arXiv:2402.07319 , year=

    Odin: Disentangled reward mitigates hacking in rlhf , author=. arXiv preprint arXiv:2402.07319 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  63. [63]

    2023 , publisher=

    Mathematical analysis of machine learning algorithms , author=. 2023 , publisher=

    2023

  64. [64]

    2024 , journal =

    "Alignment Guidebook , author=. 2024 , journal =

    2024

  65. [65]

    arXiv preprint arXiv:2305.11206 , year=

    Lima: Less is more for alignment , author=. arXiv preprint arXiv:2305.11206 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2309.17452 , year=

    Tora: A tool-integrated reasoning agent for mathematical problem solving , author=. arXiv preprint arXiv:2309.17452 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2404.02078 , year=

    Advancing llm reasoning generalists with preference trees , author=. arXiv preprint arXiv:2404.02078 , year=

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint arXiv:2308.09583 , year=

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct , author=. arXiv preprint arXiv:2308.09583 , year=

    Pith/arXiv arXiv

  69. [69]

    arXiv preprint arXiv:2404.19733 , year=

    Iterative reasoning preference optimization , author=. arXiv preprint arXiv:2404.19733 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:2305.20050 , year=

    Let's Verify Step by Step , author=. arXiv preprint arXiv:2305.20050 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2405.03548 , year=

    Mammoth2: Scaling instructions from the web , author=. arXiv preprint arXiv:2405.03548 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2404.16767 , year=

    REBEL: Reinforcement Learning via Regressing Relative Rewards , author=. arXiv preprint arXiv:2404.16767 , year=

    Pith/arXiv arXiv

  73. [73]

    arXiv preprint arXiv:2405.00451 , year=

    Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning , author=. arXiv preprint arXiv:2405.00451 , year=

    Pith/arXiv arXiv

  74. [74]

    Advances in Neural Information Processing Systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in Neural Information Processing Systems , volume=

  75. [75]

    arXiv preprint arXiv:2406.10858 , year=

    Step-level Value Preference Optimization for Mathematical Reasoning , author=. arXiv preprint arXiv:2406.10858 , year=

    Pith/arXiv arXiv

  76. [76]

    nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=

    2016

  77. [77]

    arXiv preprint arXiv:1312.5602 , year=

    Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=

    Pith/arXiv arXiv

  78. [78]

    arXiv preprint arXiv:2312.10003 , year=

    Rest meets react: Self-improvement for multi-step reasoning llm agent , author=. arXiv preprint arXiv:2312.10003 , year=

    Pith/arXiv arXiv

  79. [79]

    arXiv preprint arXiv:2402.10176 , year=

    OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset , author=. arXiv preprint arXiv:2402.10176 , year=

    Pith/arXiv arXiv

  80. [80]

    arXiv preprint arXiv:2410.01560 , year=

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data , author=. arXiv preprint arXiv:2410.01560 , year=

    Pith/arXiv arXiv

Showing first 80 references.