Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Coaching feedback beats scalar rewards for post-training language models on open-ended tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:57 UTC pith:BPTH4LXY

load-bearing objection Novel training loop, but the flagship claim is held hostage by top-3 checkpoint averaging, a contradictory cell, and an unreported cross-evaluator check. the 3 major comments →

arxiv 2607.18110 v1 pith:BPTH4LXY submitted 2026-07-20 cs.LG cs.CL

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Tianzhu Ye , Li Dong , Guanheng Chen , He Zhu , Xun Wu , Shaohan Huang , Furu Wei This is my paper
classification cs.LG cs.CL
keywords Experiential LearningLLM-as-a-Coachnon-verifiable taskson-policy context distillationreward hackingfeedback bandwidthrubric-based reinforcement learningopen-ended instruction following
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that, on tasks without a single correct answer, a language model can be trained more effectively from rich textual coaching than from a scalar score. It introduces Experiential Learning, which turns the feedback model into a coach that distills each response into transferable guidance, conditions a teacher on that guidance, and trains the policy to match the teacher's distribution through on-policy context distillation. The paper reports that this signal consistently outperforms rubric-based reinforcement learning on held-out and unseen benchmarks, generalizes better beyond the training distribution, and reduces reward hacking. A sympathetic reader would care because the method turns evaluation text that standard RL throws away into a dense, usable learning signal, and it works with both self-feedback and a stronger external feedback model.

Core claim

On open-ended tasks, response quality has no ground truth, so post-training usually asks a judge model to score responses against rubrics and optimizes only that score. The paper's central claim is that this scalar bottleneck is the problem, and that a better policy emerges when the same feedback model acts as a coach: it writes down transferable guidance about each on-policy response, a teacher model is conditioned on that guidance, and the policy is trained to match the teacher's distribution through token-level reverse-KL distillation. Under this objective the policy never receives a scalar reward; the learning signal is the full distributional shift induced by the coaching text. The pape

What carries the argument

The load-bearing mechanism is experiential knowledge plus on-policy context distillation. For each training prompt, the policy samples a response; a coach model evaluates it against rubrics and extracts a few sentences of reusable strategy. That text is prepended to the context of a teacher model, and the policy is trained by minimizing the token-level reverse KL divergence between its own distribution and the teacher's context-conditioned distribution, averaged over the policy's own samples. This turns dense textual feedback into dense per-token supervision. The paper estimates that this textual channel carries roughly 17,600 bits per sample, versus 3.3 bits for a discrete 1-10 score, and a

Load-bearing premise

The paper's comparisons rest on the assumption that the model used as evaluator is a fair, unbiased judge of response quality for both RL and EL outputs; the same model also serves as the coach in several EL runs, and the cross-check with a stronger independent evaluator is mentioned but not reported.

What would settle it

Have human raters, blinded to training method, compare EL and RL outputs on the same held-out and unseen tasks. Concretely, choose 200 prompts from the held-out set, generate one response from each method, and ask at least three human annotators to state a pairwise preference. If EL is not preferred at a rate at least matching its benchmark advantage, the central comparative claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, post-training pipelines for subjective and creative tasks can treat rubric-based evaluation as a source of dense supervision rather than a scalar loss.
  • Because EL generalizes better to unseen benchmarks, it offers a direct way to reduce reward hacking in domains without a verifier.
  • The coach can be the policy itself, so the method works even when no stronger model is available; iterative teacher updates add further gains when general-domain prompts are mixed in to prevent forgetting.
  • The distinction between judge and coach is about the learning signal, not the evaluator's capability, so any existing judge-style evaluator can be reused as a coach.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A control the paper does not run: prepend high-entropy but task-irrelevant text as the teacher context. If gains persist, bandwidth alone is not the active ingredient; the semantic content of the experience matters.
  • If human raters confirm the preference, the result supports a broader design principle: for multi-dimensional quality, match the teacher's distribution rather than maximize a proxy score.
  • The coach/student split suggests a division of labor that could let a strong coach model improve many small policy models through text alone, lowering the cost of post-training for less-resourced settings.
  • The bandwidth figure is a theoretical ceiling, not usable information; future work could measure the mutual information between experiential knowledge and target quality, and test how short the extracted knowledge can be before gains disappear.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Experiential Learning (EL), a post-training method for non-verifiable open-ended tasks. Instead of compressing an LLM judge's rubric-based evaluation into a scalar reward, EL repurposes the feedback model as an LLM-as-a-Coach that extracts textual 'experiential knowledge' from each on-policy response. This knowledge conditions a teacher model, and the policy is trained by on-policy token-level reverse-KL distillation toward the context-conditioned teacher. The authors evaluate EL against rubric-based GRPO on WildChat-IF held-out test and four unseen benchmarks, using Qwen3-8B and OLMo-3-7B with either the frozen policy or GPT-4o as feedback. They report that EL mostly outperforms RL, generalizes better beyond the training distribution, and mitigates reward hacking, supported by ablations on iterative teachers, teacher context formats, and a toy distribution-matching analysis.

Significance. If the central empirical claim holds, EL is a useful contribution: it provides a practical, bandwidth-rich alternative to scalar-reward RL for non-verifiable tasks, with a simple training pipeline and potential reduction of reward overoptimization. The paper also honestly hedges the bandwidth intuition and the toy analysis in footnotes. However, the headline claim of 'consistently outperforming' rubric-based RL is not yet backed by the reported measurement protocol. The top-3 checkpoint selection, the absence of variance or seeds, and the undisclosed 'stronger proprietary evaluator' cross-check are all load-bearing for a comparative claim. The central idea is plausible and worth publishing after the evidence is strengthened.

major comments (3)
  1. [§3.2, Table 2] The main results are averages over the 'top-3 performing checkpoints' with no selection rule, no seeds, and no standard deviations. Margins are often 0.3–1.1 points (e.g., WildChat Qwen3-8B+Qwen3-8B: RL 79.2 vs EL 80.0; OLMo-3-7B+GPT-4o ArenaHard-v2: 22.5 vs 25.1). Without error bars or a fixed checkpoint rule (e.g., last checkpoint), the reader cannot determine whether the reported ordering is an artifact of noisy training. In fact, one cell contradicts 'consistently outperforms': Qwen3-8B+GPT-4o on AlpacaEval v2 has RL 38.2 vs EL 37.4. Please report variance over seeds, all checkpoints, or a principled selection procedure, and qualify the generalization claim accordingly.
  2. [§3.1] The paper states: 'We additionally cross-checked the main conclusions with a stronger proprietary evaluator and observed consistent relative trends' — but provides no numbers, benchmark, protocol, or even which evaluator was used. This is the only evidence against the concern that GPT-4o, which also serves as coach in several EL runs, rewards outputs stylistically aligned with GPT-4o. Because the central claim is comparative across feedback models, this omitted cross-check is load-bearing. Please include the actual cross-check results, or otherwise provide an independent (e.g., non-GPT-4o) evaluation on at least one held-out and one unseen benchmark.
  3. [§3.3, Figure 3] The generalization claim is based on a single training-set subset of 1000 examples and one run, with no variance or significance test. The figure shows EL has smaller training-set gain but larger test-set gain; however, given the checkpoint-selection issue in §3.2, this could also be an artifact of choosing checkpoints that favor EL on test. To support 'EL generalizes better', provide checkpoint curves or multiple seeds showing that the cross-over is stable.
minor comments (4)
  1. [§2.4, footnote 2] The bandwidth comparison is clearly labeled as an upper-bound intuition, which is good. However, the footnote comes after the calculation; consider moving it before the 17,600-bit claim to avoid overstatement.
  2. [§3.4, Table 4 and Appendix A.2] The Multiple-Choice ablation lists 9 directives, but the text says '10 context choices' because the empty option is counted. Clarify this in the main paper, not just in the appendix.
  3. [Eq. (2)] The notation DKL(πθ(·|x, y<t) ∥ π_teacher(·|e, x, y<t)) is clear, but consider adding a brief remark that the teacher is also conditioned on previous student tokens, to avoid confusion with the teacher's unconditional distribution.
  4. [§1, Figure 1 caption] Figure 1 says 'over 5,000× the bandwidth of a scalar reward' and the text also mentions 'over 1,000×' relative to bf16. This is consistent, but the visual may mislead; consider labeling both comparisons explicitly.

Circularity Check

0 steps flagged

No significant circularity; the derivation is self-contained and the central claim is an empirical comparison against external benchmarks.

full rationale

The paper's central claim is an empirical comparison: EL (reverse-KL distillation toward a coach-conditioned teacher, Eq. 2) outperforms rubric-based RL (GRPO on a scalar reward, Eq. 1) on held-out and unseen open-ended tasks. No parameter in Eq. (2) is fitted to the benchmarks used for the comparison; the coach is either the frozen policy itself or GPT-4o, and the teacher is a policy checkpoint, with no fitted coefficients. The toy distribution-matching analysis in §3.5 explicitly labels the 'distributional guidance' arm as 'an idealized proxy for EL' and disclaims that it is 'not intended as a faithful simulation of the full RL and EL training pipelines, nor as a proof of a fundamental limitation shared by all scalar-reward methods' — this is a stated illustration rather than a load-bearing derivation, and its limitations are acknowledged in the text. The paper invokes several prior works by the same authors ([YDW+26] for on-policy context distillation, [YDD+26] as Part II, [MWZ+26, LL25] for OPD teachers), but the core method is defined directly in Eq. (2) and Algorithm 1, and the empirical advantage does not rest on the truth of those citations; they are implemented building blocks, not evidence for the headline result. The only load-bearing external validation is the claim that a 'stronger proprietary evaluator' showed 'consistent relative trends' (§3.1); this is an unreported external cross-check, which is a measurement-reporting weakness rather than circularity, since GPT-4o judging EL outputs trained using GPT-4o as coach is a possible confound, not a definitional reduction of the result into its inputs. No equation in the paper reduces to a fitted value, no prediction is derived from a fitted parameter, and no uniqueness argument is imported from self-citations. The comparison is externally benchmarked (AlpacaEval-v2, WildBench, ArenaHard, CreativeWritingV3), so the claim has independent grounding even if the evaluation protocol has statistical weaknesses.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claim rests on the assumption that textual experiential knowledge is a better training signal than scalar rewards, plus several domain assumptions about teacher quality, the reverse-KL surrogate, and the validity of GPT-4o as evaluator. Hyperparameters and checkpoint selection are chosen by hand. No new physical or mathematical entities are introduced.

free parameters (5)
  • Training hyperparameters (lr=1e-6, batch 256, 8 rollouts, 3 epochs) = lr=1e-6, batch=256, rollouts=8, epochs=3
    Chosen by hand; no sensitivity analysis is reported, so the RL/EL comparison could depend on these settings.
  • Reverse-KL top-256 token truncation = top 256 tokens, unnormalized
    Ad hoc approximation of the full token distribution; its effect on learning quality is not analyzed.
  • Tulu3 mixing ratio for iterative teacher = 1:0.25 WildChat:Tulu3
    Chosen to stabilize iterative teacher updates; no sweep is reported.
  • Bandwidth example context length and vocabulary size = L=1024, |V|=150,000
    Assumed numbers used to compute the 17,600-bit figure; not measured from actual experiential knowledge contexts.
  • Checkpoint selection = top-3 by evaluation score
    Reporting choice that can inflate point estimates and hide variance; not a fixed protocol.
axioms (5)
  • domain assumption GPT-4o rubric-conditioned judgments are a valid, unbiased measure of response quality for both RL and EL outputs.
    Used as the evaluator for all main results and benchmarks; if EL outputs align with GPT-4o's stylistic preferences, the reported gains may not reflect general quality. Location: §3.1 Evaluation.
  • domain assumption Experiential knowledge extracted by the coach carries usable transferable supervision, and the teacher distribution π_teacher(·|e,x) is a good training target.
    This is the core premise of the method; no independent verification that the experience text is informative beyond the internal benchmark improvements. Location: §2.2, Eq. (2).
  • domain assumption The initial frozen policy checkpoint is an appropriate teacher for on-policy context distillation.
    Teacher and student share the same base; if the teacher is too weak, the method's ceiling is limited. Location: §2.2.
  • domain assumption Reverse KL over the top-256 tokens is a sufficient surrogate for full token-level divergence.
    Used in training; truncation may ignore important low-probability tokens. Location: §3.1 Training.
  • domain assumption WildChat-IF and Tulu3 SFT mixtures adequately represent non-verifiable open-ended instruction tasks.
    Training and generalization claims are relative to these task distributions. Location: §3.1.
invented entities (1)
  • Experiential knowledge no independent evidence
    purpose: A textual training signal distilled by an LLM-as-a-Coach from rubric-based evaluation, used to condition the teacher and provide dense supervision.
    The construct is operationalized only within this paper's pipeline and evaluated through internal benchmark improvements; there is no independent falsifiable handle outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 16413 in / 13849 out tokens · 140981 ms · 2026-08-01T15:57:44.635256+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

Figures

Figures reproduced from arXiv: 2607.18110 by Furu Wei, Guanheng Chen, He Zhu, Li Dong, Shaohan Huang, Tianzhu Ye, Xun Wu.

Figure 1
Figure 1. Figure 1: Comparison of the learning signals in RL and EL. RL compresses evaluation into a scalar [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of RL and EL. Reinforcement learning updates the policy by maximizing [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Score improvement over the base model. “WildChat Train” is evaluated on a subset of the train￾ing data after training completes. EL generalizes better than RL despite achieving lower training-set scores. We sample a subset of 1000 WildChat training examples and evaluate on it af￾ter training completes. We compare the results with those on WildChat test set and two benchmarks, as shown in Fig￾ure 3. Althoug… view at source ↗
Figure 4
Figure 4. Figure 4: Controlled distribution-matching analysis. Left: for a binary target, RL with binary re [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompt template for EL knowledge extraction. The text within the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template for conditioning the teacher model on experiential knowledge. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template for RL rubric-based scoring. The integer score extracted from the [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Full Critique ablation. The scoring prompt is the same as RL. The full textual output from M (including per-rubric analysis and score) is used as context for the teacher. Scoring prompt: None Context template: Here are quality rubrics that your response will be evaluated against: <rubrics> {rubric_list_string} </rubrics> Please solve the following problem, keeping the above evaluation criteria in mind: {pr… view at source ↗
Figure 9
Figure 9. Figure 9: Rubrics Only ablation. There is no scoring prompt. Only the rubrics are provided as context, with no response-specific feedback from M. knowledge is left empty. So there are 10 context choices in total. The context template is the same as default EL ( [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Multiple-Choice ablation. The scoring prompt appends a directive selection task. The selected directive is extracted from the <experience> tags and used as context with the same template as default EL. If M outputs none of required directives, the experiential knowledge is left empty. So there are 10 context choices in total. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Three rubric examples from the training set, covering an open-ended analytical ques [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Three examples of experiential knowledge extracted by the LLM-as-a-Coach. Each [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: End-to-end example of the EL pipeline. Given a prompt and rubrics, the policy generates [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Flux-OPD: On-Policy Distillation with Evolving Contexts

    cs.LG 2026-07 conditional novelty 6.0

    Evolving contexts improve open-ended on-policy distillation when injected as conflict-weighted corrections into a stable context-free teacher rather than used as direct targets.

Reference graph

Works this paper leans on

300 extracted references · 67 linked inside Pith · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2507.19457 , year=

    Gepa: Reflective prompt evolution can outperform reinforcement learning , author=. arXiv preprint arXiv:2507.19457 , year=

    Pith/arXiv arXiv

  2. [2]

    differentiation

    Textgrad: Automatic" differentiation" via text , author=. arXiv preprint arXiv:2406.07496 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2511.07919 , year=

    Feedback descent: Open-ended text optimization via pairwise comparison , author=. arXiv preprint arXiv:2511.07919 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2506.03106 , year=

    Critique-grpo: Advancing llm reasoning with natural language and numerical feedback , author=. arXiv preprint arXiv:2506.03106 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2603.10165 , year=

    Openclaw-rl: Train any agent simply by talking , author=. arXiv preprint arXiv:2603.10165 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2602.12125 , year=

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation , author=. arXiv preprint arXiv:2602.12125 , year=

    Pith/arXiv arXiv

  7. [7]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  8. [8]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  9. [9]

    arXiv preprint arXiv:2510.08558 , year=

    Agent learning via early experience , author=. arXiv preprint arXiv:2510.08558 , year=

    Pith/arXiv arXiv

  10. [10]

    Google AI , volume=

    Welcome to the era of experience , author=. Google AI , volume=

  11. [11]

    arXiv preprint arXiv:2603.16856 , year=

    Online experiential learning for language models , author=. arXiv preprint arXiv:2603.16856 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2606.11627 , year=

    When Context Returns: Toward Robust Internalization in On-Policy Distillation , author=. arXiv preprint arXiv:2606.11627 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2606.26790 , year=

    OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning , author=. arXiv preprint arXiv:2606.26790 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2605.10781 , year=

    Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled rlvr , author=. arXiv preprint arXiv:2605.10781 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2604.17535 , year=

    Opsdl: On-policy self-distillation for long-context language models , author=. arXiv preprint arXiv:2604.17535 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2604.24005 , year=

    Tcod: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents , author=. arXiv preprint arXiv:2604.24005 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2605.07725 , year=

    Sod: Step-wise on-policy distillation for small language model agents , author=. arXiv preprint arXiv:2605.07725 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2603.11178 , year=

    PACED: Distillation and on-policy self-distillation at the frontier of student competence , author=. arXiv preprint arXiv:2603.11178 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2603.05433 , year=

    Crisp: Compressed reasoning via iterative self-policy distillation , author=. arXiv preprint arXiv:2603.05433 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2604.01193 , year=

    Embarrassingly simple self-distillation improves code generation , author=. arXiv preprint arXiv:2604.01193 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2603.24472 , year=

    Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? , author=. arXiv preprint arXiv:2603.24472 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2604.02288 , year=

    Unifying group-relative and self-distillation policy optimization via sample routing , author=. arXiv preprint arXiv:2604.02288 , year=

    arXiv

  23. [23]

    arXiv preprint arXiv:2604.10674 , year=

    Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents , author=. arXiv preprint arXiv:2604.10674 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2604.03128 , year=

    Self-distilled rlvr , author=. arXiv preprint arXiv:2604.03128 , year=

    Pith/arXiv arXiv

  25. [25]

    ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning , year=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning , year=

    2026

  26. [26]

    Continual Adaptation at Scale: Towards Sustainable AI Workshop , year=

    Aligning Language Models from User Interactions , author=. Continual Adaptation at Scale: Towards Sustainable AI Workshop , year=

  27. [27]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    G-eval: NLG evaluation using gpt-4 with better human alignment , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  28. [28]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  29. [29]

    arXiv preprint arXiv:2606.30406 , year=

    MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training , author=. arXiv preprint arXiv:2606.30406 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2606.19327 , year=

    Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation , author=. arXiv preprint arXiv:2606.19327 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2606.12507 , year=

    Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers , author=. arXiv preprint arXiv:2606.12507 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2602.12275 , year=

    On-Policy Context Distillation for Language Models , author=. arXiv preprint arXiv:2602.12275 , year=

    Pith/arXiv arXiv

  33. [33]

    Advances in Neural Information Processing Systems , volume=

    Reward reasoning models , author=. Advances in Neural Information Processing Systems , volume=

  34. [34]

    arXiv preprint arXiv:2408.11791 , year=

    Critique-out-loud reward models , author=. arXiv preprint arXiv:2408.11791 , year=

    Pith/arXiv arXiv

  35. [35]

    Eq-bench creative writing benchmark v3 , author=

  36. [36]

    Forty-second International Conference on Machine Learning , year=

    From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline , author=. Forty-second International Conference on Machine Learning , year=

  37. [37]

    The Thirteenth International Conference on Learning Representations , year=

    WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild , author=. The Thirteenth International Conference on Learning Representations , year=

  38. [38]

    Alpacaeval: An automatic evaluator of instruction-following models , author=

  39. [39]

    NeurIPS 2025 Workshop on Efficient Reasoning , year=

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains , author=. NeurIPS 2025 Workshop on Efficient Reasoning , year=

    2025

  40. [40]

    arXiv preprint arXiv:2411.15124 , year=

    Tulu 3: Pushing frontiers in open language model post-training , author=. arXiv preprint arXiv:2411.15124 , year=

    Pith/arXiv arXiv

  41. [41]

    The Twelfth International Conference on Learning Representations , year=

    WildChat: 1M ChatGPT Interaction Logs in the Wild , author=. The Twelfth International Conference on Learning Representations , year=

  42. [42]

    arXiv preprint arXiv:2512.13961 , year=

    Olmo 3 , author=. arXiv preprint arXiv:2512.13961 , year=

    Pith/arXiv arXiv

  43. [43]

    First Workshop on Foundations of Reasoning in Language Models , year=

    Language Models That Think, Chat Better , author=. First Workshop on Foundations of Reasoning in Language Models , year=

  44. [44]

    arXiv preprint arXiv:2603.04597 , year=

    Bootstrapping exploration with group-level natural language feedback in reinforcement learning , author=. arXiv preprint arXiv:2603.04597 , year=

    arXiv

  45. [45]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Infiniteicl: Breaking the limit of context window size via long short-term memory transformation , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  46. [46]

    Complex & Intelligent Systems , volume=

    ETHOS: a multi-label hate speech detection dataset , author=. Complex & Intelligent Systems , volume=. 2022 , publisher=

    2022

  47. [47]

    A Survey on In-context Learning

    Dong, Qingxiu and Li, Lei and Dai, Damai and Zheng, Ce and Ma, Jingyuan and Li, Rui and Xia, Heming and Xu, Jingjing and Wu, Zhiyong and Chang, Baobao and Sun, Xu and Li, Lei and Sui, Zhifang. A Survey on In-context Learning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.64

    doi:10.18653/v1/2024.emnlp-main.64 2024

  48. [48]

    HateCheck: Functional tests for hate speech detection models , author=. Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers) , pages=

  49. [49]

    Findings of the association for computational linguistics: EMNLP 2020 , pages=

    TweetEval: Unified benchmark and comparative evaluation for tweet classification , author=. Findings of the association for computational linguistics: EMNLP 2020 , pages=

    2020

  50. [50]

    Conference on health, inference, and learning , pages=

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering , author=. Conference on health, inference, and learning , pages=. 2022 , organization=

    2022

  51. [51]

    arXiv preprint arXiv:2505.09666 , year=

    System Prompt Optimization with Meta-Learning , author=. arXiv preprint arXiv:2505.09666 , year=

    arXiv

  52. [52]

    arXiv preprint arXiv:2510.18874 , year=

    Retaining by doing: The role of on-policy data in mitigating forgetting , author=. arXiv preprint arXiv:2510.18874 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2509.04259 , year=

    Rl's razor: Why online reinforcement learning forgets less , author=. arXiv preprint arXiv:2509.04259 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2509.25052 , year=

    Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planning , author=. arXiv preprint arXiv:2509.25052 , year=

    arXiv

  57. [57]

    arXiv preprint arXiv:2504.11442 , year=

    TextArena , author=. arXiv preprint arXiv:2504.11442 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2510.01857 , year=

    Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning , author=. arXiv preprint arXiv:2510.01857 , year=

    Pith/arXiv arXiv

  59. [59]

    Advances in neural information processing systems , volume=

    Generative adversarial imitation learning , author=. Advances in neural information processing systems , volume=

  60. [60]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  61. [61]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv preprint arXiv:2502.03387 , year=

    Limo: Less is more for reasoning , author=. arXiv preprint arXiv:2502.03387 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv preprint arXiv:2506.04178 , year=

    OpenThoughts: Data Recipes for Reasoning Models , author=. arXiv preprint arXiv:2506.04178 , year=

    Pith/arXiv arXiv

  64. [64]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    s1: Simple test-time scaling , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  65. [65]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  66. [66]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  67. [67]

    1998 , publisher=

    Reinforcement learning: An introduction , author=. 1998 , publisher=

    1998

  68. [68]

    arXiv preprint arXiv:2508.05629 , year=

    On the generalization of sft: A reinforcement learning perspective with reward rectification , author=. arXiv preprint arXiv:2508.05629 , year=

    arXiv

  69. [69]

    arXiv e-prints , pages=

    Knowledge distillation of black-box large language models , author=. arXiv e-prints , pages=

  70. [70]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  72. [72]

    The twelfth international conference on learning representations , year=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. The twelfth international conference on learning representations , year=

  73. [73]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  74. [74]

    the method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. the method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    1952

  75. [75]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Seqgan: Sequence generative adversarial nets with policy gradient , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  76. [76]

    Advances in neural information processing systems , volume=

    Generative adversarial nets , author=. Advances in neural information processing systems , volume=

  77. [77]

    arXiv preprint arXiv:2501.17161 , year=

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training , author=. arXiv preprint arXiv:2501.17161 , year=

    Pith/arXiv arXiv

  78. [78]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  79. [79]

    An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Li Chengyuan and Dayiheng Liu and Fei Huang and Haoran Wei and others , year=

  80. [80]

    Introducing GPT-5 , url =

Showing first 80 references.