Pith. sign in

REVIEW 4 major objections 6 minor 40 references

When open-ended RL training prompts stop eliciting quality gaps among rollouts, a tutor LLM that appends atomic constraints restores a usable reward signal and beats adapting rubrics alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:21 UTC pith:6DYYGLOP

load-bearing objection Clean, usable idea—policy-aware append-only prompt adaptation for rubric RL—with solid ablations, but thin model coverage and a detector that is only indirectly validated. the 4 major comments →

arxiv 2607.04412 v1 pith:6DYYGLOP submitted 2026-07-05 cs.AI

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Yujin Kim , Namgyu Ho , Sangmin Hwang , Joonkee Kim , Yongjin Yang , Sangmin Bae , Seungone Kim , Jaehun Jung
show 2 more authors
Se-Young Yun Hwanjun Song
This is my paper
classification cs.AI
keywords non-verifiable RLLLM-as-a-judgeprompt adaptationrubric-based rewardsinstruction followingpolicy-aware curriculumpairwise comparisonatomic constraints
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

In non-verifiable instruction following, reinforcement learning depends on an LLM judge with prompt-specific rubrics, but that judge can only discriminate if the prompt actually produces rollouts of different quality. Fixed prompt corpora often become too easy or too hard for the evolving policy, so every rollout scores alike and the advantage signal collapses. This paper argues that the missing fix is policy-aware prompt adaptation, not only rubric adaptation. It introduces LLM-as-a-Tutor: one model pairwise-compares two on-policy rollouts to detect non-challenging prompts, then appends a single atomic constraint and matching rubric criteria. The append-only design raises difficulty in lockstep with the policy without external schedules, and on three complex instruction-following benchmarks it outperforms both static-prompt training and prior methods that only adapt rubrics or rewrite prompts.

Core claim

The paper claims that policy–prompt mismatch is a precondition that limits rubric-equipped judges in non-verifiable RL, and that treating the LLM as a tutor—examiner plus constraint generator—fixes it. Detecting non-discriminative prompts via pairwise rollout comparison and appending atomic constraints produces a self-calibrating curriculum that tracks policy capability, yielding higher scores than policy-unaware baselines and than methods that adapt rubrics or fully rewrite prompts on FollowBench, AdvancedIF, and InfoBench.

What carries the argument

LLM-as-a-Tutor: a single model that (1) pairwise-judges two policy rollouts to decide whether a prompt is non-discriminative and (2) if so appends one atomic constraint plus matching rubric criteria, leaving seed text and base rubrics otherwise intact so difficulty rises monotonically while the source distribution is preserved.

Load-bearing premise

That two on-policy rollouts plus one pairwise tutor judgment are a reliable enough detector of whether a prompt still creates reward variance, and that the same tutor’s appended constraints restore useful variance without drifting the original task.

What would settle it

Train the same policy with the same tutor on the same seed set but force constraint appends on Always, Random, or Wrong-model rollouts, or swap Append for full Rewrite; if those variants match or beat targeted append on the three benchmarks and if non-challenging prompts no longer show higher mean and lower reward variance, the claimed mechanism fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Prompt adaptation is a separate axis of policy-awareness from rubric adaptation; when rollouts do not vary in quality, no rubric refinement recovers a discriminative signal.
  • Append-only atomic constraints give a per-prompt curriculum that escalates with policy strength without hand-designed difficulty schedules.
  • Pairwise saturation checks align with empirical difficulty: non-challenging prompts show higher mean reward and lower variance than challenging ones.
  • As policy scale grows, the fraction of prompts the tutor augments rises, automatically allocating harder training signal to stronger models.
  • The same examiner–generator pattern can generalize to other domains by swapping the append template (extra reasoning steps, edge cases, tighter sourcing) while keeping the saturation check.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If discrimination is easier for LLMs than generation, tutor-shaped RL may outscale pure teacher-response distillation because the student’s ceiling tracks the tutor’s comparative skill rather than its answer quality.
  • Saturation detection could be reused as an online data filter even without generation—dropping or delaying easy prompts—to cut wasted rollouts in any rubric-based RL stack.
  • The Always/Random underperformance suggests many seed corpora already contain a large fraction of currently non-informative prompts; corpus curation alone may leave substantial signal on the table once policies improve.
  • A natural next stress test is multi-family policies and tutors: if pairwise judgments degrade when tutor and policy differ in style, the method’s gains may be partly style-matching rather than pure difficulty matching.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that rubric-based non-verifiable RL is limited by policy–prompt mismatch: static training prompts often fail to induce quality variance among rollouts, so group-relative advantages (Eq. 3) collapse regardless of rubric quality. It proposes LLM-as-a-Tutor, in which a single LLM acts as an examiner that pairwise-compares two on-policy rollouts to flag non-discriminative prompts and as a generator that appends an atomic constraint plus matching rubric criteria. The append-only design is claimed to raise difficulty monotonically with policy capability without external curricula. On FollowBench, AdvancedIF, and InfoBench, training Qwen3-1.7B with a Qwen3-8B tutor under GRPO yields the best average (51.96) versus fixed-prompt, policy-adaptive-rubric, Evol-Instruct, and EVA baselines, with ablations favoring adaptive targeting and append over rewrite/reset.

Significance. If the result holds, the paper cleanly identifies prompt adaptation as a missing axis of policy-awareness complementary to evolving rubrics, and supplies a practical, distribution-preserving mechanism (atomic appends triggered by pairwise saturation checks) that fits existing rubric-based GRPO stacks. Strengths include a controlled training stack, five-run means±SEM on three held-out benchmarks with an external API judge, targeted ablations (Always/Random/Wrong; Append vs Rewrite/Reset), and analyses linking tutor labels to reward mean/variance and base-rubric discriminativeness (Fig. 2). The framing is useful for non-verifiable instruction following and potentially transferable to other appendable difficulty notions. The contribution is incremental rather than foundational: gains over the strongest baselines are modest (~0.9 average points), and significance depends on whether the pairwise examiner is a faithful detector of the group-level variance that actually drives learning.

major comments (4)
  1. §3.2 and Eq. (3): The central claim that adaptation is policy-aware rests on detecting prompts for which group-relative advantages collapse. GRPO advantages use mean and σ over G=8 rollouts, but the examiner decides from only two samples and a binary pairwise judgment of “indistinguishable quality.” The manuscript never reports agreement between that two-sample decision and group-level σ(x) (precision/recall, calibration, false-append rate, or correlation with empirical reward variance). Fig. 2b only shows post-hoc aggregate mean/std separation of tutor-labeled buckets; that is consistent with a useful detector but does not establish that the examiner is a low-noise proxy for low-σ prompts. Without this calibration, the ~0.9-point average edge over policy-adaptive rubrics and EVA (Table 1) cannot be confidently attributed to accurate policy-aware selection rather than to co-adding constr
  2. Table 1 and §4.3: The headline “consistently outperforms” claim is overstated relative to the numbers. LLM-as-a-Tutor wins the average and five of six metrics, but on InfoBench DRFR it underperforms Policy-adaptive rubrics (73.59 vs 74.19) and is within error of several fixed-prompt baselines. The Adaptive–Random gap in Table 2 is also small (43.19 vs 42.82 on the authors’ three-benchmark average). The paper should either qualify the claim (best average; mixed per-benchmark) or provide statistical tests / more seeds so that the policy-aware prompt axis is not oversold relative to rubric adaptation alone.
  3. §4.1, Limitations (Appendix A), and Fig. 2a: Main results use a single policy–tutor pair (Qwen3-1.7B / Qwen3-8B) and one seed corpus (4K WildChat). Fig. 2a only sweeps constraint-addition rate with policy size, not end-to-end benchmark scores. Given that the method’s value proposition is self-calibration to the current policy, at least one additional policy size or model family with full evaluation is load-bearing for the generality claim. The Wrong ablation (Table 2) already shows that examining the wrong model’s rollouts hurts; that makes broader policy–tutor coverage more important, not less.
  4. §3.2 / Eq. (4) and reward construction: When a constraint is appended, new criteria Rc are unioned into the rubric and weights are renormalized. Adaptation therefore simultaneously changes the prompt distribution and the reward definition. Table 2’s Always/Random/Wrong variants share the same append-and-rubric-update mechanism, so they do not isolate “better targeting of non-discriminative prompts” from “extra rubric criteria on some fraction of prompts.” A cleaner control—append constraints but score only with the base rubric, or adapt prompts while freezing rubric structure—would make the causal role of the examiner clearer.
minor comments (6)
  1. Table 2 reports FB/AdvIF/IB averages that do not match the six-metric Average column of Table 1 (e.g., Adaptive 43.19 vs 51.96). Clarify that Table 2 uses a reduced metric set, or report the same metrics for comparability.
  2. §4.2 / Appendix D: EVA is re-implemented under GRPO with a rubric reward and Qwen3-8B evolver rather than DPO + scalar RM + gemini-1.5-pro. State more prominently in the main text that this is a controlled re-implementation, not a direct reproduction of published EVA numbers.
  3. Fig. 2c: Trends of base-rubric mean and std vs number of appended constraints are described as “tends to”; report sample sizes per bin and uncertainty so the amplification claim is quantifiable.
  4. §5.2 / Fig. 3 and Appendix F: Qualitative examples are helpful; note explicitly that some “successful” rollouts still hallucinate (e.g., originators in Fig. 4), so restored variance is not always restored correctness.
  5. Typos / polish: “EV A” spacing in the abstract and tables; “inducerollouts” / missing spaces in §1; arXiv id and “Preprint” framing are fine for review but should be cleaned for camera-ready.
  6. Appendix A cost discussion is useful; a short main-text sentence on tutor FLOPs relative to 8×N judge calls would help practitioners.

Circularity Check

0 steps flagged

Empirical methods paper with external-benchmark evaluation; no derivation reduces a claimed result to its inputs by construction.

full rationale

LLM-as-a-Tutor is an empirical RL methods paper: it proposes pairwise saturation detection plus append-only constraint generation, then measures success on held-out FollowBench, AdvancedIF, and InfoBench with an external gpt-5-mini judge (Tables 1–3, Fig. 2). There is no first-principles derivation, no free parameter fitted to a subset and re-presented as a prediction, no uniqueness theorem imported from the same authors, and no load-bearing self-citation chain that forces the central claim. Tutor decisions are checked against independent group reward mean/std (Fig. 2b), and ablations (Always/Random/Wrong, Append vs Rewrite/Reset) compare alternative procedures rather than redefining the metric as the training objective. The usual train-with-LLM-judge setup is not circularity under the stated criteria: evaluation is on external corpora and a different judge API. Skeptical concerns about two-sample calibration vs G=8 σ(x) are validity/correctness issues, not circular reductions. Score 0; steps empty.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Load-bearing content is methodological and empirical, not a formal derivation. The claim rests on standard RL optimization assumptions, the domain premise that LLM pairwise judgments track rollout discriminativeness, and design choices (append-only atomic constraints, adaptation every epoch, specific model pair) rather than newly postulated physical entities. Free parameters are training/adaptation knobs; invented entities are named framework constructs without independent existence outside the method.

free parameters (4)
  • Tutor adaptation interval = 1 epoch
    Set to one full epoch for the main runs; controls how often prompts can escalate and is not derived from theory.
  • GRPO group size G and other RL hyperparameters = G=8; LR=5e-6; KL=0.001; 3 epochs
    Rollouts per prompt (8), LR 5e-6, clip 0.2, KL 0.001, batch sizes, temperature 0.6, etc. (Table 4) are chosen by hand and affect whether reward variance is observed and optimized.
  • Policy–tutor size pairing = 1.7B policy / 8B tutor
    Main results fix Qwen3-1.7B policy with Qwen3-8B tutor/judge; Fig. 2a shows constraint rate depends strongly on policy scale, so reported gains are conditioned on this pairing.
  • Fraction of prompts modified (emergent) = ~28% average
    Average ~28% of prompts receive constraints under the adaptive trigger; used as the match rate for the Random ablation, not a free fit to benchmark scores, but still a run-dependent operating point.
axioms (5)
  • domain assumption Group-relative GRPO advantages require within-group reward spread to provide a learning signal (Eqs. 2–3).
    Standard RL premise restated in §3.1; motivates the whole method but is not re-proved.
  • domain assumption LLM pairwise quality comparison is sharper and more human-aligned than pointwise scoring for detecting non-discriminative prompts.
    Invoked in §1 and §3.2 citing prior judge literature; the paper’s detector quality inherits this assumption.
  • domain assumption In complex instruction following, difficulty is well operationalized by the number of simultaneous requirements, so append-only constraints monotonically raise difficulty while preserving the seed distribution.
    Stated in abstract/§3.2 and tied to FollowBench-style difficulty; underpins preferring append over rewrite.
  • domain assumption Instance-specific rubric scores from an LLM judge are a valid continuous reward for non-verifiable instruction following.
    Shared with RLCF/Rubrics-as-Rewards/DR Tulu; baseline and method both depend on it (§2.2, §3.1).
  • standard math Standard GRPO/PPO clipped surrogate optimization with KL to a reference policy is a valid policy improvement procedure under the constructed rewards.
    Uses established policy-gradient machinery (Schulman PPO; Shao GRPO) without new theory.
invented entities (2)
  • LLM-as-a-Tutor (examiner + generator dual role) no independent evidence
    purpose: Names the combined pairwise saturation detector and atomic constraint generator that adapts prompts online.
    Framework label for the method; no claim of a new natural phenomenon, only a training-time agent role.
  • Atomic constraint (append-only difficulty unit) no independent evidence
    purpose: Single additive requirement used as the monotone difficulty step co-updated into the rubric.
    Design construct chosen to match instruction-following difficulty; validated only via training outcomes and qualitative examples.

pith-pipeline@v1.1.0-grok45 · 27553 in / 3719 out tokens · 44667 ms · 2026-07-11T19:21:28.536092+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL." pith.science (2026). https://pith.science/paper/6DYYGLOP

@misc{pith2026260704412,
  author       = {Pith},
  title        = {Pith review of: LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DYYGLOP}},
  note         = {Machine review of arXiv:2607.04412}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

Figures

Figures reproduced from arXiv: 2607.04412 by Hwanjun Song, Jaehun Jung, Joonkee Kim, Namgyu Ho, Sangmin Bae, Sangmin Hwang, Seungone Kim, Se-Young Yun, Yongjin Yang, Yujin Kim.

Figure 1
Figure 1. Figure 1: Overview of LLM-as-a-Tutor: when the policy’s answers to a prompt are indistin￾guishable in quality, a tutor LLM adds a constraint to make the prompt more challenging. (Left) Static RL training corpora contain prompts that are non-challenging for the current policy and provide little learning signal. (Right) The tutor examines a pair of policy rollouts; if their quality is indistinguishable, it appends a c… view at source ↗
Figure 2
Figure 2. Figure 2: Analysis of the tutor’s decisions and their effect on the reward signal. (a) Ratio of prompts identified as non-challenging by the tutor (and thus augmented with constraints) across policies of different sizes. The ratio grows with policy size, showing that the tutor allocates more constraints to stronger policies. (b) Per-prompt reward mean and standard deviation, grouped by the tutor’s challenging/non-ch… view at source ↗
Figure 3
Figure 3. Figure 3: A single atomic constraint reinstates discriminative reward signal on a non-challenging prompt. Before adding a constraint c, all rollouts score 100 (std = 0); after, reward mean drops to 92.5 with std 14.9. Rollout A explicitly cites a named policy (satisfying c); Rollout B deflects to “consult your supervisor” without naming one. Generalizing the LLM-as-a-Tutor framework beyond constraint-additive tasks.… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example: defining humanistic techniques and walking through the five named [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative example: the original rollouts score consistently high on formatting the citation. [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative example: composing a monologue that covers the seven listed guilty actions is [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative example: producing the alien-gestalt narrative from the writing prompt is a task [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 10 linked inside Pith

  1. [1]

    Online difficulty filtering for reasoning oriented reinforcement learning

    Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 700–719, 2026. URL https://aclanthology.org/2026. eac...

    2026

  2. [2]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning (ICML), pages 41–48, 2009

    2009

  3. [3]

    Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Ségerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob...

    2023

  4. [4]

    Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 4299–4307. Curran Associates, Inc., 2017....

    2017

  5. [5]

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean M. Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. InThe Fourteenth International Conference on Learning Representations, 2026. URL https:// openreview.net/forum?id=c1bTcrDmt4

    2026

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Advancedif: Rubric-based bench- marking and reinforcement learning for advancing llm instruction following.arXiv preprint arXiv:2511.10507, 2025

    Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, et al. Advancedif: Rubric-based bench- marking and reinforcement learning for advancing llm instruction following.arXiv preprint arXiv:2511.10507, 2025

    arXiv 2025

  8. [8]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  9. [9]

    Large language models are reasoning teachers

    Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, 2023

    2023

  10. [10]

    R-zero: Self-evolving reasoning LLM from zero data

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning LLM from zero data. arXiv preprint arXiv:2508.05004, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Vcrl: Variance-based curriculum reinforcement learning for large language models

    Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, Yuewei Zhang, Guohua Liu, and Hao Wang. Vcrl: Variance-based curriculum reinforcement learning for large language models. arXiv preprint arXiv:2509.19803, 2025

    arXiv 2025

  12. [12]

    Followbench: A multi-level fine-grained constraints following benchmark for large language models

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4667–4688, 2024. 10

    2024

  13. [13]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id= 8euJaTveKw

    2024

  14. [14]

    Language self-play for data-free training.arXiv preprint arXiv:2509.07414, 2025

    Jakub Grudzien Kuba, Mengting Gu, Qi Ma, Yuandong Tian, Vijai Mohan, and Jason Chen. Language self-play for data-free training.arXiv preprint arXiv:2509.07414, 2025

    arXiv 2025

  15. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023

    2023

  16. [16]

    SPICE: Self-play in corpus environments improves reasoning.arXiv preprint arXiv:2510.24684, 2025

    Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. SPICE: Self-play in corpus environments improves reasoning.arXiv preprint arXiv:2510.24684, 2025

    arXiv 2025

  17. [17]

    Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.arXiv preprint arXiv:2510.07743, 2025

    Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.arXiv preprint arXiv:2510.07743, 2025

    arXiv 2025

  18. [18]

    G- Eval: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2511–2522, 2023

    2023

  19. [19]

    Aligning with human judgement: The role of pairwise preference in large language model ev aluators

    Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vuli´c, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model ev aluators. InFirst Conference on Language Modeling (COLM), 2024

    2024

  20. [20]

    LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models

    Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 139–151, 2024

    2024

  21. [21]

    Teaching small language models to reason

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, 2023

    2023

  22. [22]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human fee...

    2022

  23. [23]

    Infobench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuan- sheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. InFindings of the Association for Computational Linguistics: ACL 2024, pages 13025–13048, 2024

    2024

  24. [24]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  25. [25]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. 11

    Pith/arXiv arXiv 2017

  26. [26]

    Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, and Pang Wei Koh. Dr tulu: Reinforcement learning with ev...

    Pith/arXiv arXiv 2025

  27. [27]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025

    2025

  29. [29]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  30. [30]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, second edition, 2018

    2018

  31. [31]

    Proximal curriculum for reinforcement learning agents.Transactions on Machine Learning Research, 2023

    Georgios Tzannetos, Bárbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal curriculum for reinforcement learning agents.Transactions on Machine Learning Research, 2023

    2023

  32. [32]

    Checklists are better than reward models for aligning language models

    Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. InAdvances in Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id= RPRqKhjrr6. Spotlight

    2025

  33. [33]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine Learning, 8:229–256, 1992

    1992

  34. [34]

    Wizardlm: Empowering large pre-trained language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Le, Qijun Tan, and Yuan Liu

    Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V . Le, Qijun Tan, and Yuan Liu. Scalable reinforcement post-training beyond static human prompts: Evolving alignment via asymmetric self-play. InProceedings of the 42nd International Conference on Machine Learning, 2025. URLhttps://arxiv.org/abs/2411.00062

    Pith/arXiv arXiv 2025

  37. [37]

    Absolute zero: Reinforced self-play reasoning with zero data

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. InAdvances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.03335. Spotlight

    Pith/arXiv arXiv 2025

  38. [38]

    Wildchat: 1M chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1M chatgpt interaction logs in the wild. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. URLhttps://openreview.net/forum?id=Bl8u7ZRlbM

    2024

  39. [39]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InAdvances in Neu- ral Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track,

    2023

  40. [40]

    , and the criterion might be

    URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html. 12 A Limitations Scope of evaluated policy–tutor configurations.Our main results in Table 1 are reported for a single policy–tutor pair (Qwen3-1.7B policy with Qwen3-8B tutor), and we do not run the full benchmark suite a...

    2023