REVIEW 4 major objections 3 minor 142 references
Weak-to-Strong On-Policy Distillation
T0 review · 4 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper claims that a strong LLM can improve by distilling from the logit difference between two weaker models, even when both are less capable than the student.
desk verdict A clean, likeable distillation paper whose headline weak-to-strong claim is undercut by missing OPD baselines in the very settings where all sources are genuinely weaker. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the proxy teacher, a product-of-experts composition in logit space: πT,α(a|s) ∝ πbase(a|s) · (π+(a|s) / π−(a|s))^α. The ratio π+/π− is the capability reward carried by the contrast pair; subtracting logits cancels shared weakness; α controls how far the resulting distribution may move from the student's base. Distillation minimizes reverse KL on student-generated prefixes, so the teacher corrects exactly the states the student visits. A multi-teacher extension sums several such directions in one run.
What would settle it
A randomized-direction control: take the same positive and negative models but replace z+ − z− at every token with a random vector of identical norm, and distill that. If the student improves as much as with the true direction, the direction itself is not the carrier; if it does not, the identity of the contrast direction is causal.
Extended reading notes
Core claim
The central claim is that the logit difference z+(s) − z−(s) between a positive and a negative weak model cancels what the two share and preserves the direction along which the positive improves over the negative. Adding this direction with a coefficient α to the student's base logits yields a proxy teacher πT,α = softmax(zbase + α(z+ − z−)) that is simultaneously more capable in the target skill and distributionally close to the student. The student minimizes per-token reverse KL toward this proxy on its own rollouts. The paper argues this isolates and transfers capability rather than imitating a weak supervisor, and reports gains across three contrast types: pre-RL vs post-RL, larger vs sm
Load-bearing premise
The assumption that carries the paper is that the logit difference between two weak models is a clean, transferable capability direction; if it mostly encodes style, format, or benchmark artifacts, the proxy teacher would just be a tuned interpolation and the weak-to-strong story collapses.
Editorial extensions
If this is right
- A frontier model can keep improving without any stronger teacher, as long as a contrast pair with a separable capability gap exists.
- The method transfers RL-acquired skills to a large student without ever running RL at the student's scale.
- Contrast directions from different sources (RL, scale, hints) can be merged in a single distillation run by adding their logit differences.
- Out-of-domain evaluation suggests the transferred skill does not erode general ability; in the paper's experiments it improves the student on scientific reasoning and instruction following.
Reading between the lines
- The same recipe could mine other contrast pairs — before/after safety tuning, generalist vs specialist, native vs translated — as long as their logit difference is directionally meaningful, making weak-to-strong a general post-training primitive.
- Since the method requires only logits, it could be run with the student's own checkpoints from different training steps as the positive/negative pair, turning self-distillation into a weak-to-strong signal without any extra model.
- A testable extension is per-token or per-example α: the paper uses a single scalar and shows performance peaks at moderate values, so an adaptive schedule might push gains further.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces W2S-OPD, a method to distill a strong student LLM from one or more weaker models. A proxy teacher is synthesized in logit space as softmax(z_base + α(z_+ − z_−)), where the contrast pair (m_+, m_−) is intended to isolate a capability direction; the student is trained by on-policy reverse-KL distillation against this proxy teacher. Three contrast instantiations are evaluated: a post-RL 4B expert against its pre-RL initialization, two off-the-shelf base models (4B and 0.6B), and a single 4B model conditioned on correct versus wrong hints. The student is Qwen3-8B. The authors report gains over the student base on math and code benchmarks, claim W2S-OPD outperforms OPD, and present token-level analyses of which reasoning episodes are reinforced.
Significance. If the empirical claims hold, the idea is significant: it offers a way to improve frontier models without a stronger teacher, using direction/subtraction in logit space rather than direct imitation. The formal connection in Appendix B (exponential tilting / product-of-experts view) is elegant and makes the method easy to adopt. The three contrast instantiations are practical and well motivated, and the paper includes useful analysis of learning signals and runtime overhead. However, the central attribution is currently under-supported: the OPD baseline is only reported in the pre-RL/post-RL setting, where the positive teacher is not actually weaker than the student, and the pure weak-to-strong settings lack the OPD control that would isolate the effect of the contrast subtraction. In addition, the main results rely on α tuned on the evaluation benchmarks without multiple seeds or error bars, making the claimed superiority to OPD and the 'weak-to-strong' interpretation fragile.
major comments (4)
- [§4.2, Tables 3 and 4] In the two genuinely weak-to-strong settings (smaller/larger and correct/wrong hints), the only baseline is the student's base model; no OPD-from-the-positive-model baseline is reported. Without that control, the observed improvement cannot be attributed to the contrast subtraction z_+ − z_−. It could equally come from plain on-policy distillation toward the weak positive model, from a distributional-smoothing effect, or from the α hyperparameter. Please add an OPD baseline with the positive model (Qwen3-4B) as teacher under the identical configuration for these settings, and also an α=0 control.
- [§4.3.1, Figure 4 and Table 7] The main results are reported after selecting α on the same evaluation benchmarks (Figure 4), and Table 7 fixes α=1.0 for math and 0.75 for code. Figure 4 shows that the optimal α varies across benchmarks (e.g., on LiveCodeBench α=0.75 gives 26.8 while α=1.25 gives 19.7, below the OPD reference of 22.0). This makes the headline gains optimistic and does not support the abstract's claim of consistent outperformance. Report results with a validation-selected α, or provide a robustness table across α values and multiple seeds/error bars for at least the main settings.
- [§3.1, Eq. (2) and Abstract] The proxy teacher is anchored at the student's own base logits, so the target is partially defined by the student. The paper's interpretation that the logit difference 'isolates a capability direction' is not independently tested; the method could reduce to a tuned interpolation between the student and a weak model. Provide a control that separates the contrast from the anchor, e.g., α=0, a negative-direction control, or a random/permuted logit difference. Also, the abstract says 'keeps improving the student even when every supervision source is weaker,' but the only setting with an OPD comparison (pre-RL/post-RL, Table 2) has a positive model that is much stronger on math (48.8 vs 17.0), so it is smaller but not weaker. The pure weak settings have no OPD baseline, leaving the central claim empirically unverified.
- [§4.2 and Table 2] The claim that 'W2S-OPD enables the student to surpass the domain teacher' is only demonstrated in the pre-RL/post-RL setting, where the positive model is a strong domain expert (Qwen3-4B-RL: 48.8 math vs student 17.0). That is not a weak-to-strong transfer in the sense claimed in the abstract and introduction. Please clearly separate the strong-expert setting from the genuinely weak settings, and state that the OPD comparison applies only to the former. This is not a fatal issue if the weak-setting results are supported with proper baselines, but the current presentation overstates the scope of the finding.
minor comments (3)
- [§3.2, third bullet] Minor grammar/wording issues: 'which token-level supervision efficiently' appears to be missing a verb; the sentence should be revised.
- [§3.1, after Eq. (3)] 'W2S-OPD address both challenges' should be 'addresses'.
- [Figure 2 caption] The caption contains an incomplete phrase ('divide and ...') and would benefit from clearer labels for the logit-offset path.
Circularity Check
No circular reduction found: the proxy-teacher construction is an empirical hypothesis tested on external benchmarks; missing OPD controls are an attribution gap, not a circular step.
full rationale
The paper's derivation chain is not circular. Equation (2) defines the proxy teacher as an exponential tilt of the student's anchor, and Equations (8)-(10) are algebraic identities re-expressing that definition as KL-regularized reward maximization; they do not by themselves entail the reported accuracy gains. The headline improvements come from external evaluations (AIME/HMMT, HumanEval+/MBPP+/LCB) against the student's own base and against OPD. alpha is a single hyperparameter swept in Figure 4, not a parameter fitted to the benchmark outcome, and no evaluation label enters the proxy-teacher construction. The phrase 'isolates the capability direction' is an interpretive label for z+ - z-; whether that direction transfers to a stronger student is an empirical hypothesis that could have failed and is tested, not assumed. The only self-citations (ThinkARM in Section 4.4 and Yu et al. 2024 in background) are not load-bearing for the central improvement claim. The reviewer's concern that Tables 3-4 omit the OPD-from-positive baseline is a valid attribution/experimental-control limitation, but an omitted control is not a reduction of a prediction to its inputs; therefore the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- α (amplification coefficient) =
1.0 (math), 0.75 (code)
assumptions (4)
- domain assumption Logit-space linear combination of models yields a valid proxy distribution that can be used as a distillation target.
- domain assumption The logit difference z+ − z− between two weak models isolates a transferable capability direction independent of scale.
- domain assumption Anchoring at zbase keeps the proxy teacher distributionally adjacent to the student, stabilizing optimization.
- standard math Top-K truncation with K=32 preserves almost all teacher signal.
Cite this review
Pith. "Pith review of Weak-to-Strong On-Policy Distillation." pith.science (2026). https://pith.science/paper/THL3IADC
@misc{pith2026260726246,
author = {Pith},
title = {Pith review of: Weak-to-Strong On-Policy Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/THL3IADC}},
note = {Machine review of arXiv:2607.26246}
}
read the original abstract
On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Benjamin Elizalde and Soham Deshmukh and Mahmoud Al Ismail and Huaming Wang , title =. 2023 , url =. doi:10.1109/ICASSP49357.2023.10095889 , timestamp =
arXiv 2023
-
[2]
Advances in Neural Information Processing Systems , volume=
Checklists are better than reward models for aligning language models , author=. Advances in Neural Information Processing Systems , volume=
-
[3]
arXiv preprint arXiv:2605.12474 , year=
Reward hacking in rubric-based reinforcement learning , author=. arXiv preprint arXiv:2605.12474 , year=
-
[4]
2025 , note =
Gemini 3.1 Pro , howpublished =. 2025 , note =
2025
-
[5]
arXiv preprint arXiv:2602.05125 , year=
Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks , author=. arXiv preprint arXiv:2602.05125 , year=
-
[6]
arXiv preprint arXiv:2501.12948 , year=
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=
-
[7]
arXiv preprint arXiv:2602.21628 , year=
Rucl: Stratified rubric-based curriculum learning for multimodal large language model reasoning , author=. arXiv preprint arXiv:2602.21628 , year=
-
[8]
arXiv preprint arXiv:2602.00846 , year=
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis , author=. arXiv preprint arXiv:2602.00846 , year=
Show all 142 references
-
[9]
arXiv preprint arXiv:2603.16600 , year=
Rationale matters: Learning transferable rubrics via proxy-guided critique for VLM reward models , author=. arXiv preprint arXiv:2603.16600 , year=
-
[10]
arXiv preprint arXiv:2604.13029 , year=
Visual preference optimization with rubric rewards , author=. arXiv preprint arXiv:2604.13029 , year=
-
[11]
arXiv preprint arXiv:2510.14738 , year=
AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning , author=. arXiv preprint arXiv:2510.14738 , year=
-
[12]
arXiv preprint arXiv:2602.04649 , year=
Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models , author=. arXiv preprint arXiv:2602.04649 , year=
-
[13]
arXiv preprint arXiv:2602.01511 , year=
Alternating reinforcement learning for rubric-based reward modeling in non-verifiable llm post-training , author=. arXiv preprint arXiv:2602.01511 , year=
-
[14]
arXiv preprint arXiv:2510.07284 , year=
Online rubrics elicitation from pairwise comparisons , author=. arXiv preprint arXiv:2510.07284 , year=
-
[15]
arXiv preprint arXiv:2602.10885 , year=
Reinforcing chain-of-thought reasoning with self-evolving rubrics , author=. arXiv preprint arXiv:2602.10885 , year=
-
[16]
arXiv preprint arXiv:2511.19399 , year=
Dr tulu: Reinforcement learning with evolving rubrics for deep research , author=. arXiv preprint arXiv:2511.19399 , year=
-
[17]
arXiv preprint arXiv:2508.16949 , year=
Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general llm reasoning , author=. arXiv preprint arXiv:2508.16949 , year=
-
[18]
arXiv e-prints , pages=
Auto-rubric: Learning to extract generalizable criteria for reward modeling , author=. arXiv e-prints , pages=
-
[19]
Findings of the Association for Computational Linguistics: ACL 2025 , pages=
CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=
2025
-
[20]
arXiv preprint arXiv:2510.07743 , year=
Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment , author=. arXiv preprint arXiv:2510.07743 , year=
-
[21]
International Conference on Learning Representations , volume=
Prometheus: Inducing fine-grained evaluation capability in language models , author=. International Conference on Learning Representations , volume=
-
[22]
Advances in Neural Information Processing Systems , volume=
Rule based rewards for language model safety , author=. Advances in Neural Information Processing Systems , volume=
-
[23]
arXiv preprint arXiv:2512.20061 , year=
Scaling Reinforcement Learning for Content Moderation with Large Language Models , author=. arXiv preprint arXiv:2512.20061 , year=
-
[24]
arXiv preprint arXiv:2602.02994 , year=
Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation , author=. arXiv preprint arXiv:2602.02994 , year=
-
[25]
arXiv preprint arXiv:1810.08575 , year=
Supervising strong learners by amplifying weak experts , author=. arXiv preprint arXiv:1810.08575 , year=
-
[26]
arXiv preprint arXiv:2505.09388 , year=
Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=
-
[27]
International Conference on Learning Representations , volume=
On-policy distillation of language models: Learning from self-generated mistakes , author=. International Conference on Learning Representations , volume=
-
[28]
arXiv preprint arXiv:2312.09390 , year=
Weak-to-strong generalization: Eliciting strong capabilities with weak supervision , author=. arXiv preprint arXiv:2312.09390 , year=
-
[29]
arXiv preprint arXiv:2606.00564 , year=
Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding , author=. arXiv preprint arXiv:2606.00564 , year=
-
[30]
arXiv preprint arXiv:2605.21924 , year=
Visual-Advantage On-Policy Distillation for Vision-Language Models , author=. arXiv preprint arXiv:2605.21924 , year=
-
[31]
arXiv preprint arXiv:2603.07079 , year=
Entropy-aware on-policy distillation of language models , author=. arXiv preprint arXiv:2603.07079 , year=
-
[32]
arXiv preprint arXiv:2603.11137 , year=
Scaling reasoning efficiently via relaxed on-policy distillation , author=. arXiv preprint arXiv:2603.11137 , year=
-
[33]
Findings of the Association for Computational Linguistics: ACL 2026 , pages=
Stable on-policy distillation through adaptive target reformulation , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=
2026
-
[34]
arXiv preprint arXiv:2606.01249 , year=
Trust Region On-Policy Distillation , author=. arXiv preprint arXiv:2606.01249 , year=
-
[35]
arXiv preprint arXiv:2602.12275 , year=
On-policy context distillation for language models , author=. arXiv preprint arXiv:2602.12275 , year=
-
[36]
arXiv preprint arXiv:2604.13016 , year=
Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe , author=. arXiv preprint arXiv:2604.13016 , year=
-
[37]
arXiv preprint arXiv:2601.19897 , year=
Self-Distillation Enables Continual Learning , author=. arXiv preprint arXiv:2601.19897 , year=
-
[38]
arXiv preprint arXiv:2605.27115 , year=
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation , author=. arXiv preprint arXiv:2605.27115 , year=
-
[39]
arXiv preprint arXiv:2601.20802 , year=
Reinforcement Learning via Self-Distillation , author=. arXiv preprint arXiv:2601.20802 , year=
-
[40]
arXiv preprint arXiv:2601.18734 , year=
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=
-
[41]
arXiv preprint arXiv:2604.20244 , year=
Hybrid policy distillation for llms , author=. arXiv preprint arXiv:2604.20244 , year=
-
[42]
arXiv preprint arXiv:2402.08679 , year=
Cold-attack: Jailbreaking llms with stealthiness and controllability , author=. arXiv preprint arXiv:2402.08679 , year=
-
[43]
arXiv preprint arXiv:2607.05394 , year=
Weak-to-Strong Generalization via Direct On-Policy Distillation , author=. arXiv preprint arXiv:2607.05394 , year=
-
[44]
arXiv preprint arXiv:2606.11709 , year=
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation , author=. arXiv preprint arXiv:2606.11709 , year=
-
[45]
arXiv preprint arXiv:2604.03128 , year=
Self-distilled rlvr , author=. arXiv preprint arXiv:2604.03128 , year=
-
[46]
2014 , publisher=
Mathematical problem solving , author=. 2014 , publisher=
2014
-
[47]
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Schoenfeld’s anatomy of mathematical reasoning by language models , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[48]
Advances in Neural Information Processing Systems , volume=
Reinforcement learning teachers of test time scaling , author=. Advances in Neural Information Processing Systems , volume=
-
[49]
Advances in Neural Information Processing Systems , volume=
Can llms learn by teaching for better reasoning? a preliminary study , author=. Advances in Neural Information Processing Systems , volume=
-
[50]
Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=
Gedi: Generative discriminator guided sequence generation , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=
2021
-
[51]
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=
FUDGE: Controlled text generation with future discriminators , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=
2021
-
[52]
Advances in Neural Information Processing Systems , volume=
Cold decoding: Energy-based constrained text generation with langevin dynamics , author=. Advances in Neural Information Processing Systems , volume=
-
[53]
arXiv preprint arXiv:2006.03535 , year=
Cocon: A self-supervised approach for controlled text generation , author=. arXiv preprint arXiv:2006.03535 , year=
2006 arXiv
-
[54]
arXiv preprint arXiv:1909.05858 , year=
Ctrl: A conditional transformer language model for controllable generation , author=. arXiv preprint arXiv:1909.05858 , year=
1909 arXiv
-
[55]
Proceedings of the 28th International Conference on Computational Linguistics , pages=
Exploring controllable text generation techniques , author=. Proceedings of the 28th International Conference on Computational Linguistics , pages=
-
[56]
Advances in Neural Information Processing Systems , volume=
Generalizing verifiable instruction following , author=. Advances in Neural Information Processing Systems , volume=
-
[57]
arXiv preprint arXiv:2606.30406 , year=
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training , author=. arXiv preprint arXiv:2606.30406 , year=
-
[58]
arXiv preprint arXiv:2602.15763 , year=
Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year=
-
[59]
arXiv preprint arXiv:2606.19348 , year=
Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=
-
[60]
Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
Weak-to-strong reasoning , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
2024
-
[61]
arXiv preprint arXiv:2311.12022 , year=
Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=
-
[62]
DExperts: Decoding-time controlled text generation with experts and anti-experts , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers...
-
[63]
arXiv preprint arXiv:2401.08565 , year=
Tuning language models by proxy , author=. arXiv preprint arXiv:2401.08565 , year=
-
[64]
Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=
Contrastive decoding: Open-ended text generation as optimization , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=
-
[65]
arXiv preprint arXiv:2604.00626 , year=
A survey of on-policy distillation for large language models , author=. arXiv preprint arXiv:2604.00626 , year=
-
[66]
arXiv preprint arXiv:2401.17256 , year=
Weak-to-strong jailbreaking on large language models , author=. arXiv preprint arXiv:2401.17256 , year=
-
[67]
arXiv preprint arXiv:2601.02780 , year=
Mimo-v2-flash technical report , author=. arXiv preprint arXiv:2601.02780 , year=
-
[68]
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Incentivizing strong reasoning from weak supervision , author=. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[69]
arXiv preprint arXiv:2506.03109 , year=
On weak-to-strong generalization and f-divergence , author=. arXiv preprint arXiv:2506.03109 , year=
-
[70]
arXiv preprint arXiv:2606.06021 , year=
OPRD: On-Policy Representation Distillation , author=. arXiv preprint arXiv:2606.06021 , year=
-
[71]
International Conference on Learning Representations , volume=
Minillm: Knowledge distillation of large language models , author=. International Conference on Learning Representations , volume=
-
[72]
forward KL , author=
Revisiting weak-to-strong generalization in theory and practice: Reverse KL vs. forward KL , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=
2025
-
[73]
Thinking Machines Lab: Connectionism , year =
Kevin Lu and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =
-
[74]
arXiv preprint arXiv:2502.01456 , year=
Process reinforcement through implicit rewards , author=. arXiv preprint arXiv:2502.01456 , year=
-
[75]
American Invitational Mathematics Examination (AIME) 2025 , author=
2025
-
[76]
arXiv preprint arXiv:2504.11456 , year=
Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning , author=. arXiv preprint arXiv:2504.11456 , year=
-
[77]
2024 , howpublished =
2024
-
[78]
arXiv preprint arXiv:2402.03898 , year=
Distillm: Towards streamlined distillation for large language models , author=. arXiv preprint arXiv:2402.03898 , year=
-
[79]
International Conference on Learning Representations , volume=
Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=
-
[80]
URL https://matharena
Matharena: Evaluating llms on uncontaminated math competitions, February 2025 , author=. URL https://matharena. ai , volume=
2025
-
[81]
Advances in neural information processing systems , volume=
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation , author=. Advances in neural information processing systems , volume=
- [82]
-
[83]
2023 , url =
Bharathan Balaji and Venkata Sai Gargeya Vunnava and Nina Domingo and Shikhar Gupta and Harsh Gupta and Geoffrey Guest and Aravind Srinivasan , title =. 2023 , url =. doi:10.1145/3616385 , timestamp =
2023 doi
-
[84]
Sakshi and Jaehyeon Kim and Wei Ping and Rafael Valle and Dinesh Manocha and Bryan Catanzaro , title =
Sreyan Ghosh and Zhifeng Kong and Sonal Kumar and S. Sakshi and Jaehyeon Kim and Wei Ping and Rafael Valle and Dinesh Manocha and Bryan Catanzaro , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.03983 , eprinttype =. 2503.03983 , timestamp =
-
[85]
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models , journal =
Arushi Goel and Sreyan Ghosh and Jaehyeon Kim and Sonal Kumar and Zhifeng Kong and Sang. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models , journal =. 2025 , url =. doi:10.48550/ARXIV.2507.08128 , eprinttype =. 2507.08128 , timestamp =
-
[86]
Proceedings of the 30th ACM International Conference on Multimedia , pages=
AVQA: A Dataset for Audio-Visual Question Answering on Videos , author=. Proceedings of the 30th ACM International Conference on Multimedia , pages=
-
[87]
Goucher and Adam Perelman and Aditya Ramesh and Aidan Clark and AJ Ostrow and Akila Welihinda and Alan Hayes and Alec Radford and Aleksander Madry and Alex Baker
Aaron Hurst and Adam Lerer and Adam P. Goucher and Adam Perelman and Aditya Ramesh and Aidan Clark and AJ Ostrow and Akila Welihinda and Alan Hayes and Alec Radford and Aleksander Madry and Alex Baker. GPT-4o System Card , journal =. 2024 , url =. doi:10.48550/ARXIV.2410.21276...
- [88]
-
[89]
Wav2CLIP: Learning Robust Audio Representations from Clip , booktitle =
Ho. Wav2CLIP: Learning Robust Audio Representations from Clip , booktitle =. 2022 , url =. doi:10.1109/ICASSP43922.2022.9747669 , timestamp =
2022
-
[90]
Pengi: An Audio Language Model for Audio Tasks , booktitle =
Soham Deshmukh and Benjamin Elizalde and Rita Singh and Huaming Wang , editor =. Pengi: An Audio Language Model for Audio Tasks , booktitle =. 2023 , url =
2023
-
[91]
Liu and Leonid Karlinsky and James R
Yuan Gong and Hongyin Luo and Alexander H. Liu and Leonid Karlinsky and James R. Glass , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =
2024
-
[92]
Optimal Transport for Treatment Effect Estimation , booktitle =
Hao Wang and Jiajun Fan and Zhichao Chen and Haoxuan Li and Weiming Liu and Tianqiao Liu and Quanyu Dai and Yichao Wang and Zhenhua Dong and Ruiming Tang , editor =. Optimal Transport for Treatment Effect Estimation , booktitle =. 2023 , url =
2023
-
[93]
CoRR , volume =
Jiajun Fan , title =. CoRR , volume =. 2021 , url =. 2112.04145 , timestamp =
2021 arXiv
-
[94]
Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection , booktitle =
Jiajun Fan and Yuzheng Zhuang and Yuecheng Liu and Jianye Hao and Bin Wang and Jiangcheng Zhu and Hao Wang and Shu. Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection , booktitle =. 2023 , url =
2023
-
[95]
Generalized Data Distribution Iteration , booktitle =
Jiajun Fan and Changnan Xiao , editor =. Generalized Data Distribution Iteration , booktitle =. 2022 , url =
2022
-
[96]
CoRR , volume =
Jiajun Fan and Changnan Xiao and Yue Huang , title =. CoRR , volume =. 2021 , url =. 2106.06232 , timestamp =
2021 arXiv
-
[97]
CoRR , volume =
Ye Li and Yuan Meng and Zewen Sun and Kangye Ji and Chen Tang and Jiajun Fan and Xinzhu Ma and Shutao Xia and Zhi Wang and Wenwu Zhu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.12723 , eprinttype =. 2506.12723 , timestamp =
2025 doi
- [98]
-
[99]
CoRR , volume =
Changnan Xiao and Haosen Shi and Jiajun Fan and Shihong Deng , title =. CoRR , volume =. 2021 , url =. 2106.00707 , timestamp =
2021 arXiv
-
[100]
CoRR , volume =
Hao Wang and Zhichao Chen and Jiajun Fan and Yuxin Huang and Weiming Liu and Xinggao Liu , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2210.11039 , eprinttype =. 2210.11039 , timestamp =
2022 doi
-
[101]
CoRR , volume =
Jiajun Fan and He Ba and Xian Guo and Jianye Hao , title =. CoRR , volume =. 2020 , url =. 2011.06752 , timestamp =
2020 arXiv
-
[102]
The Thirteenth International Conference on Learning Representations,
Jiajun Fan and Shuaike Shen and Chaoran Cheng and Yuxin Chen and Chumeng Liang and Ge Liu , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =
2025
-
[103]
CoRR , volume =
Ziwen Wang and Jiajun Fan and Ruihan Guo and Thao Nguyen and Heng Ji and Ge Liu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.07459 , eprinttype =. 2506.07459 , timestamp =
2025 doi
-
[104]
CoRR , volume =
Ziwen Wang and Jiajun Fan and Thao Nguyen and Heng Ji and Ge Liu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.07413 , eprinttype =. 2506.07413 , timestamp =
2025 doi
-
[105]
CoRR , volume =
John Schulman and Filip Wolski and Prafulla Dhariwal and Alec Radford and Oleg Klimov , title =. CoRR , volume =. 2017 , url =. 1707.06347 , timestamp =
2017 arXiv
-
[106]
CoRR , volume =
Changnan Xiao and Haosen Shi and Jiajun Fan and Shihong Deng , title =. CoRR , volume =. 2021 , url =. 2105.03923 , timestamp =
2021 arXiv
- [107]
-
[108]
The Twelfth International Conference on Learning Representations,
Changli Tang and Wenyi Yu and Guangzhi Sun and Xianzhao Chen and Tian Tan and Wei Li and Lu Lu and Zejun Ma and Chao Zhang , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =
2024
-
[109]
Forty-first International Conference on Machine Learning,
Zhifeng Kong and Arushi Goel and Rohan Badlani and Wei Ping and Rafael Valle and Bryan Catanzaro , title =. Forty-first International Conference on Machine Learning,. 2024 , url =
2024
- [110]
-
[111]
CoRR , volume =
Yunfei Chu and Jin Xu and Qian Yang and Haojie Wei and Xipin Wei and Zhifang Guo and Yichong Leng and Yuanjun Lv and Jinzheng He and Junyang Lin and Chang Zhou and Jingren Zhou , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2407.10759 , eprinttype =. 2407.10759 ,...
-
[112]
CoRR , volume =
Jin Xu and Zhifang Guo and Jinzheng He and Hangrui Hu and Ting He and Shuai Bai and Keqin Chen and Jialin Wang and Yang Fan and Kai Dang and Bin Zhang and Xiong Wang and Yunfei Chu and Junyang Lin , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.20215 , eprint...
-
[113]
2025 , eprint=
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models , author=. 2025 , eprint=
2025
-
[114]
Chi and Quoc V
Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed H. Chi and Quoc V. Le and Denny Zhou , editor =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , booktitle =. 2022 , url =
2022
- [115]
- [116]
-
[118]
CoRR , volume =
Zhifei Xie and Mingbao Lin and Zihang Liu and Pengcheng Wu and Shuicheng Yan and Chunyan Miao , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.02318 , eprinttype =. 2503.02318 , timestamp =
2025 doi
- [119]
-
[120]
2025 , eprint=
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=
2025
-
[121]
arXiv preprint arXiv:2406.05673 , year=
Flow of reasoning: Training llms for divergent reasoning with minimal examples , author=. arXiv preprint arXiv:2406.05673 , year=
-
[122]
arXiv preprint arXiv:2601.06559 , year=
ArrowGEV: Grounding Events in Video via Learning the Arrow of Time , author=. arXiv preprint arXiv:2601.06559 , year=
-
[123]
arXiv preprint arXiv:2601.04171 , year=
Agentic Rubrics as Contextual Verifiers for SWE Agents , author=. arXiv preprint arXiv:2601.04171 , year=
-
[124]
Advances in Neural Information Processing Systems , volume=
Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=
-
[125]
arXiv preprint arXiv:2511.12344 , year=
Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning , author=. arXiv preprint arXiv:2511.12344 , year=
-
[126]
arXiv preprint arXiv:2507.17746 , year=
Rubrics as rewards: Reinforcement learning beyond verifiable domains , author=. arXiv preprint arXiv:2507.17746 , year=
-
[127]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =
DeepSeek. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , journal =. 2025 , url =. doi:10.48550/ARXIV.2501.12948 , eprinttype =. 2501.12948 , timestamp =
- [128]
-
[129]
arXiv preprint arXiv:2503.11197 , year=
Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering , author=. arXiv preprint arXiv:2503.11197 , year=
-
[130]
arXiv preprint arXiv:2410.19168 , year=
Mmau: A massive multi-task audio understanding and reasoning benchmark , author=. arXiv preprint arXiv:2410.19168 , year=
-
[131]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Audio-Thinker: Guiding Large Audio Language Model When and How to Think via Reinforcement Learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[132]
arXiv preprint arXiv:2604.12527 , year=
Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models , author=. arXiv preprint arXiv:2604.12527 , year=
-
[133]
arXiv preprint arXiv:2510.11454 , year=
Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning , author=. arXiv preprint arXiv:2510.11454 , year=
-
[134]
arXiv preprint arXiv:2602.13685 , year=
AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoning , author=. arXiv preprint arXiv:2602.13685 , year=
-
[135]
arXiv preprint arXiv:2602.10439 , year=
AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoning , author=. arXiv preprint arXiv:2602.10439 , year=
-
[136]
arXiv preprint arXiv:2511.15848 , year=
Step-Audio-R1 Technical Report , author=. arXiv preprint arXiv:2511.15848 , year=
-
[137]
arXiv preprint arXiv:2510.20867 , year=
Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards , author=. arXiv preprint arXiv:2510.20867 , year=
-
[138]
ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
Wav2clip: Learning robust audio representations from clip , author=. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2022 , organization=
2022
-
[139]
International conference on machine learning , pages=
Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=
2021
-
[140]
ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=
Clap learning audio concepts from natural language supervision , author=. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2023 , organization=
2023
-
[141]
arXiv preprint arXiv:2503.02318 , year=
Audio-reasoner: Improving reasoning capability in large audio language models , author=. arXiv preprint arXiv:2503.02318 , year=
-
[142]
arXiv preprint arXiv:2506.04779 , year=
Mmsu: A massive multi-task spoken language understanding and reasoning benchmark , author=. arXiv preprint arXiv:2506.04779 , year=
-
[143]
arXiv preprint arXiv:2505.13032 , year=
Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix , author=. arXiv preprint arXiv:2505.13032 , year=
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.