Pith. sign in

REVIEW 4 major objections 5 minor 56 references

ReflectRL claims that failed expert trajectories can be turned into dense training signals by reflective-to-direct reasoning, improving model accuracy across nine benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ReflectRL repurposes failed expert reasoning traces as reflective scaffolding during RL and distillation training, then transitions the policy to direct reasoning, improving math and science benchmark scores.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful idea — expert failures as reflective hints with an annealed curriculum — but the claimed transition mechanism is under-tested and the 'consistent improvements' headline overreaches. the 4 major comments →

arxiv 2608.03972 v1 pith:VWFEEQU7 submitted 2026-08-04 cs.AI

ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

Jinhe Bi , Chennan Zhou , Zengjie Jin , Aniri , Shuo Lu , Wenke Huang , Hu Cao , Xun Xiao
show 5 more authors
Zhihong Zhu Volker Tresp Fei Shen Yunpu Ma Tat-Seng Chua
This is my paper
classification cs.AI
keywords Golden Negative TrajectoriesReflective ReasoningReflective-to-Direct Policy TransitionOn-policy reinforcement learningGRPOOn-policy distillationLLM reasoningProcess reward models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the failed reasoning attempts of a stronger model—what it calls Golden Negative Trajectories—are not waste. Even though such trajectories are wrong overall, they contain a long valid reasoning prefix and a localized error, and asking the training model to critique and repair that trace can be easier and more effective than asking it to solve the same hard problem from scratch. The paper names this the Reflection Advantage and builds ReflectRL, a lightweight training addition that feeds these failed traces as context during on-policy reinforcement learning or on-policy distillation, then gradually phases the context out. Across nine benchmarks, four model backbones, and four on-policy training methods, ReflectRL reports consistent accuracy gains—including outside the math training distribution—with shorter responses and lower per-step training cost than the base methods. The reason to care is that hard problems are exactly where expert models fail, so recovering useful signal from those failures widens the set of problems on which trajectory-guided training can help.

Core claim

The paper claims that expert failures contain a transferable reasoning structure: a valid prefix plus a localized error region. When the learner is prompted to reflect on such a trace, its success rate on that question rises markedly compared with direct solve-from-scratch prompting; the paper measures this as a positive Reflection Gain. ReflectRL exploits this by constructing training prompts that include the failed trajectory, training with the standard GRPO or OPD objective unchanged, and then using a cosine-decay transition kernel to reduce the fraction of reflective rollouts to zero. The intended effect is that the error-correction and repair behavior learned under reflection is compile

What carries the argument

Two named objects carry the argument. First is the Reflection Advantage, formalized as the difference in expected correctness between reflective prompting (question plus golden negative trajectory, asking the model to identify errors and repair them) and direct prompting (question only). Second is the Reflective-to-Direct Policy Transition, a cosine-decay kernel g(t)=p_l+(p_h−p_l)/2·[1+cos(πτ(t))] that sets how many rollouts in each mixed group receive the reflective context; in OPD the same schedule governs how many teacher contexts are reflective, while the student always sees direct prompts. These turn a discarded negative sample into a scaffold for dense learning signals without adding l

Load-bearing premise

The load-bearing premise is that gradually reducing reflective rollouts to zero lets the policy internalize error-correction behavior so that it persists in direct reasoning; if the gains actually come from continued exposure to expert hints during training, the transition mechanism would not be doing the work claimed.

What would settle it

Train ReflectRL with the reflective rollout fraction held constant at its initial value for the entire run, then evaluate under direct prompting with no GNT. If fixed-reflection training matches or exceeds the cosine-decay variant, the gradual transition is not what transfers the skill, and the reported direct-reasoning gains are better explained by ongoing exposure to GNT hints.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Hard problems with no correct expert trace can still be used for training, as long as the expert produces a structured near-miss with a usable prefix and error.
  • ReflectRL can be layered onto existing GRPO-family and on-policy distillation pipelines without modifying their objectives, reward computation, or inference interface.
  • Training can become cheaper per step even as accuracy rises, because reflective prompts are cheap to prefill and the model's continuations are substantially shorter.
  • The transition schedule maintains higher policy entropy, delaying the collapse to a narrow response mode that limits further learning.
  • Improvements outside the math distribution suggest the reflect–repair–solve procedure teaches a generalizable error-correction behavior, not memorized problem patterns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the causal analysis suggests selecting or synthesizing GNTs by process-reward quality—maximizing valid-prefix length while keeping a localized error—could improve data efficiency, rather than relying on whatever failed traces the expert happens to produce.
  • Beyond the paper: a model could bootstrap its own high-quality failures, filtered by a process reward model, as GNTs, reducing or removing dependence on a stronger teacher.
  • Beyond the paper: the observed response-length compression during the transition raises a testable hypothesis that reflective training acts as an implicit reasoning-precision regularizer, which could be probed on non-math reasoning tasks.
  • Beyond the paper: the teacher-side privileged-context recipe in the OPD variant suggests a general pattern for distilling repair skills in other domains—such as code or tool-use—where a flawed trace can guide the teacher while the student learns only the direct interface.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReflectRL, a plug-and-play framework for on-policy LLM reasoning training that repurposes incorrect expert trajectories ('Golden Negative Trajectories', GNTs) as contextual scaffolds. The method defines two prompt interfaces—Direct Reasoning and Reflective Reasoning—and, during training, anneals the fraction of reflective rollouts from a high initial proportion to a small terminal value via a cosine transition kernel (Eq. 7). ReflectRL is instantiated on top of GRPO-style RLVR by mixing reflective and direct rollouts in the same group-relative advantage computation (Eq. 9), and on top of On-Policy Distillation by using the GNT only as teacher-side privileged context (Eq. 11). Experiments span 9 benchmarks, 4 backbones, and 4 training methods, and the paper reports consistent accuracy gains, reduced response length, and higher entropy retention, alongside ablations on trajectory provenance, trajectory components, and transition-kernel shape.

Significance. If the claimed effect is real, ReflectRL would be a useful and low-cost way to convert abundant expert failures into training signal, with the notable strength that it leaves the base RLVR/OPD objectives unchanged and does not require online expert queries. The paper also contributes a released 69k GNT dataset and a reasonably thorough behavioral analysis (Reflection Gain, PRM profiling, component ablation). However, the current evidence does not fully establish the central mechanistic claim—that annealing from reflective to direct rollouts causes the policy to internalize error-correction behavior—or the 'consistently improves' claim, because the key control (fixed reflective mixture) is absent and the main tables report single-seed point estimates with several flat or negative entries. The idea is plausible and testable, so the paper is worth revising rather than rejecting.

major comments (4)
  1. [§4.4, Figure 6, Eq. (7)] The Reflective-to-Direct Policy Transition is the central mechanism, but the experiments never compare the cosine decay against a fixed reflective mixture (e.g., g(t)=0.5 throughout) or against a schedule matched for total reflective exposure. Figure 6 only compares different decay shapes against a no-GNT baseline. Consequently, the reported direct-mode gains could be caused by continued exposure to GNT hints, by the regularizing effect of mixed rollout groups, or by the higher average reward of reflective rollouts—not by 'compiling' reflection into direct parameters. A fixed-mixture control is required to support the claimed transition mechanism.
  2. [§3.4, Eq. (11), Appendix C] In the OPD instantiation the default terminal proportion is pl=0.05; with a rollout batch of B=1024, this leaves about 51 teacher-side reflective samples per batch, so the schedule never reaches direct-only teacher context. The claim that the student distills reflective capability into the direct pathway is therefore not actually tested under the reported configuration. Please report OPD runs with pl=0 (or a hard cutoff to zero) and examine whether the OPD results are sensitive to the residual 5% reflective context.
  3. [Tables 1–2, §4.2] The 'consistently improves' claim is not supported by single-seed point estimates. Several reported entries go in the opposite direction: DAPO+ReflectRL GPQA* drops from 16.2 to 14.7 in Table 1; OPD+ReflectRL MMLU-Pro drops from 41.6 to 38.1; and Qwen2.5-3B AIME25 drops from 3.44 to 2.08 in Table 2. Given the small test sets and stochasticity of RL training, at least 3 seeds (or a significance test) are needed for the main comparisons; otherwise the headline 'consistent improvement' is not established.
  4. [Abstract, §3.3, §D] The paper repeatedly describes the GNT signal as 'dense', but in the RLVR instantiation (Eq. 9) the reward remains the binary verifier r(q,o); the GNT only changes the prompt context. No token-level, process-level, or auxiliary dense reward is introduced in the RLVR variant. The 'dense' characterization is accurate only for the OPD variant (Eq. 11), where the teacher distribution provides token-level KL targets. Please revise the wording or provide evidence that reflective prompting alone creates denser credit assignment.
minor comments (5)
  1. [Eq. (7)] The transition kernel uses τ(t) but this normalized progress is not defined in the main text. Define it in Eq. (7) or point immediately to Appendix C; the current text says 'Appendix C provides the exact schedule' but the main equation is incomplete.
  2. [Figure 6] The right panel labels the schedulers with generalized equations, but the notation is inconsistent with Appendix C: 'UFT Cosine + Cutoff' in the figure has denominator td, while 'Cosine' uses td−tw. Align the two presentations and specify the parameter values used for each curve.
  3. [Figure 1] The Reflection Gain curves in (a) and the PRM profiles in (b) are shown without error bars or seed information. Since these measurements motivate the entire framework, add at least the number of runs or confidence intervals, and clarify whether the curve is a running mean over a single training run.
  4. [§4.1 and Table 2] The Llama-3.1-8B GRPO baseline is far below its own no-RL baseline on the ID average (3.89 vs. 7.05). This suggests an unstable or undertrained GRPO baseline; please comment on this, since it affects how much credit ReflectRL deserves for 'recovering' performance.
  5. [§3.3] The discussion after Eq. (9) correctly notes that with binary rewards normalization preserves the sign of advantages in non-degenerate groups, but it should also state that if all rollouts in a mixed group are correct (or all incorrect), the advantages are identically zero and the GNT context provides no learning signal in that group.

Circularity Check

0 steps flagged

Empirical systems paper; reported gains come from benchmark evaluations, not from equations that reduce to fitted inputs or self-citations.

full rationale

The paper is an empirical systems paper whose central claims are validated on external benchmarks. Nothing in the derivation chain reduces to its own inputs. The Reflection Advantage is defined in Eq. 6 but is measured from rollout correctness statistics (Figure 1a), not assumed or fitted. The RLVR objective (Eq. 9) is GRPO with mixed direct/reflective prompt templates and no extra loss term, and the OPD objective (Eq. 11) is reverse KL against teacher distributions; neither objective encodes the reported accuracy gains. The transition kernel g(t) (Eq. 7) is a schedule with hyperparameters selected on validation accuracy (Figure 6), which is standard model selection rather than fitting to test results. The paper's self-citations (e.g., EchoRL [8]) appear as baselines or related work, and the proposed mechanism does not depend on any cited 'uniqueness theorem' or on unverified prior results by the same authors. The skeptic's missing control — comparing against a fixed reflective mixture during the Reflective-to-Direct transition — is a causal-attribution limitation, not circularity: the absence of that control does not make any performance number true by construction. Similarly, the OPD terminal value pl=0.05 (Appendix C) with batch size 1024 leaves roughly 51 reflective teacher-side samples per batch, weakening the 'fully direct' interpretation of the transition, but this is an experimental-design detail rather than a circular step. No equation is fitted to the reported benchmark improvements, and the benchmarks are external to the training objective. Hence the honest finding is no significant circularity; the one-point-two-point allowance reflects only the presence of minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The method depends on several explicit design choices (transition schedule hyperparameters, rollout count) and on empirical assumptions about the quality of expert failures and the verifier. Most are reasonable for an empirical RL paper, but the lack of independent evidence for the 'goldenness' of negative trajectories and the PRM-based mechanism is a burden the paper carries without external validation.

free parameters (6)
  • ph (initial reflective proportion) = 0.5
    Chosen by hand; sets the starting fraction of reflective rollouts in Eq. (7).
  • pl (terminal reflective proportion) = 0.05 (0.0 for cutoff variants)
    Chosen by hand; controls the final residual reflective fraction.
  • tw (warm-up steps) = 50
    Chosen by hand; delay before cosine decay starts.
  • td (decay horizon) = 250
    Chosen by hand; length of the cosine decay period.
  • tc (cutoff step for cutoff variants) = 300
    Used only in schedule variants 2, 4, and 5; chosen by hand.
  • N (rollouts per query) = 8
    Standard GRPO setting; affects advantage estimation in Eq. (2).
axioms (4)
  • domain assumption Binary verifier rewards are correct for all nine benchmarks (Math-Verify for math, answer matching for choice tasks).
    Section 4.1 and Eq. (9) rely on r(q,o)=MathVerify(a,o) or exact-match; labeling errors would distort advantages.
  • domain assumption DeepSeek-R1 failures verified as incorrect are 'golden' negatives containing a valid prefix and a localized error.
    The PRM profile in Figure 1(b) is the primary evidence; if real-world expert failures are messier, the reflection gain may not generalize.
  • domain assumption The cosine transition kernel causes behavioral transfer from reflective to direct reasoning.
    The method assumes Eq. (7) compiles reflective behavior into direct reasoning; validation is indirect via Figure 6.
  • domain assumption Qwen2.5-Math-PRM-7B scores are a valid measure of step-level reasoning quality.
    Figure 1(b) and the causal story rest on this PRM; the PRM is itself a model that can be biased.
invented entities (2)
  • Golden Negative Trajectory (GNT) no independent evidence
    purpose: A failed expert reasoning trajectory used as reflective context in training prompts.
    A new data construct introduced by the paper; no external falsifiable handle beyond the paper's own dataset.
  • Reflection Advantage no independent evidence
    purpose: A measured quantity (Eq. 6) capturing the benefit of reflecting on a flawed trajectory over direct solving.
    A behavioral metric, not a physical entity; its existence is asserted from the paper's own experiments.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning." pith.science (2026). https://pith.science/paper/VWFEEQU7

@misc{pith2026260803972,
  author       = {Pith},
  title        = {Pith review of: ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWFEEQU7}},
  note         = {Machine review of arXiv:2608.03972}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 34 canonical work pages · 2 internal anchors

  1. [1]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    work page 2025

  2. [2]

    DAPO: An open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao ...

    work page 2025

  3. [3]

    Cot-kinetics: A theoretical modeling assessing lrm reasoning process.ArXiv, abs/2505.13408, 2025

    JinheBi, DanqiYan, YifanWang, WenkeHuang, HaokunChen, GuanchengWan, MangYe, XunXiao, Hin rich Schuetze, Volker Tresp, and Yunpu Ma. Cot-kinetics: A theoretical modeling assessing lrm reasoning process.ArXiv, abs/2505.13408, 2025. URLhttps://api.semanticscholar.org/CorpusID:278769227

    Pith/arXiv arXiv 2025

  4. [4]

    Loong: Synthesize long chain-of-thoughts at scale through verifiers, 2025

    Xingyue Huang, Rishabh, Gregor Franke, Ziyi Yang, Jiamu Bai, Weijie Bai, Jinhe Bi, Zifeng Ding, Yiqun Duan, Chengyu Fan, Wendong Fan, Xin Gao, Ruohao Guo, Yuan He, Zhuangzhuang He, Xianglong Hu, Neil Johnson, Bowen Li, Fangru Lin, Siyu Lin, Tong Liu, Yunpu Ma, Hao Shen, Hao Sun, Beibei Wang, Fangyijie Wang, Hao Wang, Haoran Wang, Yang Wang, Yifeng Wang, Z...

    work page 2025

  5. [5]

    Reinforcement mid-training, 2025

    Yijun Tian, Shaoyu Chen, Zhichao Xu, Yawei Wang, Jinhe Bi, Peng Han, and Wei Wang. Reinforcement mid-training, 2025. URLhttps://arxiv.org/abs/2509.24375

    work page arXiv 2025

  6. [6]

    Self-evolving multi-agent systems via textual backpropagation

    Xiaowen Ma, Yunpu Ma, Chenyang Lin, Sikuan Yan, Jinhe Bi, Zixuan Cao, Yijun Tian, Volker Tresp, and Hinrich Schuetze. Self-evolving multi-agent systems via textual backpropagation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors,Findings of the Association for 11 Computational Linguistics: ACL 2026, pages 9918–9951, San Dieg...

    work page doi:10.18653/v1/2026.findings-acl 2026

  7. [7]

    On-policy distillation.Thinking Machines Lab: Connectionism,

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connectionism,

  8. [8]

    EchoRL: Reinforcement learning via rollout echoing

    Jinhe Bi, Aniri, Minglai Yang, Xingcheng Zhou, Wenke Huang, Sikuan Yan, Yujun Wang, Zixuan Cao, Michael Färber, Xun Xiao, Volker Tresp, and Yunpu Ma. EchoRL: Reinforcement learning via rollout echoing. InForty-third International Conference on Machine Learning, 2026. URL https: //openreview.net/forum?id=A6az59SGtF

    work page 2026

  9. [9]

    Learning to reason under off-policy guidance

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Information Processing Systems, volume 38, pages 117157–117186. Curran Associates, Inc., 2025. URLhttps://proce...

    work page 2025

  10. [10]

    Ozdaglar

    Mingyang Liu, Gabriele Farina, and Asuman E. Ozdaglar. UFT: Unifying supervised and reinforcement fine-tuning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=usOkGv1S7M

    work page 2025

  11. [11]

    Self-distilled reasoner: On-policy self-distillation for large language models, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026. URLhttps://arxiv. org/abs/2601.18734

    Pith/arXiv arXiv 2026

  12. [12]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yang Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. ArXiv, abs/2501.07301, 2025. URLhttps://api.semanticscholar.org/CorpusID:275470671

    Pith/arXiv arXiv 2025

  13. [13]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URLhttps://openreview.net/forum?id=7Bywt2mQsCe

    2021

  14. [14]

    Solving quantitative reasoning problems with lan- guagemodels

    Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guagemodels. InAliceH.Oh, AlekhAgarwal, DanielleBelgrave, andKyunghyunCho, e...

    work page 2022

  15. [15]

    OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings o...

    2024

  16. [16]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  17. [17]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=Ti67584b98

    2024

  18. [18]

    MMLU-pro: A more robust and challenging multi-task language understanding 12 benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding 12 benchmark. InThe Thirty-eight Conference on Neural Information Proces...

    work page 2024

  19. [19]

    SimpleRL-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. SimpleRL-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. InSecond Conference on Language Modeling, 2025. URLhttps://openreview.net/forum?id=vSMCBUgrQj

    work page 2025

  20. [20]

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps: //openreview.net/forum?id=NFM8F5cV0V

    work page 2025

  21. [21]

    Process reinforcement through implicit rewards.Transactions on Machine Learning Research,

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards.Transaction...

    work page

  22. [22]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  23. [23]

    The llama 3 herd of models, 2024

    Llama 3 Team. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  24. [24]

    Think outside the policy: In-context steered policy optimization

    Hsiu-Yuan Huang, Chenming Tang, Weijie Liu, Clive Bai, Saiyong Yang, and Yunfang Wu. Think outside the policy: In-context steered policy optimization. InFindings of the Association for Computational Linguistics: ACL 2026, pages 2758–2776, 2026. URLhttps://aclanthology.org/2026.findings-acl. 132/

    work page 2026

  25. [25]

    SFT-then-RL outperforms mixed-policy methods for LLM reasoning, 2026

    Alexis Limozin, Eduard Durech, Torsten Hoefler, Imanol Schlag, and Valentina Pyatkin. SFT-then-RL outperforms mixed-policy methods for LLM reasoning, 2026. URLhttps://arxiv.org/abs/2604.23747

    Pith/arXiv arXiv 2026

  26. [26]

    When more is less: Understanding chain-of-thought length in llms.ArXiv, abs/2502.07266, 2025

    Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms.ArXiv, abs/2502.07266, 2025. URLhttps://api.semanticscholar. org/CorpusID:276259519

    Pith/arXiv arXiv 2025

  27. [27]

    Don’t overthink it

    Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. Don’t overthink it. preferring shorter thinking chains for improved LLM reasoning, 2026. URLhttps://openreview.net/forum?id= nhUlA8iMkD

    work page 2026

  28. [28]

    LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation- steering

    Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation- steering. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for ...

    work page 2025

  29. [29]

    Prism: Self- pruning intrinsic selection method for training-free multimodal data selection.ArXiv, abs/2502.12119,

    Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Prism: Self- pruning intrinsic selection method for training-free multimodal data selection.ArXiv, abs/2502.12119,

    Pith/arXiv arXiv

  30. [30]

    Beyond nl2code: A structured survey of multimodal code intelligence,

    Xuanle Zhao, Qiushi Sun, Jingyu Xiao, Xuexin Liu, Haoyue Yang, Qiaosheng Chen, Xianzhen Luo, Jing Huang, Yufeng Zhong, Lei Chen, Shuai Fu, Zhenlin Wei, Jinhe Bi, Lei Jiang, Haibo Qiu, Siqi Yang, Peng Shi, Jian Hu, and Zhixiong Zeng. Beyond nl2code: A structured survey of multimodal code intelligence,

    work page

  31. [31]

    Spot! revisiting video-language models for event understanding.arXiv preprint arXiv:2311.12919, 2023

    Gengyuan Zhang, Jinhe Bi, Jindong Gu, Yanyu Chen, and Volker Tresp. Spot! revisiting video-language models for event understanding.arXiv preprint arXiv:2311.12919, 2023. 13

    Pith/arXiv arXiv 2023

  32. [32]

    Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo, Jinhe Bi, Henry Zhang, Haris Riaz, Xuan Zhang, Yunze Xiao, Bangya Liu, Tom Tang, Yunfei Zhao, Qunshu Lin, Zihan Wang, Minghao Liu, Michael Lingzhi Li, Yilun Du, Jesse Thomason, Rogerio Feris, Alex Pentland, and Zexue He. Dr. docbench: A comprehens...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  33. [33]

    URLhttps://api.semanticscholar.org/CorpusID:276421326

  34. [34]

    Kore: Enhancing knowledge injection for large multimodal models via knowledge-oriented augmentations and constraints, 2025

    Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, and Qing Li. Kore: Enhancing knowledge injection for large multimodal models via knowledge-oriented augmentations and constraints, 2025. URLhttps://arxiv.org/abs/2510.19316

    Pith/arXiv arXiv 2025

  35. [35]

    URLhttps://arxiv.org/abs/2606.15932

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Aditya Prakash, Yizhou Sun, and Wei Wang

    Guancheng Wan, Lucheng Fu, Haoxin Liu, Yiqiao Jin, Hui Yi Leong, Eric Hanchen Jiang, Hejia Geng, Jinhe Bi, Yunpu Ma, Xiangru Tang, B. Aditya Prakash, Yizhou Sun, and Wei Wang. Beyond magic words: Sharpness-aware prompt evolving for robust large language models with tare, 2025. URL https://arxiv.org/abs/2509.24130

    arXiv 2025

  37. [37]

    HYPERION: Fine-grained hypersphere alignment for robust federated graph learning

    Guancheng Wan, Xiaoran Shang, Yuxin Wu, Guibin Zhang, Jinhe Bi, Liangtao Zheng, Xin Lin, Yue Liu, Yanbiao Ma, Wenke Huang, and Bo Du. HYPERION: Fine-grained hypersphere alignment for robust federated graph learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=TZB6YT8Owr

    work page 2025

  38. [38]

    Alignsae: Concept-aligned sparse autoencoders, 2026

    Minglai Yang, Xinyu Guo, Zhengliang Shi, Jinhe Bi, Steven Bethard, Mihai Surdeanu, and Liangming Pan. Alignsae: Concept-aligned sparse autoencoders, 2026. URLhttps://arxiv.org/abs/2512.02004

    work page arXiv 2026

  39. [39]

    Can visual input be compressed? a visual token compression benchmark for large multimodal models, 2025

    Tianfan Peng, Yuntao Du, Pengzhou Ji, Shijie Dong, Kailin Jiang, Mingchuan Ma, Yijun Tian, Jinhe Bi, Qian Li, Wei Du, Feng Xiao, and Lizhen Cui. Can visual input be compressed? a visual token compression benchmark for large multimodal models, 2025. URLhttps://arxiv.org/abs/2511.02650

    arXiv 2025

  40. [40]

    Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm.Proceedings of the AAAI Conference on Artificial Intelligence, 40(12): 10306–10314, Mar

    Yujun Wang, Aniri , Jinhe Bi, Soren Pirk, and Yunpu Ma. Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm.Proceedings of the AAAI Conference on Artificial Intelligence, 40(12): 10306–10314, Mar. 2026. doi: 10.1609/aaai.v40i12.38000. URL https://ojs.aaai.org/index.php/ AAAI/article/view/38000

    work page doi:10.1609/aaai.v40i12.38000 2026

  41. [41]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601, 2024

    work page 2024

  42. [42]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  43. [43]

    Backdoor cleaning without external guidance in MLLM fine-tuning

    Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, and Mang Ye. Backdoor cleaning without external guidance in MLLM fine-tuning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=os4QYDf3Ms

    work page 2025

  44. [44]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pages 32694–32717, 2024

    work page 2024

  45. [45]

    Mined: Probing and updating with multimodal time-sensitive knowledge for large multimodal models, 2025

    Kailin Jiang, Ning Jiang, Yuntao Du, Yuchen Ren, Yuchen Li, Yifan Gao, Jinhe Bi, Yunpu Ma, Qingqing Liu, Xianhao Wang, Yifan Jia, Hongbo Jiang, Yaocong Hu, Bin Li, and Lei Liu. Mined: Probing and updating with multimodal time-sensitive knowledge for large multimodal models, 2025. URL https://arxiv.org/abs/2510.19457

    Pith/arXiv arXiv 2025

  46. [48]

    Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. InInternational Conference on Learning Representations, volume 2024, pages 5009–5042, 2024

    work page 2024

  47. [50]

    Generating sequences by learning to self-correct, 2022

    Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct, 2022. URLhttps://arxiv.org/abs/2211.00053. 14 Appendix A Prompt Templates Here we present the prompt templates used in ReflectRL. During on-policy training, all model inputs are structured using a shared sys...

    Pith/arXiv arXiv 2022

  48. [51]

    UFT Cosine + Cutoff g(t) =    ph, t < t w, pl + ∆p 2 h 1 + cos π t−tw td i , t w ≤t < tc, 0, t≥t c

    work page

  49. [52]

    Linear g(t) = ( ph −∆p· t td , t < td, pl, t≥t d

    work page

  50. [53]

    Cosine g(t) =    ph, t < t w, pl + ∆p 2 h 1 + cos π t−tw td−tw i , t w ≤t < td, pl, t≥t d

    work page

  51. [54]

    17 Figure 10Case Study: Reflective Reasoning (Success)

    Plateau + Linear g(t) =    ph, t < t w, ph −∆p· t−tw td−tw , t w ≤t < td, pl, t≥t d. 17 Figure 10Case Study: Reflective Reasoning (Success)

    work page

  52. [55]

    Plateau + Cutoff g(t) =    ph, t < t w, ph −∆p· t−tw td−tw , t w ≤t < td, pl, t d ≤t < tc, 0, t≥t c

    work page

  53. [56]

    The reference solution suggests there were some errors in the previous attempts. Recomputing:

    Inverse Sigmoid g(t) =p l + ∆p 1 + exp (k(t−t m)) wheret m = tw+td 2 andkcontrols the steepness of the curve. D Related Work Reinforcement Learning with Verifiable Rewards (RLVR)Recently, large language models and multi-modal LLMs (MLLMs) have demonstrated remarkable capabilities across various complex reasoning tasks [28, 29, 30, 31, 12, 32, 33, 34, 35, ...

    work page 2048

  54. [483]

    URLhttps://aclanthology.org/2026.findings-acl.483/

    work page 2026

  55. [2025]

    https://thinkingmachines.ai/blog/on-policy-distillation

    doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation

    doi:10.64434/tml.20251026

  56. [2026]

    URLhttps://openreview.net/forum?id=9SkkifLopZ

    ISSN 2835-8856. URLhttps://openreview.net/forum?id=9SkkifLopZ

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.