{"id":"dde8e3cf-b685-4198-9517-635dfdf8b419","arxiv_id":"2608.03972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ReflectRL repurposes failed expert reasoning traces as reflective scaffolding during RL and distillation training, then transitions the policy to direct reasoning, improving math and science benchmark scores.","lead":"A new training method, ReflectRL, teaches language models to reason by showing them high-quality failed solutions from stronger models and asking the model to find and fix the errors, then gradually trains it back to solving problems from scratch. It reports consistent accuracy gains on reasoning benchmarks with little extra compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control for Reflective-to-Direct transition: fixed reflective mixture could match cosine-decay gains, so Eq. 7's mechanism is unverified.","rationale":"The reader's weakest assumption and my stress-test converge on the same load-bearing concern: Eq. 7's transition is the core novel component, yet the paper provides no ablation that isolates it. The strongest claim—that ReflectRL improves reasoning by transferring reflective behavior to direct reasoning—would be false if a fixed reflective mixture yields the same or better direct accuracy, because then the cosine decay is not doing the work. This is not a fatal objection: the empirical gains are plausible and the paper has broad coverage, but the missing control is exactly what a conditional acceptance should require. I keep the reader's CONDITIONAL verdict (UNCHANGED) because the paper has independent support (multiple backbones, methods, and benchmarks), and the concern is a specific missing ablation rather than a demonstrated internal contradiction. The per-benchmark inconsistencies, such as GPQA decreases for DAPO+ReflectRL, are real but secondary; they weaken 'consistently improves' but do not overturn the average improvements. The proposed test would settle the mechanism question directly.","tokens_in":17914,"tokens_out":5979,"duration_ms":72127,"concrete_test":"Run the same RLVR recipe (Qwen2.5-Math-7B, GRPO, OpenR1-GNT-69k) with g(t) held constant at ph=0.5 for all 500 training steps, then evaluate the final policy with the direct prompt only. Compare this fixed-mixture direct accuracy against the cosine-decay run. If fixed mixture matches or exceeds cosine decay, the Reflective-to-Direct transition is unnecessary and Eq. 7's mechanism is unsupported; also record per-interface mean advantages during training to check whether direct rollouts are systematically penalized early.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism, Reflective-to-Direct Policy Transition (Eq. 7), assumes that gradually annealing the reflective rollout fraction to zero makes the policy internalize error-correction behavior that then survives in direct reasoning. The only evidence is Figure 6, which compares different decay shapes against a no-GNT baseline; it never compares against keeping a fixed reflective mixture throughout training. Without that control, the reported direct-mode gains could come from continued exposure to GNT hints, from the regularizing effect of mixed rollout groups, or from the higher average reward of reflective rollouts—not from 'compilation' of reflection into direct parameters. This matters because Eq. 9 computes advantages jointly over mixed direct/reflective groups; if reflective rollouts are systematically more often correct, direct rollouts receive systematically negative advantages early in training, so the policy may be learning to distrust the direct interface while the transition schedule only later compensates. The claimed transition from reflective to direct is thus not actually demonstrated. Additionally, in the OPD instantiation (Eq. 11), the terminal pl=0.05 with batch size 1024 leaves about 51 teacher-side reflective samples per batch, so the transition does not even reach direct-only in OPD, further weakening the claim that the schedule is causally responsible for the improvements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReflectRL, a plug-and-play framework for on-policy LLM reasoning training that repurposes incorrect expert trajectories ('Golden Negative Trajectories', GNTs) as contextual scaffolds. The method defines two prompt interfaces—Direct Reasoning and Reflective Reasoning—and, during training, anneals the fraction of reflective rollouts from a high initial proportion to a small terminal value via a cosine transition kernel (Eq. 7). ReflectRL is instantiated on top of GRPO-style RLVR by mixing reflective and direct rollouts in the same group-relative advantage computation (Eq. 9), and on top of On-Policy Distillation by using the GNT only as teacher-side privileged context (Eq. 11). Experiments span 9 benchmarks, 4 backbones, and 4 training methods, and the paper reports consistent accuracy gains, reduced response length, and higher entropy retention, alongside ablations on trajectory provenance, trajectory components, and transition-kernel shape.","tokens_in":18220,"tokens_out":6213,"duration_ms":71509,"significance":"If the claimed effect is real, ReflectRL would be a useful and low-cost way to convert abundant expert failures into training signal, with the notable strength that it leaves the base RLVR/OPD objectives unchanged and does not require online expert queries. The paper also contributes a released 69k GNT dataset and a reasonably thorough behavioral analysis (Reflection Gain, PRM profiling, component ablation). However, the current evidence does not fully establish the central mechanistic claim—that annealing from reflective to direct rollouts causes the policy to internalize error-correction behavior—or the 'consistently improves' claim, because the key control (fixed reflective mixture) is absent and the main tables report single-seed point estimates with several flat or negative entries. The idea is plausible and testable, so the paper is worth revising rather than rejecting.","major_comments":[{"comment":"The Reflective-to-Direct Policy Transition is the central mechanism, but the experiments never compare the cosine decay against a fixed reflective mixture (e.g., g(t)=0.5 throughout) or against a schedule matched for total reflective exposure. Figure 6 only compares different decay shapes against a no-GNT baseline. Consequently, the reported direct-mode gains could be caused by continued exposure to GNT hints, by the regularizing effect of mixed rollout groups, or by the higher average reward of reflective rollouts—not by 'compiling' reflection into direct parameters. A fixed-mixture control is required to support the claimed transition mechanism.","section":"§4.4, Figure 6, Eq. (7)"},{"comment":"In the OPD instantiation the default terminal proportion is pl=0.05; with a rollout batch of B=1024, this leaves about 51 teacher-side reflective samples per batch, so the schedule never reaches direct-only teacher context. The claim that the student distills reflective capability into the direct pathway is therefore not actually tested under the reported configuration. Please report OPD runs with pl=0 (or a hard cutoff to zero) and examine whether the OPD results are sensitive to the residual 5% reflective context.","section":"§3.4, Eq. (11), Appendix C"},{"comment":"The 'consistently improves' claim is not supported by single-seed point estimates. Several reported entries go in the opposite direction: DAPO+ReflectRL GPQA* drops from 16.2 to 14.7 in Table 1; OPD+ReflectRL MMLU-Pro drops from 41.6 to 38.1; and Qwen2.5-3B AIME25 drops from 3.44 to 2.08 in Table 2. Given the small test sets and stochasticity of RL training, at least 3 seeds (or a significance test) are needed for the main comparisons; otherwise the headline 'consistent improvement' is not established.","section":"Tables 1–2, §4.2"},{"comment":"The paper repeatedly describes the GNT signal as 'dense', but in the RLVR instantiation (Eq. 9) the reward remains the binary verifier r(q,o); the GNT only changes the prompt context. No token-level, process-level, or auxiliary dense reward is introduced in the RLVR variant. The 'dense' characterization is accurate only for the OPD variant (Eq. 11), where the teacher distribution provides token-level KL targets. Please revise the wording or provide evidence that reflective prompting alone creates denser credit assignment.","section":"Abstract, §3.3, §D"}],"minor_comments":[{"comment":"The transition kernel uses τ(t) but this normalized progress is not defined in the main text. Define it in Eq. (7) or point immediately to Appendix C; the current text says 'Appendix C provides the exact schedule' but the main equation is incomplete.","section":"Eq. (7)"},{"comment":"The right panel labels the schedulers with generalized equations, but the notation is inconsistent with Appendix C: 'UFT Cosine + Cutoff' in the figure has denominator td, while 'Cosine' uses td−tw. Align the two presentations and specify the parameter values used for each curve.","section":"Figure 6"},{"comment":"The Reflection Gain curves in (a) and the PRM profiles in (b) are shown without error bars or seed information. Since these measurements motivate the entire framework, add at least the number of runs or confidence intervals, and clarify whether the curve is a running mean over a single training run.","section":"Figure 1"},{"comment":"The Llama-3.1-8B GRPO baseline is far below its own no-RL baseline on the ID average (3.89 vs. 7.05). This suggests an unstable or undertrained GRPO baseline; please comment on this, since it affects how much credit ReflectRL deserves for 'recovering' performance.","section":"§4.1 and Table 2"},{"comment":"The discussion after Eq. (9) correctly notes that with binary rewards normalization preserves the sign of advantages in non-degenerate groups, but it should also state that if all rollouts in a mixed group are correct (or all incorrect), the advantages are identically zero and the GNT context provides no learning signal in that group.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper does not currently provide a clear code/data availability statement beyond placeholder icons. For a methods paper whose value depends on reproducibility, I would encourage the editor to make code and the OpenR1-GNT-69k dataset release a condition of acceptance. The missing fixed-mixture control should be a required revision, not merely a suggested experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core idea is good, the empirical sweep is broad, and the paper deserves a serious referee. But the specific mechanism they push — Reflective-to-Direct Policy Transition — is not actually isolated by the experiments, and the headline numbers are less uniform than the abstract claims.\n\nWhat's new: using expert failure trajectories as contextual scaffolds that elicit reflection, rather than discarding them, is a neat twist on the usual golden-trajectory recipe. The Reflection Advantage analysis in Figure 1 is the strongest part: the PRM-based profiling and the causal component ablation (valid prefix vs. error region) give real evidence about where the signal lives. The plug-and-play framing is also genuinely useful — no auxiliary losses, drop-in for GRPO, DAPO, and OPD — and the overhead estimate is honest. OpenR1-GNT-69k, if actually released, is a useful artifact.\n\nWhere I'd push back. First, the empirical claims outrun the evidence. Every result is a single run with no error bars. Several per-benchmark numbers go the wrong way — DAPO+ReflectRL drops on GPQA* relative to DAPO (14.7 vs 16.2) — so 'consistently improves' is simply not supported. The ARC-c jumps (e.g., 22.5 to 65.4 for GRPO) are dramatic enough that I want repeated seeds and a contamination check before believing them. Second, the transition mechanism is under-tested. Figure 6 compares decay shapes against each other and against a no-GNT baseline, but never against keeping a fixed reflective mixture throughout training. Without that control, the gains could come from continued exposure to GNT hints or from the mixed-group advantage computation, not from 'compilation' of reflection into direct parameters. The OPD version has a related issue: the terminal pl=0.05 means the teacher still gets reflective context at the end, so 'transition to direct' is not fully realized there. Third, the paper omits Reflexion from related work, which is the most obvious prior on reflecting on failures. That citation gap should be fixed. Fourth, the release artifacts are mentioned with icons but no links in the text; if the dataset and code are public, the authors should make them accessible for review.\n\nNone of this is fatal to the central idea. The empirical gains are directionally plausible, and the Reflection Advantage analysis is solid descriptive work. But the mechanism claim is over-argued, and the single-seed numbers are too fragile to support the current level of confidence.\n\nWho should read this: anyone working on RL post-training, learning from mistakes, or self-correction. I'd send it to a careful reviewer — the idea is worth engaging with, and a revision that adds seeds, error bars, the fixed-mixture control, and the missing artifacts could turn this into a solid paper. I would not desk-reject it.","headline":"A genuinely useful idea — expert failures as reflective hints with an annealed curriculum — but the claimed transition mechanism is under-tested and the 'consistent improvements' headline overreaches.","tokens_in":18773,"tokens_out":3656,"would_cite":false,"duration_ms":38833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReflectRL claims that failed expert trajectories can be turned into dense training signals by reflective-to-direct reasoning, improving model accuracy across nine benchmarks.","keywords":["Golden Negative Trajectories","Reflective Reasoning","Reflective-to-Direct Policy Transition","On-policy reinforcement learning","GRPO","On-policy distillation","LLM reasoning","Process reward models"],"falsifier":"Train ReflectRL with the reflective rollout fraction held constant at its initial value for the entire run, then evaluate under direct prompting with no GNT. If fixed-reflection training matches or exceeds the cosine-decay variant, the gradual transition is not what transfers the skill, and the reported direct-reasoning gains are better explained by ongoing exposure to GNT hints.","tokens_in":17782,"feed_emoji":"🧠","tokens_out":7675,"duration_ms":80786,"temperature":0.7,"pith_summary":"This paper tries to establish that the failed reasoning attempts of a stronger model—what it calls Golden Negative Trajectories—are not waste. Even though such trajectories are wrong overall, they contain a long valid reasoning prefix and a localized error, and asking the training model to critique and repair that trace can be easier and more effective than asking it to solve the same hard problem from scratch. The paper names this the Reflection Advantage and builds ReflectRL, a lightweight training addition that feeds these failed traces as context during on-policy reinforcement learning or on-policy distillation, then gradually phases the context out. Across nine benchmarks, four model backbones, and four on-policy training methods, ReflectRL reports consistent accuracy gains—including outside the math training distribution—with shorter responses and lower per-step training cost than the base methods. The reason to care is that hard problems are exactly where expert models fail, so recovering useful signal from those failures widens the set of problems on which trajectory-guided training can help.","feed_headline":"Reflecting on expert mistakes beats solving from scratch","feed_subtitle":"Failed expert traces become reflection prompts, lifting model accuracy across 9 math and science benchmarks.","key_machinery":"Two named objects carry the argument. First is the Reflection Advantage, formalized as the difference in expected correctness between reflective prompting (question plus golden negative trajectory, asking the model to identify errors and repair them) and direct prompting (question only). Second is the Reflective-to-Direct Policy Transition, a cosine-decay kernel g(t)=p_l+(p_h−p_l)/2·[1+cos(πτ(t))] that sets how many rollouts in each mixed group receive the reflective context; in OPD the same schedule governs how many teacher contexts are reflective, while the student always sees direct prompts. These turn a discarded negative sample into a scaffold for dense learning signals without adding l","core_discovery":"The paper claims that expert failures contain a transferable reasoning structure: a valid prefix plus a localized error region. When the learner is prompted to reflect on such a trace, its success rate on that question rises markedly compared with direct solve-from-scratch prompting; the paper measures this as a positive Reflection Gain. ReflectRL exploits this by constructing training prompts that include the failed trajectory, training with the standard GRPO or OPD objective unchanged, and then using a cosine-decay transition kernel to reduce the fraction of reflective rollouts to zero. The intended effect is that the error-correction and repair behavior learned under reflection is compile","pith_inferences":["Beyond the paper: the causal analysis suggests selecting or synthesizing GNTs by process-reward quality—maximizing valid-prefix length while keeping a localized error—could improve data efficiency, rather than relying on whatever failed traces the expert happens to produce.","Beyond the paper: a model could bootstrap its own high-quality failures, filtered by a process reward model, as GNTs, reducing or removing dependence on a stronger teacher.","Beyond the paper: the observed response-length compression during the transition raises a testable hypothesis that reflective training acts as an implicit reasoning-precision regularizer, which could be probed on non-math reasoning tasks.","Beyond the paper: the teacher-side privileged-context recipe in the OPD variant suggests a general pattern for distilling repair skills in other domains—such as code or tool-use—where a flawed trace can guide the teacher while the student learns only the direct interface."],"forward_implications":["Hard problems with no correct expert trace can still be used for training, as long as the expert produces a structured near-miss with a usable prefix and error.","ReflectRL can be layered onto existing GRPO-family and on-policy distillation pipelines without modifying their objectives, reward computation, or inference interface.","Training can become cheaper per step even as accuracy rises, because reflective prompts are cheap to prefill and the model's continuations are substantially shorter.","The transition schedule maintains higher policy entropy, delaying the collapse to a narrow response mode that limits further learning.","Improvements outside the math distribution suggest the reflect–repair–solve procedure teaches a generalizable error-correction behavior, not memorized problem patterns."],"supporting_citations":[{"why":"Defines the GRPO objective and supplies the strong-expert trajectories from which GNTs are drawn.","marker":"[1]"},{"why":"DAPO is one of the RLVR baselines ReflectRL is integrated into and compared against.","marker":"[2]"},{"why":"Defines the on-policy distillation objective that ReflectRL extends with teacher-side reflective context.","marker":"[7]"},{"why":"EchoRL is a baseline that recovers signals from advantage-degenerated groups; ReflectRL is measured against it.","marker":"[8]"},{"why":"Uses correct expert trajectories to guide on-policy rollouts, the positive-trajectory approach whose failure mode ReflectRL addresses.","marker":"[9]"},{"why":"Unifies supervised and RL fine-tuning on correct trajectories, representing the prior paradigm that discards expert failures.","marker":"[10]"},{"why":"Prior on-policy self-distillation method whose teacher-student setup the OPD variant builds on.","marker":"[11]"},{"why":"Supplies the process reward model used to profile step-level quality of GNTs versus other failure sources.","marker":"[12]"}],"fun_headline_variants":["Turn expert failures into reflective training wins","ReflectRL: Learn from wrong answers to reason better","Mistake-driven learning boosts model reasoning","From error to insight: ReflectRL's reflective transition","Failed expert paths become powerful reflection prompts"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that gradually reducing reflective rollouts to zero lets the policy internalize error-correction behavior so that it persists in direct reasoning; if the gains actually come from continued exposure to expert hints during training, the transition mechanism would not be doing the work claimed.","fun_headline_variants_meta":{"raw":{"variants":["Turn expert failures into reflective training wins","ReflectRL: Learn from wrong answers to reason better","Mistake-driven learning boosts model reasoning","From error to insight: ReflectRL's reflective transition","Failed expert paths become powerful reflection prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1340,"prompt_tokens":708,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":452,"tokens_out":632,"duration_ms":6997,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:48:10.177269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ReflectRL with the reflective rollout fraction held constant at its initial value for the entire run, then evaluate under direct prompting with no GNT. If fixed-reflection training matches or exceeds the cosine-decay variant, the gradual transition is not what transfers the skill, and the reported direct-reasoning gains are better explained by ongoing exposure to GNT hints.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GRPO objective and supplies the strong-expert trajectories from which GNTs are drawn."},{"cited_title":"DAPO: An open-source LLM reinforcement learning system at scale","cited_arxiv_id":null,"evidence_quote":"DAPO is one of the RLVR baselines ReflectRL is integrated into and compared against."},{"cited_title":"EchoRL: Reinforcement learning via rollout echoing","cited_arxiv_id":null,"evidence_quote":"EchoRL is a baseline that recovers signals from advantage-degenerated groups; ReflectRL is measured against it."},{"cited_title":"Learning to reason under off-policy guidance","cited_arxiv_id":null,"evidence_quote":"Uses correct expert trajectories to guide on-policy rollouts, the positive-trajectory approach whose failure mode ReflectRL addresses."},{"cited_title":"Ozdaglar","cited_arxiv_id":null,"evidence_quote":"Unifies supervised and RL fine-tuning on correct trajectories, representing the prior paradigm that discards expert failures."}],"review_version":1}