{"id":"66f20587-9826-4951-b8e6-916e92180bb2","arxiv_id":"2608.09123","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RISE-RL uses rubric criteria that are frequently missed to generate and filter guided trajectories, weights tokens by natural-policy support, and optimizes them through a separate auxiliary loss, beating Rubric-RL on all eight benchmarks tested.","lead":"This paper introduces RISE-RL, a reinforcement learning method that uses the rubric criteria an LLM judge says a response missed to generate targeted practice examples, then trains on only the improved ones. It reports consistent score gains over standard rubric-based RL on writing, chat, health, and science benchmarks with 4B and 14B models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The support-weighting in Eqs. 9-11 conflates genuine capability gaps with off-policy distribution shift: teacher-forcing a rubric-conditioned trajectory under the original prompt makes every token downstream of a hint-induced divergence low-probability, so the auxiliary loss may reinforce…","rationale":"The reader's weakest assumption is exactly the load-bearing step: Eq. 9-11 assumes that low token probability under the original prompt identifies a real capability gap rather than the distribution shift induced by the privileged suffix. I agree and highlight why the shift is compounding: because p_{j,t} conditions on a prefix that was itself generated under q_priv, the first hint-induced divergence depresses the probability of all later tokens, so the support weight cannot isolate the criterion-related behavior. This concern is central because the paper's mechanistic claims and the policy-support-shaping ablation depend on that identification. I do not think it overturns the empirical comparisons, but it means the reported gains are not yet attributable to the stated mechanism; a diagnostic that localizes the loss to criterion-expressing tokens would settle the question. The verdict remains CONDITIONAL: the method is promising and the empirical claims are worth checking with error bars, compute-matched baselines, and this token-level attribution test.","tokens_in":27871,"tokens_out":6955,"duration_ms":81709,"concrete_test":"On a held-out batch of retained privileged trajectories, annotate the minimal token spans that directly express the selected missed criteria (e.g., 'consult an obstetric provider', 'spontaneous labor', 'individual pregnancy history'), using the rubric definitions and a critic or human annotators. Compute what fraction of the total Lguide weight (eA_j/|o_ref| times -log ρ) falls on criterion-related spans versus all other tokens, and compute the correlation between token-level -log ρ and criterion membership. If non-criterion tokens receive a substantial share of the loss, or the correlation is weak, then low p under q is substantially off-policy drift, and Eqs. 9-11 are not cleanly targeting capability gaps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RISE-RL's central mechanism rests on interpreting p_{j,t} = π_θ(o_ref_{j,t} | q, o_ref_{j,<t}) in Eq. 9 as a measure of how weakly the natural policy supports the high-value behavior. But o_ref is sampled from q_priv = q ⊕ C_fb (Eq. 6). Once the privileged suffix changes even one early token of the response, teacher-forcing under q conditions on a prefix that is off-distribution for π_θ(·|q); all later token probabilities inherit that shift. Low p_{j,t} then reflects the cumulative divergence of the whole rubric-conditioned continuation from the natural-prompt distribution, not the marginal absence of the missed criterion. The ρ transform in Eq. 10 and the weighted NLL in Eq. 11 convert that low p into a strong gradient to reproduce the entire hinted trajectory under q. The paper's control 'uniform vs support-weighted' (Fig. 7b) only shows that emphasizing low-p tokens helps; it does not show those tokens are the missed criteria, nor does any experiment separate a genuine capability gap from off-policy lexical/style drift. Since the headline claims the selective mechanism, not extra sampling or conditioning, is the driver, this unvalidated identification is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes RISE-RL, a rubric-informed selective exploration method for open-ended reinforcement learning. The method identifies rubric criteria that are frequently failed by natural rollouts, appends the top-M such criteria to the prompt to sample privileged trajectories, filters those trajectories by whether their complete-rubric reward exceeds the mean natural reward, re-evaluates the retained trajectories under the original prompt, and weights each token by a policy-support factor rho = p/(p+gamma). The resulting auxiliary loss is optimized separately from GRPO and is removed at a training step s_switch. Experiments on Qwen3-4B and Qwen3-14B across writing, chat, health, and science report higher mean scores than Rubric-RL on all eight downstream benchmarks, with average gains of 1.3 and 3.3 points, plus ablations on the guidance coefficient, number of injected criteria, support weighting, and coupled versus decoupled optimization.","tokens_in":28133,"tokens_out":8988,"duration_ms":92441,"significance":"RISE-RL targets a genuine problem: scalar rubric rewards obscure which criteria are persistently missed, and unguided exploration may fail to discover rare high-reward modes. The decoupled auxiliary objective is a clean way to keep privileged trajectories out of the GRPO advantage estimator, and the training-dynamics analysis in Figure 3, the criterion-level health case in Table 12, and the full CreativeWriting-V3 criterion-level results in Table 10 are valuable evidence that targeted guidance can be internalized and then removed. The evaluation spans four domains and includes objectively scored benchmarks such as MedQA and GPQA-Diamond, which goes beyond style-level comparisons. The method is described precisely enough to be reimplementable, although no code is provided. The core weakness is that the mechanism claimed to drive the gains — policy-support weighting of low-probability tokens in Eq. (11) — is not actually validated against the off-policy distribution-shift alternative.","major_comments":[{"comment":"Equation (9) computes p_{j,t} by teacher-forcing under the original prompt q a trajectory that was sampled under q_priv = q ⊕ C_fb. Once the privileged suffix changes an early token, every later conditional probability is evaluated on an off-policy prefix, so a low p_{j,t} conflates the cumulative distribution shift of the whole hinted continuation with the model's weak support for the specific missed criterion. Equation (11) then up-weights all low-support tokens, including lexical or stylistic artifacts of the hint. The ablation in Figure 7(b) shows only that support weighting helps relative to uniform weighting; it does not show that the low-p tokens correspond to the targeted rubric criteria. A control that weights only criterion-relevant spans, or a comparison against a version that applies the same support weights to equal-reward trajectories sampled without the hint, is needed to establish the paper's central attribution of the gains to 'policy support shaping.'","section":"Method, Eqs. (9)–(11)"},{"comment":"Appendix A.3 promises that 'each selected checkpoint is evaluated independently three times, and we report the mean and standard deviation across the three evaluation runs,' but Table 1 reports only means and no standard deviations. Several headline RISE-RL versus Rubric-RL differences are small, for example 20.89 versus 20.76 on Arena-Hard-v2 at 4B and 52.32 versus 52.00 on HealthBench at 4B. The claim that RISE-RL beats Rubric-RL on every benchmark is a mean-score claim, but without the promised variance information the reader cannot assess whether these differences are stable across evaluation runs or checkpoints. Please report the standard deviations, confidence intervals, or individual run values for the main table.","section":"Table 1 and Appendix A.3"},{"comment":"During the guided stage, RISE-RL spends roughly twice the rollout budget of Rubric-RL (G natural rollouts plus G privileged re-rollouts), and no baseline matches this compute with unguided rollouts. The paper states that the gains are not merely from extra sampling or rubric conditioning, but the support-weighting and coupled-versus-decoupled ablations keep the extra sampling fixed. A compute-matched Rubric-RL variant, for example 2G natural rollouts per step or a second unguided re-rollout branch, is needed to rule out the trivial explanation that additional on-policy samples alone drive the improvement.","section":"Experiments, 'Main Results'"},{"comment":"The guidance-removal step s_switch is set 'based on a preliminary run' near the onset of the plateau in the reward-gap curve, but the manuscript does not specify which domain/model the preliminary run used, how the plateau was detected, or how sensitive the final scores are to s_switch. Since the two-stage dynamics in Figure 3 and the comparison against persistent guidance are a stated contribution, the paper should report the actual s_switch values used for each domain and provide at least a small sensitivity analysis over early, chosen, and late removal.","section":"Decoupled Optimization and Guidance Removal"}],"minor_comments":[{"comment":"There are two empty cross-references: 'provided in Section .' appears in the RuscaRL implementation paragraph and again in the training-dynamics appendix; both should be filled in.","section":"Appendix A.2 and Appendix B.1"},{"comment":"The privileged-prompt construction says M=5 criteria are injected, but it is not stated what happens when fewer than M criteria are violated in a given natural-rollout group; please clarify whether the hint then contains only the violated criteria or is padded in some way.","section":"Section D.1"},{"comment":"The failure-count columns in Table 12 are difficult to align in the current formatting, and the stated column totals do not obviously match the row entries; please reformat the table and verify the totals.","section":"Table 12"},{"comment":"The manuscript does not state how the 'selected checkpoint' for Table 1 was chosen, for example final training step versus best validation step; this should be specified because it affects the comparability of the reported means.","section":"Main Results, Table 1"},{"comment":"Figure 5 uses criterion labels such as R0 and R9 that are defined only in Table 11; the figure caption should either restate the criteria or clearly point to the table.","section":"Figure 5 and Table 11"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies on RubricHub both as the training corpus and as the source of the claim that gpt-oss-120b has high agreement with human judgment, and at least one author of the present submission appears to overlap with the RubricHub author list. This is not a correctness issue because the downstream evaluation benchmarks are external, but the overlap should be disclosed and the independent reliability of the judge should be stated more carefully. The main technical concern in my report, the off-policy interpretation of Eq. (9), is addressable with additional control experiments; if those experiments fail, the paper's mechanism claim would need to be weakened substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful RL recipe with a decent chance of being a real advance, and the authors did the right experiments at two scales. But the paper's central interpretive claim—that the support-weighting step identifies and reinforces the specific missed rubric criteria—is not supported by the evidence. The stress-test note is on point.\n\nWhat's new: the combination of (i) selecting rubric criteria by weighted failure frequency, (ii) filtering privileged trajectories by full-rubric reward relative to natural rollouts, (iii) re-evaluating under the original prompt and weighting tokens by p/(p+γ), and (iv) a decoupled auxiliary loss with scheduled removal. Each piece exists in prior work, but this specific assembly is not in RuscaRL, RGR-GRPO, LUFFY, or OPD. The empirical evaluation is broad: four domains, two model scales, transfer to multiple-choice benchmarks. The gains over Rubric-RL are consistent and the ablations are well designed.\n\nWhere it gets soft. Table 1 reports only means of three eval runs; appendix A.3 promises standard deviations. For a claim of 'highest mean on every benchmark,' the missing error bars matter, especially where the RISE-RL vs Rubric-RL gap is under 0.5 points at 4B. The guidance-removal step s_switch is set from a preliminary run and not reported; that is a free parameter that could hide a lot. RISE-RL uses roughly twice the rollouts and judge calls of Rubric-RL, and there is no compute-matched baseline.\n\nThe bigger issue is the load-bearing interpretation of Eq. 9–11. Teacher-forcing the rubric-conditioned trajectory under the original prompt makes p_{j,t} low for every token downstream of a hint-induced divergence, not just the missed criteria. The loss then up-weights the entire off-policy continuation. The uniform-vs-support ablation shows the weighting helps, but it does not show the low-p tokens correspond to the targeted criteria. So the paper has not separated 'reinforcing a genuine capability gap' from 'imitating a useful but off-policy trajectory.' That does not kill the method—the empirical gains could still come from useful imitation with a high-reward filter—but it shifts the title claim. The authors should be asked to test this directly, e.g., by comparing token weights against human-annotated missed criteria, or by ablating the auxiliary loss to only tokens that match the selected criteria.\n\nMy take: the paper deserves a serious referee. The recipe is clearly specified, the experiments are wide, and the gains are plausible. But the revision needs error bars, s_switch and Amax values, a compute-matched Rubric-RL comparison, and a diagnostic that separates distribution shift from capability gaps. I'd send it out.","headline":"A practical rubric-guidance recipe with consistent empirical gains, but the mechanism that supposedly makes it selective is not actually isolated from plain off-policy imitation.","tokens_in":28672,"tokens_out":3063,"would_cite":true,"duration_ms":32053,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RISE-RL shows that open-ended reinforcement learning improves when the model is guided toward rubric criteria it repeatedly fails, using quality-filtered privileged trajectories and a support-weighted auxiliary loss.","keywords":["open-ended reinforcement learning","rubric-based RL","selective exploration","policy support shaping","privileged trajectories","GRPO","LLM alignment","creative writing diversity"],"falsifier":"A direct test would look at the low-probability tokens that the support factor upweights in retained guidance trajectories: if these tokens do not disproportionately implement the injected failed criteria, such as being mostly stylistic fillers or rare idioms, then the support factor is not measuring the capability gap the paper claims and the gains would not be expected to transfer to new prompts. A sharper quantitative version would retrain with the support factor replaced by a uniform weight on all retained tokens while keeping the reward filter; if the uniform-weight variant matches RISE-RL, the paper's central mechanism is the reward filter alone, not policy support shaping.","tokens_in":27627,"feed_emoji":"🎯","tokens_out":12873,"duration_ms":115236,"temperature":0.7,"pith_summary":"RISE-RL claims that open-ended reinforcement learning becomes more effective when the model is guided toward the specific rubric criteria it keeps failing, instead of compressing all criterion feedback into a single scalar reward. The method appends the most-failed criteria to the prompt and samples privileged trajectories, meaning rollouts conditioned on those criteria; it keeps only the ones whose complete-rubric reward beats the mean natural-rollout reward, then re-scores them under the original prompt so that the auxiliary loss concentrates on tokens the natural policy still supports only weakly. That guidance is optimized separately from the group-relative objective and is switched off once its marginal benefit plateaus, after which training continues with unguided exploration. Across Qwen3-4B and Qwen3-14B on writing, chat, health, and science benchmarks, the paper reports higher mean scores than standard Rubric-RL on every evaluated benchmark under guidance-free evaluation, with average gains of 1.3 and 3.3 points, plus gains on objectively scored medical and scientific questions and a 5.8% diversity improvement in creative writing. If the paper is right, selective internalization of hard-to-discover, high-value behaviors is an effective alternative to scalar rubric rewards and uniform distillation.","feed_headline":"Targeted rubric hints beat scalar rewards on all 8 benchmarks","feed_subtitle":"Failing criteria become hints; average scores rise 3.3 points at 14B scale while diversity increases too.","key_machinery":"The central object is the policy support factor $\\rho_{j,t} = p_{j,t}/(p_{j,t}+\\gamma)$, where $p_{j,t}$ is the current policy's probability of token $t$ in a retained privileged trajectory when that trajectory is re-scored token by token under the original prompt, without the injected criteria. Because $-\\log\\rho_{j,t}$ grows as $p_{j,t}$ shrinks, the factor upweights exactly the tokens that appear in a high-reward guided response yet remain unlikely under the natural policy; the clipped trajectory gain $\\tilde{A}_j = \\operatorname{clip}(A_j^\\text{ref}, 0, A_\\max)$ keeps only reward-improving trajectories in $\\mathcal{L}_\\text{guide} = -\\frac{1}{|\\mathcal{O}_\\text{ref}|}\\sum_j \\tilde{A}_j \\frac{1}{|o_j|}\\sum_t \\log\\rho_{j,t}$. This support weighting runs through a separate auxiliary objective added to the GRPO loss, so guided trajectories never enter the group-relative advantage normalization of natural rollouts, and the whole guidance branch is switched off when the privileged-versus-natural reward gap stops shrinking.","core_discovery":"On the paper's own terms, the central discovery is that selectively feeding a policy its own persistent rubric failures produces rubric alignment that survives guidance-free evaluation. After a group of natural rollouts is scored criterion by criterion, RISE-RL ranks the criteria by importance-weighted failure frequency, appends the top $M$ to the prompt, samples privileged responses, and retains only those whose complete-rubric reward exceeds the average natural reward. Each retained trajectory is then re-scored token by token under the original prompt, without the injected criteria, and the resulting token probabilities feed $\\rho_{j,t} = p_{j,t}/(p_{j,t}+\\gamma)$, so tokens that appear in a high-reward trajectory but are weakly supported by the current policy receive larger guidance. The auxiliary loss $\\mathcal{L}_\\text{guide}$ maximizes these support factors scaled by the clipped reward improvement, while the natural branch runs Group Relative Policy Optimization (GRPO) as usual; guidance is removed once the privileged-versus-natural reward gap plateaus. The experiments claim that this selective, decoupled, time-limited guidance is what lets the policy internalize behaviors that unguided exploration rarely finds, including safety-relevant health criteria and rare creative modes, rather than merely imitating rubric-conditioned responses.","pith_inferences":["The authors leave implicit that the same selective-guidance loop could run with rubrics produced by the judge itself when no curated rubric dataset exists; testing RISE-RL on model-generated criteria would show whether the benefit depends on RubricHub's rubric quality.","Because $\\rho$ increases with $p$, the support factor acts as a soft inverse-probability weight; one could vary $\\gamma$ per domain to see whether creative writing, where low-probability tokens are often stylistic, needs a different sensitivity than health or science.","The guidance switch is set from a preliminary run, but the paper measures the privileged-versus-natural reward gap throughout, so an online rule that removes guidance when that gap flattens is a natural next step that would remove the need for a fixed $s_\\text{switch}$.","The transfer to MedQA and GPQA-Diamond hints that guided trajectories may serve partly as a knowledge regularizer; a probe of factual retention on held-out domain questions would test that interpretation."],"forward_implications":["RISE-RL, if correct, gives a drop-in alternative to scalar rubric reward: training with criterion-level priority selection, reward filtering, and support shaping raises mean benchmark scores by 1.3 points at 4B and 3.3 points at 14B over standard Rubric-RL in the paper's evaluations.","Open-ended rubrics and verifiable tasks are not in tension: the same health and science training improves MedQA by 3.3 points and GPQA-Diamond by 3.6 points, so rubric-guided alignment can transfer into objectively scored knowledge.","Guidance removal is part of the benefit: persistent guidance leads to fluctuating entropy and a late-training performance drop, whereas RISE-RL keeps improving after guidance is withdrawn, suggesting a two-stage expand-then-consolidate dynamics.","Diversity need not be sacrificed: creative-writing evaluations show a 5.8% diversity gain over the base model, while Rubric-RL degrades coherence and style, so targeted guidance can expand rather than collapse the policy distribution.","Decoupling matters: placing the same filtered privileged trajectories inside the group-relative objective underperforms RISE-RL on both writing and health ablations, implying that keeping guided learning outside natural advantage estimation is a real component of the result."],"supporting_citations":[{"why":"Supplies the Group Relative Policy Optimization (GRPO) objective used for the natural rollout branch and the group-relative advantage estimator that RISE-RL keeps separate from its guidance loss.","marker":"Shao et al. 2024"},{"why":"Defines Rubric-RL, the scalar-reward baseline that RISE-RL is compared against and whose reward aggregation it diagnoses as losing criterion-level information.","marker":"Gunjal et al. 2025"},{"why":"Provides RubricHub, the source of prompts, instance-specific weighted rubrics, and the judge used to compute criterion priorities and rewards.","marker":"Li et al. 2026a"},{"why":"Supplies the saturating transformation $\\rho = p/(p+\\gamma)$ and the policy shaping idea that RISE-RL adopts for token-level support weighting.","marker":"Yan et al. 2026"},{"why":"Defines RuscaRL, the rubric-scaffolded baseline whose coupled optimization of guided and natural responses motivates RISE-RL's decoupled auxiliary objective.","marker":"Zhou et al. 2025"},{"why":"Defines the OPD baseline and documents that large teacher-student gaps can hurt distillation, which RISE-RL cites when explaining why uniform teacher matching is insufficient.","marker":"Li et al. 2026b"},{"why":"Raises the scaffold-removal and coupled-optimization concerns about RGR-GRPO that RISE-RL uses to motivate filtering and decoupling of guided trajectories.","marker":"Bi et al. 2025"}],"fun_headline_variants":["Rubric failures become targeted hints that beat scalar rewards","Selective rubric hints outperform scalar rewards on all 8 benchmarks","Failing criteria guide RL to higher scores that persist after hints","Criterion-level hints beat scalar rewards in open-ended RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that when a filtered high-reward guided response is re-scored under the original prompt, a low token probability marks a genuine capability gap that training on that token will fix, rather than an off-distribution artifact of the privileged prompt that will not transfer once guidance is removed.","fun_headline_variants_meta":{"raw":{"variants":["Rubric failures become targeted hints that beat scalar rewards","Selective rubric hints outperform scalar rewards on all 8 benchmarks","Failing criteria guide RL to higher scores that persist after hints","Criterion-level hints beat scalar rewards in open-ended RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2957,"prompt_tokens":1075,"completion_tokens":1882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":1826}},"tokens_in":691,"tokens_out":1882,"duration_ms":14018,"temperature":1.0,"reasoning_tokens":1826,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:05:08.085829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would look at the low-probability tokens that the support factor upweights in retained guidance trajectories: if these tokens do not disproportionately implement the injected failed criteria, such as being mostly stylistic fillers or rare idioms, then the support factor is not measuring the capability gap the paper claims and the gains would not be expected to transfer to new prompts. A sharper quantitative version would retrain with the support factor replaced by a uniform weight on all retained tokens while keeping the reward filter; if the uniform-weight variant matches RISE-RL, the paper's central mechanism is the reward filter alone, not policy support shaping.","supporting_citations":[],"review_version":1}