{"id":"82f3bc61-4bba-4d30-977b-265a9d68b3d2","arxiv_id":"2608.09226","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"REST co-trains a few-step, CFG-free student on the reward-scored rollout trajectories of an RL diffusion teacher, using advantage-weighted imitation to preserve reward gains.","lead":"This paper introduces REST, a training scheme that makes a small student model learn from the reward-scored intermediate steps of an RL-trained text-to-image teacher, so the final model can generate good images in 8 steps without guidance. It combines reward alignment and few-step distillation into one stage, claiming similar quality to a 40-step teacher at less than a quarter extra training cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-overhead claim is contradicted by the paper's own timing breakdown: the student branch adds ~35% overhead over pure RL, not 'below 25%'.","rationale":"The reader's weakest_assumption focuses on the CFG-to-CFG-free distillation mechanism, which is indeed a central methodological risk. However, the paper's own ablations (Tables 3 and 4) provide empirical support for the mechanism, and the qualitative results suggest it works at 8 steps. The most load-bearing and objectively verifiable flaw is the training-cost arithmetic: the abstract's 'below 25%' claim is directly contradicted by the percentages given in Section 4.4. This is an internal inconsistency, not a matter of external consensus, and it can be settled by simple arithmetic or a wall-clock measurement. The reader flagged this inconsistency in the rationale but did not make it the primary concern; I elevate it because it is a precise, falsifiable error in a headline quantitative claim. The verdict should remain CONDITIONAL: the authors must either correct the accounting to match the reported 26.1% student share (which is >25%) or revise the 'below 25%' wording, and they should report the actual measured overhead ratio. The core distillation method may still be valuable, but the central efficiency claim as stated is not supportable.","tokens_in":17624,"tokens_out":12828,"duration_ms":128760,"concrete_test":"Recompute the additional overhead from the percentages reported in Section 4.4: pure RL time = rollout+reward (40.7%) + teacher training (33.3%) = 74.0%; additional REST cost = student training (26.1%) / 74.0% = 35.3%. If the authors instead intend pure RL to exclude rollout+reward (which is impossible, since reward computation is intrinsic to RL post-training), the statement 'below 25%' is still unsupported. A definitive check is to run the identical teacher RL pipeline with and without the student branch under the same hardware and measure wall-clock; if the ratio exceeds 1.25, the abstract's claim is refuted. This check requires only the authors' existing code and scheduler.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.4 reports that sampling rollout with reward computation accounts for 40.7% of wall-clock time, teacher training for 33.3%, and student training for 26.1% (summing to ~100%). A pure RL baseline would consist of the first two components, totaling 74.0% of the REST wall-clock time. The student branch therefore adds 26.1 percentage points over a 74.0% baseline, i.e. 26.1/74.0 ≈ 35.3% additional overhead, not 'below 25%.' Even if one interprets 'additional training cost' as the student share of the total (26.1%), it still exceeds 25%. The abstract and conclusion repeatedly claim 'below 25% over pure RL,' while the body explicitly says 'only 26.1% additional training overhead,' an internal inconsistency. Since the central claim includes this quantitative efficiency guarantee, the discrepancy is directly load-bearing: if the reported percentages are correct, the headline efficiency statement is false; if the 'below 25%' figure is correct, the timing breakdown in Section 4.4 is wrong. Either way, the paper's central numerical claim lacks a consistent accounting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REST, a single-stage framework that attaches a decoupled student distillation branch to an ongoing diffusion RL teacher. The student is trained on the teacher's reward-scored rollout trajectories by regressing to segment velocities, with an Advantage-Modulated Distillation (AMD) weight that reweights the base MSE loss by a fused, clipped group-normalized advantage. Experiments on Stable Diffusion 3.5 Medium cover compositional generation (GenEval), visual text rendering (OCR), and human-preference alignment (PickScore), with DrawBench as an out-of-domain quality benchmark. The paper reports that the 8-step CFG-free REST student matches or surpasses its 40-step CFG-enabled RAM teacher, that AMD and EMA regularization are both beneficial in ablations, that the method works with a PCM-style base loss as well as the default segment-velocity loss, and that the additional training overhead is below 25% over pure RL.","tokens_in":17848,"tokens_out":10138,"duration_ms":97843,"significance":"If the central claims are correct, REST is a practically useful contribution: it reuses RL rollouts that would otherwise be discarded, avoids a separate distillation stage, and provides a generic reward-aware wrapper (AMD) that can be attached to different distillation losses. The paper's strengths are its broad three-task evaluation, the external DrawBench metrics, the explicit ablations isolating the AMD shift and EMA regularization, and a comparison against RTDMD under matched rewards and prompts. The framework is not circular: AMD reweights an imitation loss with rollout advantages and success is measured on external benchmarks. However, the quantitative efficiency claim in the abstract and conclusion is internally inconsistent with the timing breakdown in Section 4.4, and the appendix's theoretical support for extending REST to DiffusionNFT restates the RAM objective in a way that disagrees with the main text. These issues are load-bearing and need to be fixed before the paper can be accepted.","major_comments":[{"comment":"The training-overhead claim is internally inconsistent. Section 4.4 reports that sampling rollouts with reward computation accounts for 40.7% of wall-clock time, teacher training for 33.3%, and student training for 26.1%. A pure-RL baseline would therefore be 40.7% + 33.3% = 74.0% of the REST wall-clock time, so the student branch adds 26.1/74.0 ≈ 35.3% training overhead, not 'below 25%'. If one instead interprets the '26.1% additional training overhead' in the contributions list as the student's share of the total, that figure is also above 25%. The abstract and conclusion repeat the 'below 25% over pure RL' claim, so this is not a wording nit: the paper's numerical efficiency guarantee lacks a consistent accounting and must be corrected.","section":"§4.4, Abstract, Conclusions"},{"comment":"The appendix's restatement of the RAM objective is inconsistent with the main text. Eq. (23) writes the RAM target as v_base + η A_RAM (v_gt - v_old), whereas Eq. (3) in the main text regresses v_φ toward sg(v_base + r(x_0)(v_gt - v_φ(x_t))). These are different objects: Eq. (3) involves the current trainable policy v_φ and the raw reward r(x_0), not a lagged policy v_old and an advantage A_RAM. Because Eq. (24) is then used to argue that RAM and DiffusionNFT share an 'anchored, reward-shifted velocity-regression' structure, and Section 5 relies on that argument to support extending REST to DiffusionNFT, this inconsistency undermines a stated theoretical claim. Either the notation must be aligned and the equivalence proved, or the approximation must be stated explicitly with its conditions.","section":"Appendix A, Eq. (23); §3.1 Eq. (3)"},{"comment":"The abstract's claim that REST 'matches or surpasses its 40-step RL teacher' is stronger than the reported table supports. In the compositional row of Tab. 1, Ours† at 8 steps has GenEval 0.94 and Aesthetic 1.05, while RAM† at 40 steps has 0.95 and 1.12; in the OCR row, Ours† has Aesthetic 1.00 versus 1.07; and in the PickScore row, Ours† has DeQA 4.15 versus 4.21. No error bars, confidence intervals, or repeated-seed results are reported, and many of the observed differences are small (0.01–0.07). The text should either soften 'surpasses' to 'is comparable to' or provide variance estimates and an explicit aggregation rule for deciding when the student surpasses the teacher.","section":"Abstract, §4.2, Tab. 1"}],"minor_comments":[{"comment":"The statement that REST 'obviously surpasses' Flow-GRPO, AWM, and DiffusionNFT is not a controlled comparison: those baseline rows are not marked as using the same multi-reward protocol as the REST rows, so differences in image quality could be due to the reward protocol rather than the method.","section":"§4.2, Tab. 1"},{"comment":"Several cells in Tables 1 and 2 are blank or ambiguous in the current rendering; please state explicitly which entries are not applicable and which are missing.","section":"Tab. 1, Tab. 2"},{"comment":"The pseudo-Huber constant c_huber used in the PCM variant is not given in the experimental setup; please report its value or state that it was chosen by validation.","section":"Algorithm 2, §4.1"},{"comment":"The claim that the PCM variant 'gets similar performance' to the segment-velocity variant is only supported by qualitative images; include the numeric DrawBench metrics for both variants.","section":"Fig. 7"},{"comment":"Please report the number of generated images per prompt and the exact evaluation protocol for DrawBench metrics, as well as whether metrics are averaged over multiple seeds.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"I do not see a circularity problem with the core method: AMD reweights a distillation loss with rollout advantages and the main results are evaluated on external DrawBench metrics. The two load-bearing issues are the internally inconsistent overhead accounting in Section 4.4 and the mismatch between Eq. (23) and Eq. (3); both appear fixable within the scope of a revision. The absence of error bars is a concern for the 'matches or surpasses' claim but can be addressed by rewording or by adding variance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is genuinely new and worth engaging with: instead of treating RL rollouts as a disposable byproduct, REST reuses them as native distillation supervision, with an advantage-weighted modulation (AMD) that up-weights good trajectories and mildly repels bad ones. That is a real departure from DMDR/RTDMD, which need warm-up DMD stages and fake-score networks. The decoupled teacher–student design is clean, the math is straightforward, and the ablation study (Tables 3–4) does its job: AMD and the EMA regularizer both earn their place. The appendix showing the structural similarity between RAM and DiffusionNFT is a nice touch.\n\nBut the paper's headline efficiency claim does not survive contact with its own numbers. Section 4.4 says sampling plus reward computation is 40.7% of wall-clock, teacher training is 33.3%, student training is 26.1%. A pure RL baseline would be the first two, 74.0%. So the student branch adds 26.1/74.0 ≈ 35.3% overhead, not \"below 25% over pure RL\" as the abstract and conclusion repeat. The body itself says \"only 26.1% additional training overhead,\" which is also above 25%. This is not a nitpick: the efficiency guarantee is a central selling point, and it is internally inconsistent.\n\nThe \"matches or surpasses its 40-step RL teacher\" claim is also overstated. In Table 1, the 8-step REST student is slightly below the 40-step RAM teacher on GenEval (0.94 vs 0.95) and on several image-quality metrics (e.g., ImgRwd 4.15 vs 4.21 for human preference). It matches or beats on some PickScore numbers, but \"surpasses\" is not supported across the board. The paper would be just as strong if it claimed \"comparable to\" with a few caveats.\n\nMissing error bars and no released code are moderate weaknesses, not fatal ones. The method is not circular: AMD reweights an MSE loss with rollout advantages, and the final assessment is on external DrawBench metrics and against RTDMD. The limitation section is honest about only testing RAM as teacher and not yet integrating DMD2.\n\nWho's this for? Anyone working on post-training for diffusion/flow-matching models. The mechanism is worth discussing even if the efficiency numbers need a rewrite. I'd send it to a serious referee, but the authors should fix the overhead accounting and soften the \"surpasses\" language before it can be accepted as-is.","headline":"A genuinely new single-stage RL-distillation mechanism whose headline efficiency claim is contradicted by the paper's own timing breakdown.","tokens_in":18412,"tokens_out":2916,"would_cite":true,"duration_ms":26978,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REST shows that reward-scored RL rollouts alone can train a few-step image generator that matches its 40-step teacher.","keywords":["text-to-image generation","few-step distillation","reinforcement learning","diffusion models","advantage-weighted regression","classifier-free guidance","reward alignment","trajectory distillation"],"falsifier":"Shuffle the per-rollout advantages across trajectories before constructing $w=\\lambda(A_{\\mathrm{mix}}+b)$ and retrain REST under the same budget. If the 8-step CFG-free student still matches the 40-step RL teacher on GenEval, OCR, and PickScore, the signed advantage modulation is not what produces the reported gains and the central claim collapses.","tokens_in":17412,"feed_emoji":"🎨","tokens_out":12591,"duration_ms":111877,"temperature":0.7,"pith_summary":"The paper's central claim is that the reward-scored trajectories produced during diffusion RL are already enough to train a few-step student, so reward alignment and few-step distillation can be merged into one stage. The proposed REST framework attaches a decoupled student branch to an existing RL teacher; the student regresses to segment velocities extracted from the teacher's own rollouts while the teacher's optimization is untouched. To keep low-reward behaviors from being copied, Advantage-Modulated Distillation (AMD) multiplies the imitation loss by a signed weight derived from the rollout advantage, strengthening preferred trajectories and mildly repelling poor ones. On Stable Diffusion 3.5 Medium, an 8-step, no-classifier-free-guidance REST student matches or surpasses the 40-step CFG-enabled RL teacher on GenEval, OCR, and PickScore with less than 25% additional training cost over pure RL.","feed_headline":"8-step model matches a 40-step RL teacher","feed_subtitle":"REST turns reward-scored RL rollouts into distillation data, merging alignment and compression in one training stage.","key_machinery":"The central object is the reward-scored teacher rollout turned into a segment-wise imitation dataset. For each teacher trajectory $T=\\{x_{t_0},\\ldots,x_{t_M}\\}$, REST picks a $K$-step subschedule $t_{q_0},\\ldots,t_{q_K}$ and defines the demonstrated action for segment $k$ as the finite-difference velocity $a_k=(x_{t_{q_{k+1}}}-x_{t_{q_k}})/(t_{q_{k+1}}-t_{q_k})$. The student regresses its velocity $v_\\theta(s_k)$ to $a_k$ with the base loss $\\ell_{\\mathrm{base}}=\\|v_\\theta(s_k)-a_k\\|^2$. The shaping mechanism is AMD: fused, clipped, group-normalized advantages $A_{\\mathrm{mix}}$ enter through the signed weight $w=\\lambda(A_{\\mathrm{mix}}+b)$ with $\\lambda=1$, $b=0.5$, so the objective decomposes into a constant-weight imitation prior plus a reward-driven correction that can turn negative for low-reward rollouts. This signed weighting is what gives the distilled student a CFG-like contrastive effect while leaving the teacher's RL objective untouched.","core_discovery":"On the paper's own terms, REST is a proof that distillation can be made RL-native: a diffusion RL run already produces reward-scored finite-step trajectories, and those intermediate states are distillation supervision rather than a disposable byproduct. The teacher branch continues its original RL objective; a separate student branch learns a $K$-step, CFG-free generator by imitating piecewise segment velocities $a_k=(x_{t_{q_{k+1}}}-x_{t_{q_k}})/(t_{q_{k+1}}-t_{q_k})$ on a $K$-step subschedule of the teacher's $M$-step rollout. The AMD loss $\\mathcal{L}_{\\mathrm{AMD}}(\\theta)=\\mathbb{E}[\\lambda(A_{\\mathrm{mix}}+b)\\,\\ell_{\\mathrm{base}}(\\theta;k)]$ turns each rollout's clipped, group-normalized advantage into a signed weight: $\\lambda=1$, $b=0.5$, so high-advantage trajectories are imitated more strongly and low-advantage ones receive a mild repulsive gradient. A KL-EMA term $\\|v_\\theta(s_k)-v_{\\theta_{\\mathrm{ema}}}(s_k)\\|^2$ anchors the student against the evolving teacher. Across GenEval, OCR, and PickScore, the resulting 8-step CFG-free student matches or surpasses the 40-step CFG-enabled RL teacher on both training rewards and held-out DrawBench quality metrics, with the student branch accounting for about 26.1% of wall-clock time and the overall added cost below 25% over pure RL.","pith_inferences":["If the reported overhead scaling holds, the reuse-of-rollouts recipe should transfer to larger text-to-image and video backbones, where sampling cost is an even larger share of training; the paper does not test this.","The signed, CFG-like modulation suggests a training-time analogue of classifier-free guidance: repulsing low-reward trajectory segments may substitute for negative prompts or guidance at inference, a connection the paper notes but does not directly verify.","The paper's own limitation section leaves DMD2-style distribution-matching losses untested, so the immediate testable extension is REST with such a loss aimed at 4-step CFG-free inference.","More broadly, any sequential decision process that produces reward-scored trajectories could use the same segment-wise, advantage-modulated regression recipe; REST's machinery is not obviously restricted to diffusion image generation."],"forward_implications":["Text-to-image post-training can collapse RL alignment and few-step distillation into a single stage, removing the risk that a separate distillation pass washes out reward gains.","The student branch enables CFG-free few-step inference, so deployment can use eight steps without classifier-free guidance while retaining the teacher's reward-aligned output quality.","Because the student reuses teacher rollouts and rewards, no extra image rollouts, distillation dataset, or fake-score network are needed; added cost stays below 25% of the RL run.","AMD is a wrapper: swapping in a different base distillation loss, such as a phase-consistency loss, preserves the advantage-modulation benefit, so the method is not tied to one distillation objective.","The decoupled design means the teacher's RL optimization is unchanged, and the paper argues that any forward-process RL teacher sharing the anchored, reward-shifted velocity-regression form can be used in the same way."],"supporting_citations":[{"why":"Supplies the RAM teacher algorithm whose 40-step CFG-enabled rollouts and terminal rewards the student branch reuses.","marker":"Bergmeister et al., 2026"},{"why":"Defines the DiffusionNFT forward-process RL objective and the DrawBench evaluation protocol the paper follows for generic image quality.","marker":"Zheng et al., 2025"},{"why":"Provides the advantage-weighted forward-process matching view that motivates AMD's use of rollout advantages as signed imitation weights.","marker":"Xue et al., 2025a"},{"why":"Establishes the Flow-GRPO group-relative advantage normalization convention used to build A_mix from multiple rewards.","marker":"Liu et al., 2025"},{"why":"Defines progressive trajectory distillation, the base per-step imitation objective that AMD wraps with advantage weights.","marker":"Salimans & Ho, 2022"},{"why":"Introduces DMDR, the unified RL-distillation baseline whose multi-stage pipeline REST compares against.","marker":"Jiang et al., 2025"},{"why":"Introduces RTDMD, the unified RL-distillation baseline REST is compared with in few-step DrawBench evaluation.","marker":"Huang et al., 2026"}],"fun_headline_variants":["8-step model matches 40-step RL teacher via RL-native distillation","REST turns RL rollouts into distillation data for 5x faster generation","Advantage-weighted distillation: few-step student equals 40-step teacher","Single-stage RL-distillation co-training cuts steps without losing quality","Scored trajectories become training signal for few-step text-to-image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a student generating in a few steps without classifier-free guidance can faithfully inherit the teacher's reward-aligned behavior just by regressing to segment velocities from a many-step, guidance-enabled rollout, without needing its own guidance-free rollouts or its own reward evaluation.","fun_headline_variants_meta":{"raw":{"variants":["8-step model matches 40-step RL teacher via RL-native distillation","REST turns RL rollouts into distillation data for 5x faster generation","Advantage-weighted distillation: few-step student equals 40-step teacher","Single-stage RL-distillation co-training cuts steps without losing quality","Scored trajectories become training signal for few-step text-to-image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1701,"prompt_tokens":1144,"completion_tokens":557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":760,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":760,"tokens_out":557,"duration_ms":5765,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:10:22.152362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the per-rollout advantages across trajectories before constructing $w=\\lambda(A_{\\mathrm{mix}}+b)$ and retrain REST under the same budget. If the 8-step CFG-free student still matches the 40-step RL teacher on GenEval, OCR, and PickScore, the signed advantage modulation is not what produces the reported gains and the central claim collapses.","supporting_citations":[],"review_version":1}