{"id":"cdf62546-de88-4e2f-8e99-3d385e886d69","arxiv_id":"2411.17310","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"The authors introduce Reward Incremental Learning for text-to-image diffusion models and show that per-task LoRA adapters plus EMA distillation mitigate catastrophic forgetting while preserving general generation quality.","lead":"This paper defines a new problem, Reward Incremental Learning, where a text-to-image diffusion model must be fine-tuned on a sequence of reward objectives without forgetting earlier ones. It proposes Reward Incremental Distillation, combining per-task LoRA adapters with EMA distillation, and reports reduced catastrophic forgetting relative to a sequential fine-tuning baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main evidence is confounded: RID and baseline differ in batch size (8 vs 128) and learning rate (5e-5 vs 1e-3), so the reported forgetting reduction may be a hyperparameter effect rather than an effect of RID's components.","rationale":"Reader's weakest assumption points to last-step distillation being insufficient. That is a reasonable unablated design concern, but I think the more immediate load-bearing issue is the experimental comparison: RID and baseline differ in learning rate and batch size by orders of magnitude, and the appendix explicitly says the baseline performs worse under RID's configuration. Until the hyperparameters are matched, every Table 2/3 comparison is consistent with the alternative explanation that RID's forgetting mitigation comes from a low learning rate, not from EMA distillation or adapter separation. This concern is independent of the last-step-sufficiency question: even if last-step distillation is sufficient, the paper does not currently show it is necessary or that it is the cause of the gains. The proposed test is cheap: train the baseline with RID's optimization settings and train RID with the baseline's settings, and compare retention. If the baseline under RID's settings still forgets dramatically, the confound is resolved and the reader's original concern about distillation depth becomes the main remaining question. If it does not, the central claim should be revised. I therefore keep the CONDITIONAL verdict, because the paper can be fixed with matched experiments and an ablation, but the claim as written is not currently supported by unconfounded evidence. The reader's rationale already listed hyperparameter mismatch as a weakness, so this is a partial agreement rather than a disagreement.","tokens_in":13606,"tokens_out":5688,"duration_ms":53509,"concrete_test":"Run both methods on the Aesthetic→HPS→Compress sequence under two matched regimens: (i) batch size 8, learning rate 5e-5, same number of optimizer steps for both methods; (ii) batch size 128, learning rate 1e-3, same number of optimizer steps for both methods. Use the same seed and the same total number of training images per task. Report all Table 2 metrics plus Eq. 3 forgetting. If the baseline under RID's configuration still shows CLIP drop >3 and zero-shot FID increase >30 while RID stays near SD1.5, the hyperparameter confound is not the explanation; if baseline matches RID's retention, the central claim is unsupported. A secondary check is to ablate RID without distillation under its own hyperparameters to isolate the LoRA group effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Supplementary Material §10, the paper states that the adapted baseline is trained with batch size 128 and learning rate 1e-3, while RID uses batch size 8 and learning rate 5e-5, and notes 'Under this configuration, the adapted baseline performs worse.' This is a direct admission that the two methods are not compared under matched training conditions. The central claim—that RID's LoRA-adapter group plus EMA last-step distillation mitigates forgetting—requires the comparison to isolate those components. Instead, Tables 1-3 jointly vary the method and the optimization schedule; per epoch, with 750 training prompts, the baseline performs about 6 updates while RID performs about 94, a 15x difference in update count. A smaller learning rate and smaller batch size can by themselves reduce catastrophic forgetting or trade off target-task adaptation, so the observed CLIP/FID retention in RID cannot be attributed to the proposed distillation without a controlled comparison. No ablation with equal learning rate and batch size, or with matched total optimizer updates, is reported. The authors' own appendix sentence is an explicit limitation that should be weighed in the verdict.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper defines Reward Incremental Learning (RIL), a continual-learning setting in which a text-to-image diffusion model is sequentially fine-tuned on differentiable reward objectives (aesthetic quality, human preference, compressibility). The authors observe that a baseline constructed by adapting reward-backpropagation methods [5,22] with a per-task LoRA adapter group suffers from both metric-wise and visual-structure-wise catastrophic forgetting. They propose Reward Incremental Distillation (RID), which combines this adapter group with an EMA teacher whose final denoising-step output is matched to the student's final output through an MSE penalty (Eq. 9). Across six task orderings of three reward tasks (Table 3), RID reports average CLIP score 24.42 vs 19.76 and zero-shot MS-COCO FID 85.95 vs 130.92 for the baseline, with original Stable Diffusion V1.5 at 25.77 and 71.54, and lower target-metric forgetting, at under 1% extra training time (Table 4).","tokens_in":13837,"tokens_out":19997,"duration_ms":169613,"significance":"If the central comparison were properly controlled, the paper would make a useful contribution: RIL is a well-motivated problem formulation, the forgetting phenomenon is documented with general metrics (CLIP score, zero-shot FID) that are not part of the training reward, which avoids circularity, and the proposed mechanism is computationally cheap (Table 4) and described with runnable pseudo-code (Alg. 1). The six task orderings in Table 3 give the evaluation reasonable breadth within its chosen scope. The main caveat is that the evidence is currently confounded with the optimization schedule (batch size 128 / lr 1e-3 for the baseline vs 8 / 5e-5 for RID, per Sec. 10), so the magnitude of the benefit attributable to the method itself is not yet established. With matched-hyperparameter ablations and at least one additional continual-learning control, the empirical claim would be substantially stronger.","major_comments":[{"comment":"The central comparison is confounded by the optimization schedule. Sec. 10 reports that the adapted baseline is trained with batch size 128 and learning rate 1e-3, while RID uses batch size 8 and learning rate 5e-5, and states that 'Under this configuration, the adapted baseline performs worse.' With 750 training prompts, the baseline performs about 6 optimizer updates per epoch versus about 94 for RID, a roughly 15-fold difference in update counts. Because Tables 1-3 vary the method and the schedule simultaneously, the large gaps in CLIP score (24.42 vs 19.76) and zero-shot FID (85.95 vs 130.92) cannot be attributed to the LoRA adapter group and the EMA distillation; a smaller batch size and a smaller learning rate can by themselves reduce forgetting or trade off task adaptation. A matched comparison (e.g., the baseline at batch 8 / lr 5e-5, and RID without the distillation term under that same schedule), or a matched total-update comparison, is needed to support the abstract's claim that RID's components mitigate forgetting.","section":"Sec. 10; Tables 1-3"},{"comment":"The sufficiency of the last-step distillation is not empirically established. The entire forgetting-mitigation mechanism of RID is the single MSE alignment between the student's and the EMA teacher's final denoising-step outputs given the same latent z1 (Eq. 8), yet no ablation is reported that varies the distillation design: no distillation, full-step distillation as in Eq. (7), last-step distillation as in Eq. (8), or alignment at intermediate steps. This matters because the paper motivates the forgetting phenomenon as affecting both metrics and visual structure across the generation trajectory (Sec. 1, Fig. 1); if the drift originates in earlier denoising steps, a one-step output anchor may be too weak, and the claim in Fig. 4 that full-step distillation suffers from accumulated error is plausible but untested. The balancing coefficient lambda in Eq. (9) is also fixed at 0.1 with no sensitivity analysis. Ablations varying the distillation depth and lambda under matched hyperparameters are required to show that the proposed component, rather than the training schedule, produces the reported retention.","section":"Sec. 4.2, Eq. (8)"},{"comment":"A core hyperparameter of the proposed method is missing. The EMA teacher in Eq. (6) depends on the momentum coefficient alpha, and Alg. 1 calls ema_model.update(model), but the value of alpha is never reported in the main text or in Sec. 10, which gives epochs, learning rates, batch sizes, and lambda but not alpha. The optimizer, its hyperparameters, and any learning-rate schedule are also unspecified. Because the EMA teacher is the mechanism claimed to retain previous knowledge, RID cannot be reproduced or ablated without these values; please add a complete implementation-details table.","section":"Eq. (6), Alg. 1"},{"comment":"The reported results are single runs with no variance or significance information, and the direction of the differences is not uniform across target metrics. In Table 3, RID's average aesthetic score (5.55) is below the baseline's (5.62) across the same six orderings, and in two orderings (Compress-HPS-Aesthetic and HPS-Compress-Aesthetic) the baseline's final-task aesthetic score exceeds RID's by a wide margin (7.23 vs 5.77 and 6.18 vs 5.75). The general-metric differences are large and likely robust, but the claim in Sec. 5.2 that RID 'consistently outperforms the adapted baseline' would be substantiated more convincingly with multiple seeds and error bars, especially where target-metric differences are small (e.g., average HPS 0.277 vs 0.258).","section":"Sec. 5.2, Table 3"},{"comment":"The comparison set contains no external continual-learning control. The only comparators are the authors' own adapted baseline (reward backpropagation plus the proposed LoRA adapter group, Sec. 4.1) and a model-soup combination with arbitrarily fixed mixing weights alpha = beta = 1/3 (Sec. 6). Since the forgetting phenomenon is presented as a general issue of sequential reward fine-tuning in diffusion models, at least one control from the existing continual-learning literature (e.g., InfLoRA alone, a regularization-based adapter method, or sequential fine-tuning with a single shared LoRA) should be evaluated to show that the combined design of the LoRA adapter group plus EMA distillation, rather than any LoRA-based approach with careful hyperparameters, is what yields the observed retention.","section":"Sec. 4.1, Sec. 6"}],"minor_comments":[{"comment":"The sentence in Sec. 5.2 that RID 'achieves notable improvements across all target metrics' is ambiguous given that Table 3's average aesthetic score is lower for RID (5.55) than for the baseline (5.62); please specify whether the comparison is against Stable Diffusion V1.5 or against the baseline.","section":"Sec. 5.2, Table 3"},{"comment":"The notation in Eq. (3) is confusing: the superscript t denotes the task index while the subscript i denotes the training stage, and T is used both as the total number of tasks and as the index of the current stage, although Table 2 reports forgetting at intermediate stages; please re-notate and define all indices.","section":"Eq. (3)"},{"comment":"In Tables 2 and 3, zero forgetting is sometimes written as '0.00' and sometimes as '-' (e.g., the Compress-HPS-Aesthetic row of Table 3), and the metric of the most recently trained task always shows '-' even though it is the reference value in Eq. (3); the convention should be stated explicitly.","section":"Tables 2-3"},{"comment":"The qualitative comparisons would be more informative with larger crops and zoom-ins; at thumbnail size, the claimed visual-structure-wise forgetting in Fig. 5 is difficult for the reader to verify.","section":"Fig. 5"},{"comment":"Sec. 2 does not discuss recent continual-learning methods designed for diffusion models (outside the classification-focused literature it cites); engaging with them would sharpen the novelty claim of the RIL formulation and the positioning of RID.","section":"Sec. 2"}],"recommendation":"major_revision","confidential_remarks":"Editorial points for you. The Sec. 10 admission that the baseline performs worse under RID's hyperparameters is the key limitation; the revision must address it with matched-schedule experiments or a skeptical reader will attribute the whole result to batch size and learning rate. The citation pattern is narrow: the method builds directly on [5,22] and InfLoRA [16] but the related work does not engage the continual-learning-for-diffusion literature, which may draw novelty objections. The absence of error bars or multiple seeds is likely to be raised again in subsequent review. If the revision adds matched-hyperparameter ablations, an ablation of distillation depth, and variance estimates, the paper is publishable; the problem formulation and the low-overhead mechanism are solid contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the RIL problem definition is genuinely useful and I suspect will be picked up by the community, but the paper's main empirical claim about RID is not backed by the experiments as they stand. What's new and good: formalizing reward incremental learning and documenting that forgetting in this setting shows up both in target metrics and in general CLIP/FID with the general metrics included is a real service. The adaptive baseline is a reasonable construction, the qualitative examples align with the numeric trend, and the last-step distillation trick is a sensible efficiency choice. Including model soup and joint tuning as comparators is above board. Soft spots: the central comparison is confounded. Appendix Sec. 10 states baseline uses batch 128 and lr 1e-3, while RID uses batch 8 and lr 5e-5, and even notes 'Under this configuration, the adapted baseline performs worse.' That means Tables 1-3 vary both the method and the optimization schedule. Smaller batch and much smaller learning rate could reduce forgetting on their own. No matched hyperparameter experiment, no equal-update-count comparison, no ablation of the distillation depth. This is not a minor omission; it's the load-bearing comparison. Also, the Figure 1 caption claims RID 'consistently improves upon the original Stable Diffusion model,' but RID's zero-shot FID is worse than SD1.5 in Table 3 (85.95 vs 71.54), so the claim overstates. The paper is honest enough to disclose the mismatch, which I respect, but disclosure doesn't fix the confound. The RIL formulation is worth publishing; the RID evidence needs a controlled rerun. Who it's for: people working on continual fine-tuning of generative models or on reward-based adaptation. It deserves a serious referee, not a desk reject, because the problem definition is timely and the method is plausible. Recommendation: send to peer review, but the decision should hinge on whether a rerun with matched training conditions reproduces the forgetting reduction.","headline":"RIL is a real, useful problem definition, but the paper's evidence for RID is confounded by mismatched batch size and learning rate between method and baseline.","tokens_in":14393,"tokens_out":2855,"would_cite":true,"duration_ms":25323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward Incremental Distillation lets text-to-image models learn new reward objectives in sequence without forgetting how to draw well.","keywords":["reward incremental learning","catastrophic forgetting","text-to-image generation","diffusion models","low-rank adaptation","knowledge distillation","EMA teacher","continual fine-tuning"],"falsifier":"Compare RID against a variant where distillation is applied to an intermediate denoising step (say $z_{25}$ of 50) instead of only $z_1$; if the intermediate version shows substantially less forgetting, then last-step anchoring alone is not the operative mechanism.","tokens_in":13357,"feed_emoji":"🎨","tokens_out":5812,"duration_ms":46770,"temperature":0.7,"pith_summary":"The authors introduce Reward Incremental Learning (RIL), a setting in which a text-to-image diffusion model is fine-tuned on a sequence of reward objectives (aesthetic quality, human preference, compressibility) rather than a single one. They show that directly adapting existing reward-gradient fine-tuning to this sequential setting causes catastrophic forgetting that appears both in standard metrics (CLIP score, zero-shot FID) and in visible image structure. To counter it, they propose Reward Incremental Distillation (RID), combining a LoRA adapter group, so each task gets its own parameters, with an exponential-moving-average teacher and a one-step distillation loss that anchors the model's final output to the teacher's. Their experiments across six orderings of the three reward tasks report that RID keeps general metrics close to the original Stable Diffusion 1.5 (average CLIP 24.42 vs. 19.76 for the baseline, average zero-shot FID 85.95 vs. 130.92) while staying competitive on target rewards, at under one percent extra training time.","feed_headline":"One-step trick keeps image models from forgetting old rewards","feed_subtitle":"Reward Incremental Distillation preserves general quality across sequential fine-tuning with under 1% extra training time.","key_machinery":"The central mechanism is Reward Incremental Distillation (RID), built from a LoRA adapter group and an exponential-moving-average (EMA) teacher. When a new reward task arrives, a fresh pair of LoRA matrices is added and all previous adapters and the pretrained weights are frozen, so each task's parameters do not interfere. During fine-tuning, the student model's final denoising output is aligned with the EMA teacher's final output from the same latent $z_1$ via an MSE loss (Eq. 8), with backpropagation truncated to that last step; the full training objective combines the reward gradient with this distillation loss (Eq. 9).","core_discovery":"The paper claims that the forgetting observed in incremental reward fine-tuning of diffusion models can be substantially mitigated by a combination of two ingredients: parameter isolation per task via a group of LoRA adapters, and momentum distillation in which the fine-tuning model is pulled, on its final denoising step, toward the output of an exponential-moving-average copy of itself. The proposed objective is $R_t(f_{W,A,B}(z_1|c)) - \\lambda\\|f_{W,A,B}(z_1|c) - f^T_{W,A,B}(z_1|c)\\|_2$, where the reward term adapts to the new objective and the distillation term preserves general generation quality. The authors argue that aligning only the last step, rather than the full denoising trajectory, keeps computation low and avoids error accumulation, and they report that across all six task orderings the general metrics stay close to the pretrained model while target rewards are preserved or improved.","pith_inferences":["The last-step distillation assumption could be tested by ablating distillation at different denoising steps; if intermediate-step drift is the main forgetting source, RID would underperform on longer trajectories.","Since RID keeps an EMA teacher of the full student, it could be combined with other continual-learning techniques such as replay of previous prompts or parameter regularization, potentially improving retention further.","The method's reliance on a differentiable reward and a frozen EMA copy suggests it could extend to non-differentiable rewards by using the teacher's output as a pseudo-target and the reward signal from a learned reward model, though that is beyond the paper.","A natural stress test is to push the number of tasks beyond three; the LoRA adapter group grows linearly with tasks, so at some point memory and interference between many adapters may need a pruning or merging step."],"forward_implications":["A single text-to-image model can be adapted to several reward objectives over time without collapsing general image quality, so practitioners do not need to retrain from scratch or maintain multiple models.","The forgetting metric defined in Eq. 3 provides a quantitative way to compare continual fine-tuning methods for diffusion models beyond classification accuracy.","Because distillation is applied only to the last denoising step, the approach adds little compute (about 2% extra steps, under 1% training time), making it practical for large models.","RID outperforms both an adapted reward-gradient baseline and weight-averaging (model soup) in keeping general metrics stable across task orderings.","Target rewards are maintained across sequences; for example, after all three tasks, RID retains aesthetic and human-preference improvements over the original model while the baseline falls below it (Table 2)."],"supporting_citations":[{"why":"Supplies the reward-gradient fine-tuning procedure that the baseline and RID build on, including DDIM and last-step backprop truncation.","marker":"[5]"},{"why":"The other single-reward fine-tuning method adapted into the RIL baseline; also source of the prompt dataset split (750/50).","marker":"[22]"},{"why":"Motivates the LoRA adapter group design that isolates parameters per task to reduce interference.","marker":"[16]"},{"why":"Provides the EMA teacher-student mechanism used for distillation.","marker":"[32]"},{"why":"The DDIM scheduler used for efficient sampling and the truncation of backpropagation to the final step.","marker":"[31]"},{"why":"The alternative multi-reward approach (model soup) that RID is compared against.","marker":"[36]"},{"why":"Supplies the human preference reward predictor and the HPD prompt dataset.","marker":"[37]"},{"why":"Supplies the aesthetic reward function used in the tasks.","marker":"[29]"}],"fun_headline_variants":["New method curbs forgetting in reward-tuned image models","Tackling catastrophic forgetting in reward-tuned diffusion models","Reward Incremental Distillation preserves image quality across tasks","Sequential reward fine-tuning keeps diffusion models sharp","Incremental rewards, no forgetting: diffusion model fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that forcing the fine-tuned model's final denoising output to match the EMA teacher's final output, from the same latent $z_1$, is enough to preserve both metric-level and visual-structure-level knowledge from earlier tasks.","fun_headline_variants_meta":{"raw":{"variants":["New method curbs forgetting in reward-tuned image models","Tackling catastrophic forgetting in reward-tuned diffusion models","Reward Incremental Distillation preserves image quality across tasks","Sequential reward fine-tuning keeps diffusion models sharp","Incremental rewards, no forgetting: diffusion model fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001497,"raw_usage":{"total_tokens":6006,"prompt_tokens":940,"completion_tokens":5066,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":4986}},"tokens_in":556,"tokens_out":5066,"duration_ms":30512,"temperature":1.0,"reasoning_tokens":4986,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:16:05.195359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare RID against a variant where distillation is applied to an intermediate denoising step (say $z_{25}$ of 50) instead of only $z_1$; if the intermediate version shows substantially less forgetting, then last-step anchoring alone is not the operative mechanism.","supporting_citations":[{"cited_title":"Directly fine-tuning diffusion models on differentiable re- wards","cited_arxiv_id":null,"evidence_quote":"Supplies the reward-gradient fine-tuning procedure that the baseline and RID build on, including DDIM and last-step backprop truncation."},{"cited_title":"Inflora: Interference-free low-rank adaptation for continual learning","cited_arxiv_id":null,"evidence_quote":"Motivates the LoRA adapter group design that isolates parameters per task to reduce interference."},{"cited_title":"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results","cited_arxiv_id":null,"evidence_quote":"Provides the EMA teacher-student mechanism used for distillation."},{"cited_title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time","cited_arxiv_id":null,"evidence_quote":"The alternative multi-reward approach (model soup) that RID is compared against."},{"cited_title":"LAION- aesthetics","cited_arxiv_id":null,"evidence_quote":"Supplies the aesthetic reward function used in the tasks."}],"review_version":1}