{"id":"e574b9b9-b008-4684-a888-86454a46f3c3","arxiv_id":"2412.08871","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Distillation++ refines the first denoising steps of distilled diffusion models by interpolating student estimates with teacher model estimates, improving FID and text alignment on several SDXL-based few-step baselines.","lead":"Distillation++ improves few-step diffusion image generation by letting a slower teacher model refine the student model's early denoising estimates during sampling. It offers a tuning-free, data-free way to close the quality gap between fast distilled generators and their slower teachers, at the cost of one extra teacher evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The teacher-interpolation update (9) is not established as a gradient step on the SDS loss: the coefficient implied by (7)-(8) is 2γ_t α_s/(1−α_s), not λ = 2γ_t√α_t/√(1−α_t), and the manifold argument requires student estimates to be on the clean manifold, which fails at t=T.","rationale":"The reader's weakest assumption focuses on the manifold validity of (9). I agree that this is the most load-bearing point, but I sharpen it: the step from (8) to (9) is not merely missing a manifold proof—it contains an algebraic mismatch in λ. The gradient of (7) is computed with respect to the student estimate xhat_θ_0(t), but the loss is defined at perturbation time s; the correct coefficient involves α_s/(1−α_s), whereas the paper defines λ via α_t. This means the paper's own derivation does not produce (9); at best, (9) is a heuristic interpolation with plausible motivation. The DDS/DreamSampler references cannot supply the missing manifold justification because they assume the starting point is already a clean estimate, which is exactly the assumption that is false for distilled students at early steps. I do not think this refutes the empirical claim; the method may well work as a heuristic, and the results in Table 1 support that. But the paper's central theoretical claim—that this is a principled inference-time distillation via SDS minimization—is unsupported. The proposed concrete test settles the derivation issue definitively. If the λ mismatch is resolved by a stated relation between γ_t and s, then the remaining concern is purely empirical and the CONDITIONAL verdict stands; if not, the paper should present (9) as an empirical recipe and temper the theoretical framing. Either way, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":14361,"tokens_out":9598,"duration_ms":96004,"concrete_test":"Analytically compute the gradient of ℓ_distill in (7) with respect to xhat_θ_0(t) and substitute it into (8); check whether the resulting update equals (9) with λ = 2γ_t√α_t/√(1−α_t) for arbitrary s and t. If equality requires an unstated condition (e.g., α_s = α_t or a specific γ_t(s) schedule), the derivation is internally inconsistent and the theoretical framing should be revised. This test settles the concern without new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism in Eq. (9) is presented as a gradient step on the SDS loss (7), justified by the manifold-constrained analysis of DDS/DreamSampler. Two problems undermine this derivation. First, the gradient of ℓ_distill in (7) with respect to the student estimate is 2α_s/(1−α_s)(x − xhat_ψ_0(s)), so the coefficient in (9) should be 2γ_t α_s/(1−α_s), not λ = 2γ_t√α_t/√(1−α_t). The paper defines λ using t rather than s and gives no relation between γ_t and s, so the interpolation does not actually follow from the SDS gradient. Second, even with a corrected coefficient, the DDS manifold argument requires the iterate to lie on the clean manifold M. For distilled students at the initial step t=T, xhat_θ_0(T) is typically an off-manifold, low-quality estimate. Renoising it and applying the teacher once maps it to xhat_ψ_0(s), so the interpolation (1−λ)xhat_θ_0(t)+λxhat_ψ_0(s) is a convex combination of an off-manifold and an approximately on-manifold point; for small λ (0.02–0.1) it remains close to the student's off-manifold estimate. The subsequent DDIM step (8) assumes the denoised estimate is a valid posterior mean, so the justified transition to M_{t−1} is not guaranteed. The empirical gains in Table 1 are plausible, but the claimed theoretical grounding—the SDS/proximal optimization connection—is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Distillation++, an inference-time method that refines the intermediate denoised estimates of a few-step diffusion distillation student model by interpolating them with the corresponding teacher model's denoised estimates. The interpolation is motivated by a score distillation sampling (SDS) loss and is applied during the first few sampling steps. The authors evaluate the method on LCM, LCM-LoRA, SDXL-Lightning, DMD2, and SDXL-Turbo, reporting consistent improvements in FID, ImageReward, and PickScore with one additional teacher evaluation, and provide ablations on the renoising schedule and solver compatibility.","tokens_in":14729,"tokens_out":6425,"duration_ms":61451,"significance":"If the empirical results hold, Distillation++ is a simple and broadly applicable post-training booster for distilled diffusion models, with low overhead and no additional training data or fine-tuning. The paper includes a thorough empirical study across five student models, two solver families, and three metrics, plus wall-clock time measurements and a renoising-schedule ablation. The code release is a strength. However, the theoretical derivation connecting the interpolation update to SDS/proximal optimization has algebraic gaps and relies on a manifold assumption that is questionable for initial student estimates; the framework is best currently understood as a well-motivated heuristic. This does not invalidate the empirical contributions but weakens the claimed principled foundation.","major_comments":[{"comment":"The claimed derivation of the interpolation update from the SDS loss is algebraically inconsistent. The gradient of ℓ_distill in Eq. (7) with respect to the denoised estimate x is 2α_s/(1−α_s)(x − xhat_ψ_0(s)). Substituting this into Eq. (8) yields a coefficient 2γ_t α_s/(1−α_s), not the stated λ = 2γ_t √α_t/√(1−α_t). The paper uses t instead of s in λ and gives no relation between γ_t and s that would reconcile the discrepancy. Because λ is subsequently treated as a free hyperparameter in Algorithm 1, the derivation does not actually establish that the implemented update is a gradient step on the SDS loss. Please correct the derivation, state the approximations explicitly, or reframe the update as a heuristic interpolation motivated by SDS.","section":"§3.1, Eq. (7)-(9)"},{"comment":"The manifold-constrained justification borrowed from DDS/DreamSampler requires the point being updated to lie on the clean data manifold. At the initial step t = T (where the paper applies guidance, k=1), the student estimate xhat_θ_0(T) from a distilled model is typically far from the clean manifold, and the convex combination (1−λ)xhat_θ_0(t) + λ xhat_ψ_0(s) with small λ (0.02–0.1) remains close to the student's off-manifold estimate. The paper does not show that this interpolated point is a valid posterior-mean estimate, so the subsequent DDIM update may not be a justified transition on the noisy manifold. Please either add evidence that the interpolated estimates are approximately on-manifold (e.g., by measuring their distance to the teacher's clean estimates or their effect on the next-step likelihood) or qualify the theoretical claims accordingly.","section":"§3.1, Eq. (8)-(9)"}],"minor_comments":[{"comment":"There are several typos, including 'samplign' and 'inferece' in the text, and a note in the supplementary about fixing a typo in Eq. (11) that should be integrated into the main text.","section":"§4.1 and supplementary"},{"comment":"The approximation ¯α_t ≈ ¯α_s is invoked without noting that for s = t − Δt, the difference can be substantial near the initial steps; the paper should discuss the regime in which this approximation is valid.","section":"§3.1, Eq. (10)"},{"comment":"No error bars or multiple seeds are reported; since the improvements are modest, it is unclear whether the differences are statistically significant. Please add error bars or state that the numbers are from a single run.","section":"Table 1"},{"comment":"The claim of being 'tuning-free' is imprecise: the method requires a tuned teacher guidance scale λ (λ=0.02 or 0.1 in the experiments). Clarify that 'tuning-free' means without fine-tuning, not without hyperparameters.","section":"Abstract and §1"},{"comment":"The paper states that the teacher model alone with the same number of steps is inferior, but no quantitative teacher-only baseline is given in Table 1; consider adding such a row to support the synergy claim.","section":"Table 1 and Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper builds heavily on the authors' own prior works (DDS, DreamSampler, CFG++) and the derivation gap may reflect an attempt to fit the new method into that framework. The empirical contribution is distinct and potentially useful, but the theoretical framing needs substantial revision or careful hedging. The novelty relative to CFG++/DreamSampler is incremental; the paper should more clearly position what is new beyond applying those ideas to distilled students."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Distillation++ is a straightforward and useful trick: at the first sampling step of a few-step distilled model, you renoise the student's denoised estimate, run the teacher once, and interpolate between student and teacher estimates. Applied to LCM, LCM-LoRA, SDXL-Lightning, DMD2, and SDXL-Turbo, it gives consistent small gains in FID, ImageReward, and PickScore. That consistency across diverse baselines is the paper's real strength, and the ablations on the renoising schedule are a nice touch. Code is linked, which helps.\n\nWhat is genuinely new is the framing: using score distillation guidance as an inference-time correction for distilled students, rather than as a training objective. The mechanism itself is basically a weighted average, and the authors are honest that the teacher-guidance reparameterization is approximate. The experiments cover the important baselines and solvers, so the empirical claim is credible.\n\nThe soft spot is the derivation. The paper presents Eq. (9) as a gradient step on the SDS loss (7), but the algebra does not match. The gradient of (7) with respect to the student estimate gives a coefficient 2γ_t·α_s/(1−α_s), while the paper defines λ = 2γ_t·√α_t/√(1−α_t). There is no stated relation between γ_t and s, so the interpolation does not follow from that loss. The DDS manifold argument also assumes the estimate lies on the clean manifold, which is precisely what fails for a distilled student at the initial step. For the small λ values used, the interpolation stays close to the student's off-manifold estimate, so the DDIM update in Eq. (8) is on shakier ground than the paper suggests. These are real issues, but they are not fatal: the method works empirically, and the theoretical story can be dialed back to an intuition. What I would like to see in revision is a corrected derivation or an explicit statement that (9) is an empirical interpolation, plus a discussion of why the manifold assumption may be violated at the first step and why it does not matter much in practice.\n\nThe quantitative evaluation is otherwise fine, but gains are small and reported without error bars or multiple seeds. Hyperparameters like λ and the number of guided steps are hand-set per model, so the practical recipe is less turnkey than claimed. Still, the paper is useful for anyone working with few-step distilled text-to-image models who wants a cheap post-training improvement. It deserves a serious referee, not a desk reject. I would push for a revision that fixes the derivation, adds uncertainty quantification or at least multiple runs, and positions the method more modestly as an empirical guidance recipe rather than a proximal-optimization result.","headline":"A simple, practical teacher-guided correction for few-step distilled T2I models, with a derivation that does not quite close; worth engaging on the evidence.","tokens_in":763,"tokens_out":904,"would_cite":true,"duration_ms":26555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single teacher-guided interpolation at the first sampling step improves few-step diffusion models without retraining or extra data.","keywords":["inference-time distillation","score distillation sampling","few-step diffusion sampling","teacher guidance","consistency models","diffusion distillation","text-to-image generation","DDIM"],"falsifier":"Run Distillation++ with a broken teacher, e.g., weights shuffled or evaluated at a wrong renoising time; if FID and ImageReward still improve, the gain does not come from teacher-guided manifold correction. A direct check is whether Eq. (9) reduces the SDS loss of Eq. (7) at the corrected point; if the loss does not drop, the interpolation is not making the student estimate more teacher-consistent.","tokens_in":1633,"feed_emoji":"🎨","tokens_out":1819,"duration_ms":68362,"temperature":0.7,"pith_summary":"Distillation++ claims that the quality gap between few-step diffusion student models and their pre-trained teacher can be narrowed at inference time, with no retraining and no extra data. The paper recasts student sampling as a proximal optimization problem whose objective is a score distillation sampling (SDS) loss evaluated by the teacher, then shows that one gradient step reduces to a simple interpolation between the student's denoised estimate and a teacher-revised estimate. Applying this one-step correction at the first sampling step improves FID, ImageReward, and PickScore across LCM, LCM-LoRA, SDXL-Lightning, DMD2, and SDXL-Turbo. A sympathetic reader would care because the method is a drop-in post-training option that turns any few-step student into a better sampler at the cost of roughly one extra function evaluation.","feed_headline":"One extra teacher step sharpens few-step diffusion images","feed_subtitle":"Distillation++ improves FID, ImageReward, and PickScore across LCM, DMD2, SDXL-Turbo with one added evaluation.","key_machinery":"The load-bearing object is the interpolation identity of Eq. (9), derived from a proximal-optimization view of DDIM sampling with an SDS loss. It turns an intractable score-Jacobian gradient step into a convex combination of the student's clean estimate $\\hat{x}^{\\theta}_0(t)$ and the teacher-revised estimate $\\hat{x}^{\\psi}_0(s)$, with $\\lambda$ as the guidance scale; the decreasing renoising schedule $s = t - \\Delta t$ is what makes the teacher estimate a useful correction. This single identity carries the framework: once the denoised estimate is replaced by the interpolated one, any compatible solver (DDIM/Euler, Euler Ancestral, DPM-solver++ 2S/2M) can continue unchanged.","core_discovery":"The central discovery is that an inference-time, tuning-free, data-free teacher correction, given by Eq. (9) as $\\hat{x}^{\\theta}_{\\mathrm{new}}(t) = (1-\\lambda)\\hat{x}^{\\theta}_{0}(t) + \\lambda \\hat{x}^{\\psi}_{0}(s)$, consistently improves few-step student sampling. The correction is derived by minimizing an SDS loss with respect to the student's denoised estimate, using the pre-trained teacher as a critic; the gradient step is replaced by a fixed-point interpolation, and the teacher estimate is obtained by renoising the student estimate to a slightly earlier time $s = t - \\Delta t$ and denoising with the teacher. The paper shows this one-step guidance improves visual fidelity and text alignment on MS-COCO 10K across five student model families and several ODE/SDE solvers, with the largest gains in the early sampling stages.","pith_inferences":["Because the correction is applied only at the first step, the framework implies the student's initial estimate is the main bottleneck; testing the same interpolation at later steps should yield smaller gains, which would confirm that early-step trajectory correction carries the effect.","The interpolation can be extended to convex combinations of multiple teacher revisions, as the paper notes; one could test whether averaging several teachers with different styles or noise levels improves robustness without extra tuning.","Since the method requires a shared latent space between student and teacher, a natural stress test is to apply Distillation++ across different latent autoencoders; if the interpolation breaks, the shared-space condition is indeed load-bearing."],"forward_implications":["A single additional teacher evaluation at the first sampling step improves FID, ImageReward, and PickScore across LCM, LCM-LoRA, SDXL-Lightning, DMD2, and SDXL-Turbo, with the largest gains in early sampling stages.","The correction works without retraining or extra source data, making it a post-training option applicable to already-distilled student models.","The decreasing renoising schedule $s = t - \\Delta t$, rather than a random timestep, is what makes the teacher guidance effective, as shown by the ablation on DMD2.","The framework generalizes to solvers beyond DDIM/Euler, including DPM-solver++ 2S Ancestral, by modulating the denoised estimate before the solver update.","The student and teacher must share a latent space for the interpolation to be valid, which the paper identifies as a limitation for cross-latent applications."],"supporting_citations":[{"why":"Supplies the Jacobian-free manifold-constrained update that turns the SDS gradient step into a simple interpolation.","marker":"[4]"},{"why":"Provides the manifold-constrained guidance view and the DPM-solver++ 2S integration used to extend Distillation++ to other solvers.","marker":"[5]"},{"why":"Provides the proximal-optimization reformulation and the decreasing renoising schedule $s = t - \\Delta t$.","marker":"[17]"},{"why":"Defines the SDS loss that Distillation++ minimizes during sampling.","marker":"[36]"},{"why":"Defines the DDIM iterate that Distillation++ modifies with the interpolated estimate.","marker":"[40]"},{"why":"Provides the consistency-model student class whose iterative denoise-renoise sampling Distillation++ corrects.","marker":"[43]"},{"why":"Supplies the DMD2 baseline and documents the multi-step student/teacher gap that motivates the method.","marker":"[47]"}],"fun_headline_variants":["Teacher guidance sharpens few-step diffusion at inference","No-training fix boosts few-step diffusion fidelity","Single extra teacher step refines few-step diffusion","Inference-time teacher guidance improves few-step samples","Teacher critic sharpens diffusion without retraining"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The method assumes the interpolated point between student and teacher estimates still lies on the clean-data manifold, so the next DDIM update is justified; this manifold property was established for standard diffusion samplers, not for distilled students whose early estimates can be far from the teacher's PF-ODE trajectory.","fun_headline_variants_meta":{"raw":{"variants":["Teacher guidance sharpens few-step diffusion at inference","No-training fix boosts few-step diffusion fidelity","Single extra teacher step refines few-step diffusion","Inference-time teacher guidance improves few-step samples","Teacher critic sharpens diffusion without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2444,"prompt_tokens":916,"completion_tokens":1528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":532,"tokens_out":1528,"duration_ms":12555,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:28:29.644785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Distillation++ with a broken teacher, e.g., weights shuffled or evaluated at a wrong renoising time; if FID and ImageReward still improve, the gain does not come from teacher-guided manifold correction. A direct check is whether Eq. (9) reduces the SDS loss of Eq. (7) at the corrected point; if the loss does not drop, the interpolation is not making the student estimate more teacher-consistent.","supporting_citations":[{"cited_title":"Barron, and Ben Mildenhall","cited_arxiv_id":null,"evidence_quote":"Defines the SDS loss that Distillation++ minimizes during sampling."},{"cited_title":"Denoising diffusion implicit models","cited_arxiv_id":null,"evidence_quote":"Defines the DDIM iterate that Distillation++ modifies with the interpolated estimate."}],"review_version":1}