{"id":"9f5b98cd-1617-4a93-a108-a9938ce30a38","arxiv_id":"2411.14871","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A diffusion-model alignment method estimates the terminal image distribution from intermediate steps, deriving a credit assignment that emphasizes middle denoising steps.","lead":"DDE is a new training method for text-to-image diffusion models that uses preference labels to tune every denoising step, not just the final image. It explicitly estimates the final clean image distribution from intermediate steps and reports modest score gains over simpler baselines on Stable Diffusion models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 7 pulls a scalar r_k out of an integral even though the density ratio pθ/q is a function of x_k; the DDE loss is therefore not a valid estimate of the DPO objective it claims to derive.","rationale":"Both the reader and I identify Eq. 10's reliance on exp{r_k} q ≈ pθ as the fragile step. The reader frames it as a moving-target/EMA-convergence issue; I see a more fundamental problem: r_k is defined as a log-density-ratio but used as a scalar coefficient, so Eq. 7 is not a valid distributional identity even for perfectly converged r_k. This invalidates the central 'naturally derives' claim. I still credit the empirical results enough to call the method a plausible heuristic: the ablations show both estimation strategies contribute, and the qualitative examples are consistent with the claimed effect. However, with no code, no significance tests, and a derivation that does not go through as written, the paper's central claim is not supported. A revised version that either integrates over x_k properly, proves the scalar surrogate preserves the preference ordering, or explicitly reframes DDE as a heuristic with strong reproducible evidence could be acceptable. For the current manuscript I would move the verdict to REJECT.","tokens_in":16350,"tokens_out":12956,"duration_ms":146129,"concrete_test":"Pick a fixed k (e.g., t=500) and a fixed pair (x0, x_{k+1}); draw N=10 values x_k ~ q(x_k|x_{k+1}, x_0) and compute R_k = log[pθ(x_k|x_{k+1}) / q(x_k|x_{k+1}, x_0)] for each. If the standard deviation of R_k is non-negligible (say >0.1 nats), the scalar pull-out in Eq. 7 fails. As a second check, compare the correction term in Eq. 10 computed with the scalar EMA r_k against the Monte Carlo average of exp(R_k) inside the integral; material disagreement in the loss or its gradient would confirm that DDE does not implement the derived DPO objective.","verdict_should_be":"REJECT","load_bearing_attack":"Eq. 7 (and Suppl. Eq. 11) replaces each pθ(x_k|x_{k+1}) in the path integral by exp{r_k} q(x_k|x_{k+1}, x_0) and moves the factor exp(Σ_{k=t}^{T-1} r_k) outside the integral over x_k. That step is valid only if r_k is independent of x_k. For DDPM/DDIM, pθ(x_k|x_{k+1}) and q(x_k|x_{k+1}, x_0) are Gaussian with the same covariance in the standard parametrization, so log[pθ/q] is affine in x_k, not constant. No scalar coefficient can make the two distributions coincide, and the pulled-out exponential in Eq. 7 is unjustified. The EMA array in Algorithm 1 stores one scalar per transition and updates it from a single draw, so it cannot represent the x_k-dependence; Fig. 5(b) only shows convergence of these scalars, not that the distributional replacement is adequate. Consequently, Eq. 10's correction term is a heuristic weighting, and the paper's central claim that DDE naturally derives a credit assignment scheme from terminal distribution estimation is unsupported by the derivation. This remains true even if the EMA moving-target objective converges. Secondary inconsistencies (main Eq. 10 sums k=t..T-1 while Suppl. Eq. 16 sums k=t..T; Algorithm line 9 updates r[t-1], a transition outside the summed segment) reinforce that the derivation is not pinned down.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Denoised Distribution Estimation (DDE), a DPO-style preference alignment method for text-to-image diffusion models. DDE splits the denoising trajectory at a sampled step t into a T→t segment estimated by stepwise Gaussian transitions with scalar calibration coefficients and a t→0 segment estimated by a single DDIM step. The resulting loss is a DPO-like objective with correction terms that the authors claim naturally derive a credit assignment scheme emphasizing intermediate denoising steps. Experiments on SD15 and SDXL with CLIP, HPS, and PS annotators show consistent improvements over uniform and discounted baselines without auxiliary reward models.","tokens_in":16695,"tokens_out":4905,"duration_ms":47738,"significance":"If the derivation were sound, DDE would be a valuable contribution: it addresses the terminal-only preference-label problem without auxiliary models, provides a simple training objective, and its empirical results on two base models are encouraging. The paper includes useful ablations (DDE-Single, DDE-Step), an analysis of step prioritization, and qualitative comparisons. However, the central mathematical derivation has load-bearing gaps, and the claim that the credit assignment scheme is 'naturally derived' is not supported by the current equations. The empirical results may still stand as a heuristic, but the paper's framing and theoretical claims require substantial revision.","major_comments":[{"comment":"The factorization of exp(Σ r_k) out of the path integral is valid only if r_k is independent of the integration variable x_k. But r_k is defined as log[p_θ(x_k|x_{k+1})/q(x_k|x_{k+1},x0)], which depends on x_k. For the Gaussian DDPM parameterization in this paper, the two densities have the same covariance, so the log-ratio is affine in x_k, not constant. A single scalar coefficient cannot make exp{r_k}q(x_k|x_{k+1},x0) equal to p_θ(x_k|x_{k+1}) for all x_k. Therefore Eq. (7) is not a valid estimate of the DPO objective; the correction term is a heuristic weighting rather than a derived quantity. This undermines the central claim that DDE naturally derives a credit assignment scheme from terminal distribution estimation.","section":"Section 3.1, Eq. (7) and Suppl. Eq. (11)"},{"comment":"The replacement of log(E[p_θ(x0|xt)]/E[p_ref(x0|xt)]) by -||x0-μ_θ||^2 + ||x0-μ_ref||^2 drops the variance and normalization terms of the Gaussian densities and treats a ratio of expectations as if it were the ratio of single-point evaluations. Even when the same xt is used for both models, the equality does not hold in expectation; it is at best a biased approximation. The paper does not analyze this bias or its effect on preference optimization. Since this equation is the basis of the actual loss in Eq. (10), the claimed equivalence to DPO is not established.","section":"Section 3.2 / Suppl. Eq. (15)"},{"comment":"The calibration coefficients r_k are updated by EMA using log-ratios evaluated on the current target model and then treated as non-gradient constants. This is a moving-target objective: the loss depends on statistics of the same model being trained, and the paper provides only empirical convergence of the coefficients (Fig. 5(b)), not a proof or analysis of training stability. If the EMA values drift or the approximation degrades at certain steps, the derived credit assignment could misweight the preference loss. The assertion that the EMA 'does not adversely affect the training process' is not established by the current evidence.","section":"Section 3.1 / Algorithm 1"},{"comment":"There is an index inconsistency: Eq. (10) in the main text sums k=t to T-1, while Suppl. Eq. (16) sums k=t to T; the coefficient array has length T with indices 0..T-1, so the upper limit T in the supplement is out of range. Algorithm 1 updates r[t-1] on line 9 while the loss uses r[t..T-1]; the relationship between the updated index and the summed range is not explained. These inconsistencies reinforce that the derivation is not pinned down.","section":"Eq. (10) vs Suppl. Eq. (16) and Algorithm 1"}],"minor_comments":[{"comment":"There are minor typos such as 'etimating' in Section 1 and inconsistent use of 'DDE' vs. 'our DDE' in the same paragraph.","section":"Abstract / Introduction"},{"comment":"In the DDE-Step row, the HPS value is reported as 2.600±0.671, but the standard deviation in other rows is around 0.2; please verify that this is not a typo.","section":"Table 2"},{"comment":"In Fig. 5(c), the caption says the red dashed line denotes the original SD15 score of 0.320, but Table 2 reports CLIP 3.200; please clarify whether the figure uses a normalized scale or a different metric.","section":"Figure 5"},{"comment":"The sentence 'The red dashed line denotes the performance of the original SD15 with a score of 0.320' is confusing because Table 2 lists the SD15 CLIP score as 3.200; please reconcile the numbers.","section":"Section 4.5"},{"comment":"Reference [39] has 'abs/2404.3715' which appears to be a malformed arXiv identifier; please correct it.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core mathematical derivation has a serious flaw: the scalar calibration coefficients cannot capture the x-dependence of the density ratio, and the single-sample Monte Carlo step introduces an unjustified approximation. The empirical results are promising, but the authors need to either supply a rigorous derivation with explicit approximation bounds or substantially reframe the contribution as a heuristic method supported by experiments. The current framing overstates the theoretical grounding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of arXiv:2411.14871. The new thing is real: DDE combines a stepwise estimation that calibrates each transition with an EMA scalar and a single-shot DDIM jump, producing a DPO-style loss that avoids auxiliary reward models. The ablations show both pieces contribute, and the middle-step prioritization analysis is a useful observation. The gains on SD15 and SDXL over uniform/discounted baselines are a few points on CLIP/HPS/PS, so this is a within-subfield improvement, not a transformation.\n\nThe stress-test note is on target. Eq. 7 (and Supp. Eq. 11) pulls exp(Σ r_k) out of an integral over x_k, but r_k is defined as log[pθ(x_k|x_{k+1})/q(x_k|x_{k+1},x_0)], which depends on x_k. With Gaussian conditionals, that ratio is affine in x_k, not constant. No scalar calibration coefficient makes the product of exp{r_k}q and the true pθ path distribution equal. So the 'natural derivation' of credit assignment is not a derivation; it's a heuristic weighting that happens to work in practice. The EMA convergence in Fig. 5(b) shows the scalars settle, but it doesn't validate the distributional replacement. This is the paper's load-bearing claim, so it matters. The authors could fix it by explicitly presenting DDE as a heuristic DPO variant and backing it empirically--the experiments are decent even without the derivation story.\n\nSecondary issues: the summation bounds differ between Eq. 10 and Supp. Eq. 16, and Algorithm 1 updates r[t-1] when the sum runs from t to T-1. No significance tests or code are provided; that's worth asking for, but it's not decisive.\n\nWho benefits: diffusion alignment folks wanting an auxiliary-model-free baseline. I'd send this to a serious referee, but with a request for major revision on the derivation. If the math cannot be repaired, the paper should be resubmitted as a purely empirical method with the credit assignment framed as a designed, not derived, scheme.","headline":"A usable heuristic DPO variant with a broken derivation; the paper should be revised to drop the 'naturally derives' claim unless the math is fixed.","tokens_in":17173,"tokens_out":3569,"would_cite":false,"duration_ms":32529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By estimating the terminal denoised distribution from any intermediate step, DDE turns terminal-only preference labels into per-step training signal and concentrates optimization on the middle of the denoising trajectory.","keywords":["denoised distribution estimation","preference alignment","direct preference optimization","diffusion models","credit assignment","text-to-image generation","DDIM","terminal-only labels"],"falsifier":"Train an off-the-shelf diffusion DPO loss while masking out the first and last 15% of denoising steps in the objective, using the same Pick-a-Pic-V2 data and SD15/SDXL backbones. If this masking reproduces DDE's reported CLIP, HPS, and PS gains, the middle-step credit assignment is the operative mechanism; if the gains vanish, the specific estimation terms in Eq. 10 are doing essential work beyond a rough weighting scheme.","tokens_in":16096,"feed_emoji":"🎯","tokens_out":12195,"duration_ms":111132,"temperature":0.7,"pith_summary":"This paper addresses a mismatch in preference-aligned image generation: human preference labels are attached to the final image, yet diffusion models generate through hundreds of noisy intermediate steps, and it is not obvious which steps deserve credit. The authors propose Denoised Distribution Estimation (DDE), which explicitly estimates the terminal denoised distribution from any intermediate step using two cheap approximations: stepwise estimation with calibrated conditional distributions for the segment before the sampled step, and a single-shot projection that maps an intermediate noisy latent straight to the final image using the DDIM sampler. Plugging these estimates into the direct preference optimization (DPO) loss gives a training signal for every step while using only terminal labels, with no auxiliary reward model. The resulting loss systematically down-weights steps near both ends, so optimization concentrates on the middle denoising steps. If the method is right, it removes a major obstacle to adapting DPO directly to diffusion models, and the paper reports consistent gains over the base Stable Diffusion 1.5 (SD15) and Stable Diffusion XL (SDXL) models as well as over uniform and discounted credit-assignment baselines.","feed_headline":"Middle denoising steps carry the preference signal","feed_subtitle":"A new method turns terminal preference labels into per-step diffusion training, without auxiliary reward models.","key_machinery":"The central object is Denoised Distribution Estimation itself: the two-segment estimate of the terminal distribution $p_\\theta(x_0)$ from an intermediate step $t$. Its workhorse identity is $p_\\theta(x_0) \\approx \\exp\\{\\sum_{k=t}^{T-1} r_k\\} \\, \\mathbb{E}_{x_t \\sim q(x_t|x_0)}[p_\\theta(x_0|x_t)]$, which lets the terminal preference label be differentiated through every step of the trajectory. The stepwise estimate supplies the correction terms $r_k$ (EMA-calibrated log-ratios between model and conditional denoising distributions), and the single-shot DDIM estimate supplies the one-pass projection $p_\\theta(\\hat{x}_0|x_t)$; together they determine how much gradient credit each step receives, and both components are needed for the reported behavior.","core_discovery":"The paper's central claim is that DDE solves the terminal-only preference problem by replacing the intractable marginal $p_\\theta(x_0)$ with an explicit estimate built from two segments around a sampled training step $t$. For the segment $T \\to t$, each model denoising distribution $p_\\theta(x_k|x_{k+1})$ is replaced by $\\exp\\{r_k\\}q(x_k|x_{k+1},x_0)$, with $q$ the known forward conditional and $r_k$ a non-gradient calibration coefficient maintained by exponential moving average; this collapses the segment into the single factor $q(x_t|x_0)$ times a correction term. For the segment $t \\to 0$, a single DDIM pass maps the intermediate latent to a predicted $\\hat{x}_0$. Substituting this estimate into the DPO log-ratio yields a loss in which the calibration terms and DDIM coefficients enter inside the logistic function, pushing the early and late steps toward gradient saturation and thereby assigning more optimization credit to middle denoising steps. The authors argue, and support with ablations, that this naturally derived middle-step credit assignment is what makes the method outperform hand-crafted uniform or discounted schemes without any auxiliary model.","pith_inferences":["A direct testable consequence the paper does not claim: if the middle-step emphasis is the active ingredient, then masking out the first and last 15% of denoising steps in an off-the-shelf diffusion DPO loss should reproduce most of DDE's reported gain; the paper's own Fig. 5(c) points in that direction.","The converged calibration coefficients could be read as a diagnostic of where the pretrained model's denoising distribution drifts from the ideal forward conditional, which could inform targeted preference-data collection or per-step learning-rate schedules.","The same two-segment estimation recipe should transfer to other generative models with a known forward noising process and a deterministic one-step inverse map, such as flow-matching or consistency models; if transfer fails, it would localize the mechanism to DDPM/DDIM-specific structure."],"forward_implications":["Terminal preference labels become usable at every denoising step without training an auxiliary reward model; the optimization signal is derived directly from estimating $p_\\theta(x_0)$.","The gradient signal concentrates on the middle of the denoising trajectory, so early near-noise steps and late near-clean steps are not over-weighted by a label that only observes the final image.","On SD15, DDE reports consistent improvements of roughly 3.3% to 6.7% over the base model on CLIP, HPS, and PS scores; on SDXL the reported gains are 1.0% to 3.1%.","Each estimation strategy is necessary: ablations that drop the calibration coefficients or perform optimization directly on the intermediate noisy sample both degrade performance."],"supporting_citations":[{"why":"Supplies the DDPM denoising distributions and the known conditional distribution that both estimation strategies replace.","marker":"[16]"},{"why":"Provides the direct preference optimization loss that DDE rewrites using estimated terminal distributions.","marker":"[37]"},{"why":"Provides the single-step deterministic projection from an intermediate latent to the predicted final image used by the t-to-0 estimate.","marker":"[46]"},{"why":"Supplies the Pick-a-Pic preference dataset on which SD15 and SDXL are fine-tuned and evaluated.","marker":"[21]"},{"why":"Baselines that assign equal preference credit to every denoising step; DDE is compared against them.","marker":"[51, 57]"},{"why":"Baseline that assigns discounted credit favoring early denoising steps; DDE's middle-step scheme is contrasted with it.","marker":"[59]"},{"why":"The Stable Diffusion 1.5 base model that DDE fine-tunes and beats in the main experiments.","marker":"[38]"},{"why":"The Stable Diffusion XL base model used for the second set of fine-tuning and evaluation experiments.","marker":"[35]"}],"fun_headline_variants":["Middle denoising steps rule preference alignment","Terminal labels now teach every diffusion step","DDE: diffusion alignment with no reward model","Preference credit shifts to middle denoising steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the running calibration coefficients keep the estimated per-step distributions close to the model's own denoising distributions throughout training, even though the coefficients are computed from the very model being updated and then treated as constants.","fun_headline_variants_meta":{"raw":{"variants":["Middle denoising steps rule preference alignment","Terminal labels now teach every diffusion step","DDE: diffusion alignment with no reward model","Preference credit shifts to middle denoising steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000433,"raw_usage":{"total_tokens":2217,"prompt_tokens":964,"completion_tokens":1253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1195}},"tokens_in":580,"tokens_out":1253,"duration_ms":10747,"temperature":1.0,"reasoning_tokens":1195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:47:08.946127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an off-the-shelf diffusion DPO loss while masking out the first and last 15% of denoising steps in the objective, using the same Pick-a-Pic-V2 data and SD15/SDXL backbones. If this masking reproduces DDE's reported CLIP, HPS, and PS gains, the middle-step credit assignment is the operative mechanism; if the gains vanish, the specific estimation terms in Eq. 10 are doing essential work beyond a rough weighting scheme.","supporting_citations":[{"cited_title":"Denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Supplies the DDPM denoising distributions and the known conditional distribution that both estimation strategies replace."},{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"Provides the direct preference optimization loss that DDE rewrites using estimated terminal distributions."},{"cited_title":"De- noising diffusion implicit models","cited_arxiv_id":null,"evidence_quote":"Provides the single-step deterministic projection from an intermediate latent to the predicted final image used by the t-to-0 estimate."},{"cited_title":"Pick-a-pic: An open dataset of user preferences for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the Pick-a-Pic preference dataset on which SD15 and SDXL are fine-tuned and evaluated."},{"cited_title":"A dense reward view on aligning text-to-image diffusion with preference","cited_arxiv_id":null,"evidence_quote":"Baseline that assigns discounted credit favoring early denoising steps; DDE's middle-step scheme is contrasted with it."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"The Stable Diffusion 1.5 base model that DDE fine-tunes and beats in the main experiments."}],"review_version":1}