{"id":"c28e5f43-3d59-4a53-b8c9-2d9ecd033dec","arxiv_id":"2412.21044","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"E2ED2 fine-tunes a pretrained diffusion model end-to-end from pure noise to the target latent, improving few-step FID and CLIP on COCO30K and HW30K over the PixArt-delta baseline.","lead":"This paper proposes E2ED2, a training procedure that fine-tunes a text-to-image diffusion model by running the full few-step sampling process and then penalizing the difference between the generated image and the real image. The authors report improved few-step FID and CLIP scores over the PixArt-delta baseline, but the strongest advertised gains, including with GAN loss and against SDXL, are not supported by the paper's own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7) defines reconstruction against a noise zT that is sampled independently of z0; its Bayes-optimal solution is the per-prompt conditional mean, so the claimed noise-to-data mapping is not learned by the objective and only survives via pretrained initialization.","rationale":"The strongest claim requires that L_recon makes the full sampling chain directly optimize final reconstruction and thus learns a noise-to-data map. The load-bearing condition is that Eq. (7) is a sound generative objective. That condition fails mathematically: because zT is independent of z0, the expected reconstruction loss is minimized by a constant per-prompt prediction. This is not a concern about gradient stability or consensus; it is an internal degeneracy of the proposed objective. The paper's empirical success is therefore attributable to the pretrained initialization and small-step fine-tuning, not to the end-to-end mechanism. This supports the Reader's REJECT verdict, though via a different and more fundamental mechanism than the Reader's weakest assumption about backprop instability. I also credit the paper for clearly disclosing the no-from-scratch limitation in Appendix A.1 and for running a human evaluation, but those do not repair the objective's ill-posedness. The concrete test of random-initialization training would settle whether the degeneracy manifests as collapse. No change to the verdict is needed.","tokens_in":15401,"tokens_out":5728,"duration_ms":62420,"concrete_test":"Run E2ED^2 from random initialization (Appendix A.1's missing experiment) with L_recon=L2 only and the same 120k-image text-conditioned dataset. After convergence, fix 20 prompts and generate 8 images per prompt from different zT; compute intra-prompt LPIPS diversity and FID against PixArt-δ. If per-prompt images collapse toward a single output (diversity near zero, FID much worse), the degeneracy of Eq. (7) is confirmed. Cheaper check: take the Table 2 row-4 checkpoint and compare intra-prompt LPIPS diversity for fixed prompts across multiple zT against the pretrained PixArt-δ baseline; if diversity is already lower, the fine-tuning is eroding the generative distribution even at 24k steps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, Eq. (7) sets L_recon(θ)=E_{q(zT,z0)}[d(z0, z0_hat)], with zT∼N(0,I) drawn independently from z0 in Algorithm 1 (line 6). For the default L2 distance, the optimal predictor satisfies G_θ(zT,c)=E[z0|c] for all zT, because zT carries no information about which image x0 was sampled; L1 gives the conditional median. Thus the reconstruction objective alone is degenerate for generation: it rewards mapping every noise to the same per-prompt average, not to the data distribution. The paper never identifies or counteracts this; the only reasons the model does not visibly collapse are the 24k-step, 1e-6-LR fine-tuning from pretrained PixArt-δ and the fact that Appendix A.1 explicitly admits training from random initialization is unexplored. Consequently, the central claim that E2ED^2 'directly transforms pure Gaussian noise to the target data distribution' is not supported by Eq. (7); the pretrained prior, not the objective, is what preserves per-prompt diversity. Further, Table 2's best configuration (L2+LPIPS, no GAN) is exactly the degenerate objective, so its FID/CLIP gains are inherited from the starting checkpoint. The only distribution-matching term, the GAN loss, degrades COCO FID from 25.27 to 25.74 and CLIP from 32.76 to 31.75, consistent with it fighting the reconstruction term rather than complementing it. The unrolled-backprop instability the Reader flags is real, but it is secondary: even with perfect gradients, Eq. (7) does not optimize a sample-quality objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes E2ED^2, a fine-tuning procedure for the pretrained PixArt-delta model in which the full few-step LCM sampling chain is unrolled and a reconstruction loss d(z0, z0_hat) is applied directly to the final predicted latent, optionally combined with LPIPS and adversarial losses. The authors claim that this end-to-end objective removes the training-sampling discrepancy and the information leakage of the forward noising process, and that it enables perceptual and GAN losses. Experiments on COCO30K and HW30K report improved FID and CLIP scores relative to PixArt-delta at 3 and 4 sampling steps, with ablations over L1/L2/LPIPS/GAN combinations and a human preference study.","tokens_in":15824,"tokens_out":6354,"duration_ms":65484,"significance":"If the central claim were valid, unrolled fine-tuning with a simple reconstruction loss would be a practically valuable and easy-to-reproduce recipe for few-step text-to-image generation. The paper has several strengths: it evaluates on external benchmarks (COCO30K, HW30K), reports ablations over loss terms, includes a careful human evaluation protocol, and explicitly acknowledges in Appendix A.1 that training from random initialization has not been explored. However, the main theoretical claim is undermined by a fundamental degeneracy in Eq. (7), and the paper's own Tables 1 and 2 contradict several headline statements. The empirical observation that unrolled fine-tuning of PixArt-delta improves few-step metrics may be of interest, but the paper does not provide a sound account of why it works, and the proposed objective as written does not learn a noise-to-data mapping.","major_comments":[{"comment":"The reconstruction objective in Eq. (7), L_recon(θ) = E_{q(z_T,z_0)}[d(z_0, z0_hat)], is degenerate as a generative objective because z_T is sampled independently of z_0 (Algorithm 1, line 6). For L2 distance, the Bayes-optimal predictor satisfies G_θ(z_T,c) = E[z_0 | c] for every z_T; for L1 it is the conditional median. Thus the objective rewards mapping every noise input to the same per-prompt average latent, not mapping noise to the data distribution. The claim that the method \"directly transforms pure Gaussian noise to the target data distribution\" (Abstract, §1, §3.2) is therefore not supported by Eq. (7). The model avoids visible collapse only because it is initialized from pretrained PixArt-delta and fine-tuned with a small learning rate for 24k steps, and Appendix A.1 explicitly states that training from random initialization has not been explored. The only term in Eq. (8) that encourages distribution matching is the GAN loss, and Table 2 shows that adding it worsens both FID and CLIP, so no component in the proposed objective counteracts the degeneracy.","section":"§3.2, Eq. (7), Algorithm 1"},{"comment":"The text claims the method \"surpasses larger models like SDXL in both image quality and alignment.\" Table 1 contradicts this: E2ED2 achieves COCO FID 25.27 versus SDXL's 14.28, and HW FID 9.76 versus SDXL's 7.96, so SDXL is substantially better on both FID measures. The CLIP score is higher (32.76 versus 31.68), but the image-quality claim as stated is false. The comparison should be reworded to report the actual trade-off between FID and CLIP, and the \"state-of-the-art\" claim in the abstract and §4.2.1 should be calibrated to the numbers in Table 1.","section":"§4.2.1, Table 1"},{"comment":"The ablation in Table 2 shows that the best configuration is L2+LPIPS (COCO FID 25.27, COCO CLIP 32.76) and that adding GAN loss degrades both metrics (COCO FID 25.74, COCO CLIP 31.75). This directly contradicts the paper's repeated claims that the framework \"seamlessly integrates\" GAN losses and that GAN loss improves image quality and text-image alignment. The discussion admits a \"trade-off,\" but the quantitative evidence is that the GAN term hurts both headline metrics, and the qualitative claim about high-frequency detail is not supported by any perceptual metric. This undermines the second stated contribution and the attribution of the main result to the GAN component.","section":"§4.2.3, Table 2"},{"comment":"Algorithm 1 is underspecified at a load-bearing point: it loops t=T to 1 without stating how many unrolled steps are used during training. If T=1000, the procedure is computationally implausible and inconsistent with the 3-4 step evaluation; if T is small, the paper should state the exact unrolled sampling schedule and analyze the behavior of gradients through the unrolled chain. No gradient-stability or training-curve analysis is provided. In addition, the information-leakage claim based on D_KL(q(x_T)||N(0,I)) > 0 is not measured or addressed: starting training from z_T ~ N(0,I) changes the training input distribution but does not alter the mutual information I(x_T;x_0) of the diffusion forward process, and the paper provides no experiment showing that leakage is actually reduced.","section":"§3.2 and Appendix C.1"}],"minor_comments":[{"comment":"The reference to \"Algorithm ??\" should be \"Algorithm 1.\"","section":"§3.2"},{"comment":"The caption and diagram contain garbled placeholder symbols such as \"??,\" \"?,\" and an unlabeled formula; these need to be cleaned up before submission.","section":"Figure 1"},{"comment":"Reference [3] is cited for the COYO dataset, but the listed paper by Carlini et al. concerns poisoning web-scale datasets and does not appear to be the COYO dataset source; this citation should be corrected.","section":"§4.1.1, References"},{"comment":"The abbreviation \"NFE\" is not defined in the table caption; it should be spelled out as the number of function evaluations or sampling steps.","section":"Table 1"},{"comment":"The phrase \"even with fewer sampling steps (less than 4)\" is imprecise because the experiments use exactly 3 and 4 steps; the wording should be adjusted accordingly.","section":"Abstract"},{"comment":"The human evaluation is reported only in the appendix; the main text should at least reference it when discussing qualitative and perceptual improvements.","section":"Appendix E"}],"recommendation":"reject","confidential_remarks":"The empirical comparison is internally coherent and the fine-tuning recipe may be of interest as an empirical study, but the central objective of Eq. (7) is mathematically degenerate as a generative objective, and the paper's own tables contradict the headline claims. I recommend rejection; a substantially rewritten paper that drops the noise-to-data mapping claim and reframes the contribution as empirical fine-tuning could be considered afresh."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one useful thing here is the recipe: fine-tune PixArt-δ by unrolling the 3–4 step LCM chain and backpropping an L2+LPIPS reconstruction loss on the final latent. That does improve COCO FID from 28.49 to 25.27 and CLIP from 30.83 to 32.76 at 4 steps, and the human eval is at least internally consistent. The ablation is honest enough about which loss combination lands where.\n\nBut the paper's framing does not survive contact with the math. Eq. (7) minimizes E_{q(z_T,z_0)}[d(z_0, \\hat z_0)] with z_T sampled independently of z_0. For L2, the optimal predictor is G_θ(z_T,c)=E[z_0|c] for every z_T; for L1 it's the conditional median. The objective alone says nothing about mapping different noises to different samples; it actively rewards producing the same per-prompt average. The only reasons the model does not visibly collapse are the pretrained PixArt-δ initialization and the short, low-LR fine-tuning (24k steps at 1e-6). Appendix A.1 admits training from scratch is not explored. So the central claim—\"directly transforms pure Gaussian noise to the target data distribution\"—is unsupported by the stated objective.\n\nThe GAN loss is the sole distribution-matching term in the paper, and the ablation shows it worsens COCO FID (25.27→25.74) and CLIP (32.76→31.75). That is consistent with the GAN fighting the degenerate reconstruction term, not complementing it. The paper instead spins this as a desirable trade-off.\n\nThere are also plain overclaims. Section 4.2 says the method \"surpasses larger models like SDXL in both image quality and alignment,\" but Table 1 shows COCO FID 25.27 vs SDXL's 14.28; only CLIP is higher. The \"information leakage\" claim is never quantified—no KL or mutual information is measured anywhere. Novelty is oversold: backpropping through the sampling chain for reward fine-tuning is in DDPO, DPOK, and DreamFusion, and adversarial distillation in SDXL-Turbo/Lightning, none of which are cited in this context.\n\nWho gets value? A practitioner might try the fine-tuning recipe in a hackathon, but the theoretical narrative is wrong and the experimental claims are inflated. This deserves a desk reject, not referee time. If the authors resubmit after dropping the unsupported language and repositioning it as an empirical study of final-sample fine-tuning, with a proper analysis of diversity and mode collapse, it could become a workshop-level paper.\n\nRecommendation: reject; do not send to peer review.","headline":"The empirical fine-tuning recipe gives modest few-step gains over PixArt-δ, but the paper's central claim—that the reconstruction objective learns a direct noise-to-data map—is mathematically wrong because z_T is independent of z_0, making the Bayes-optimal solution the per-prompt conditional mean.","tokens_in":16367,"tokens_out":3535,"would_cite":false,"duration_ms":39456,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a diffusion model by backpropagating a reconstruction loss through the full few-step sampling chain—instead of predicting noise one step at a time—directly optimizes the final generated sample, and the paper reports that this…","keywords":["diffusion models","end-to-end training","few-step sampling","latent consistency models","text-to-image generation","perceptual loss","adversarial loss","training-sampling gap"],"falsifier":"Train the identical end-to-end objective from random initialization rather than from a pretrained model; if the loss fails to decrease or the FID does not beat stepwise training, the end-to-end objective alone is not sufficient and the pretrained start is the load-bearing ingredient. A second check is to lengthen the unrolled chain to 10 or more steps and measure whether gradient norms and output quality remain stable.","tokens_in":15185,"feed_emoji":"🎨","tokens_out":7676,"duration_ms":63525,"temperature":0.7,"pith_summary":"This paper proposes that diffusion models should be trained by directly mapping pure Gaussian noise to the target data in one differentiable pass, rather than by predicting noise at randomly sampled intermediate steps. The authors introduce an end-to-end objective that measures the reconstruction error between the real latent code and the final output of the full sampling chain, so the loss is applied exactly where generation quality matters. They report that this approach removes the training-sampling discrepancy, avoids reliance on the terminal noise being perfectly Gaussian, and permits perceptual and adversarial losses to be folded into the objective. On COCO30K and HW30K benchmarks, their fine-tuned model improves FID and CLIP scores relative to stepwise-trained few-step baselines at 3–4 sampling steps, which is why a reader would care: it suggests a simple loss change can make diffusion models faster and more controllable without new architectures.","feed_headline":"Training on the final image beats step-by-step denoising","feed_subtitle":"Fine-tuning with a final-image reconstruction loss improves FID and CLIP at 3-4 sampling steps.","key_machinery":"The differentiable unrolled sampling chain is the load-bearing object. It is a fixed few-step denoising trajectory, run with a consistency-model scheduler from pure Gaussian noise to a reconstructed latent, in which every step uses the same shared network parameters; the reconstruction loss $d(z_0, \\hat{z}_0)$ is then backpropagated through the whole chain. This turns the sampling procedure itself into the training objective, so the model is optimized for the exact computation performed at inference, and it is what makes perceptual and adversarial losses attachable to the final image.","core_discovery":"The central claim is that the end-to-end reconstruction loss $L_{\\mathrm{recon}}(\\theta) = \\mathbb{E}[d(z_0, \\hat{z}_0)]$, evaluated on the final predicted latent $\\hat{z}_0$ of a fixed, fully unrolled 3–4 step denoising chain, is a sufficient and better-aligned training signal for diffusion models than the usual stepwise noise-prediction loss. The authors argue that because the loss is placed on the final output, the model learns to manage cumulative errors across steps, the training computation graph matches the sampling computation graph, and the terminal latent need not be exactly Gaussian because training is formulated as a direct noise-to-data transformation. They report that fine-tuning a pretrained text-to-image latent consistency model with this objective, using L1, L2, and LPIPS losses (and optionally an adversarial term), improves FID and CLIP scores on COCO30K and HW30K at 3–4 sampling steps, and that human evaluators prefer its outputs over the pretrained baseline.","pith_inferences":["The reported gains may depend heavily on the pretrained initialization: the paper includes no training-from-scratch experiment, so the end-to-end objective alone has not been shown to train a diffusion model stably.","The ablation shows the adversarial term actually degrades FID and CLIP slightly while adding high-frequency detail, so the advertised benefit is a qualitative realism trade-off, not a uniform improvement.","A natural testable extension is to monitor gradient norms through the unrolled chain; if gradients vanish or explode beyond the first few steps, the fixed short chain is what keeps the method stable, not the objective itself.","The information-leakage claim is conceptual: because training starts from freshly sampled Gaussian noise and only the final output is scored, the model never needs the forward noising process to reach a perfectly Gaussian terminal state."],"forward_implications":["A diffusion model can be fine-tuned with a single final-output reconstruction loss instead of per-step noise prediction, and this alone improves FID and CLIP at 3–4 sampling steps.","Perceptual and adversarial losses become straightforward additions to the core objective, because the entire sampling chain is differentiable end to end.","The training graph and the inference graph become the same computation, so error accumulation across steps is optimized directly rather than left to chance.","The same end-to-end objective is compatible with different reconstruction metrics (L1, L2, LPIPS, or combinations), making the choice of visual similarity criterion a direct design knob.","Few-step generation can be improved by fine-tuning an already-trained few-step consistency model, without adding distillation-specific losses."],"supporting_citations":[{"why":"Supplies the pretrained few-step text-to-image model used for initialization and the evaluation protocol on HW30K.","marker":"[4]"},{"why":"Supplies the consistency-model scheduler and few-step sampling procedure used to build the differentiable chain.","marker":"[23]"},{"why":"Provides the latent-space formulation the objective operates in.","marker":"[32]"},{"why":"Defines the stepwise noise-prediction objective and the training-sampling discrepancy the method targets.","marker":"[10]"},{"why":"Establishes the consistency-model idea of aligning intermediate representations that underlies few-step sampling.","marker":"[38]"},{"why":"Serves as a few-step diffusion baseline that must be beaten in the comparison.","marker":"[18]"},{"why":"Supplies the COCO benchmark used for FID and CLIP evaluation.","marker":"[19]"},{"why":"Supplies the CLIP score metric for text-image alignment.","marker":"[30]"},{"why":"Supplies the adversarial loss integrated into the end-to-end objective.","marker":"[6]"}],"fun_headline_variants":["End-to-end loss beats stepwise denoising for diffusion","Training on final output outperforms step-wise denoising","Direct noise-to-image training sharpens diffusion results","Final-image loss trims diffusion steps to 3-4","Skip stepwise losses: train diffusion on the endpoint"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that backpropagating the final reconstruction loss through the fixed 3–4 step sampling chain, starting from a pretrained model, produces stable and useful gradients that improve the generation distribution rather than destabilizing training.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end loss beats stepwise denoising for diffusion","Training on final output outperforms step-wise denoising","Direct noise-to-image training sharpens diffusion results","Final-image loss trims diffusion steps to 3-4","Skip stepwise losses: train diffusion on the endpoint"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2480,"prompt_tokens":1020,"completion_tokens":1460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":636,"tokens_out":1460,"duration_ms":10075,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:04:18.534651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical end-to-end objective from random initialization rather than from a pretrained model; if the loss fails to decrease or the FID does not beat stepwise training, the end-to-end objective alone is not sufficient and the pretrained start is the load-bearing ingredient. A second check is to lengthen the unrolled chain to 10 or more steps and measure whether gradient norms and output quality remain stable.","supporting_citations":[],"review_version":1}