{"id":"4e0c151b-2ea2-4874-98f9-69f18e699419","arxiv_id":"2608.09438","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Zero-initializing the modulation weights is the dominant reason adaLN-Zero outperforms adaLN, and replacing it with a Gaussian initialization of std 0.001 improves FID at the same training steps.","lead":"This paper dissects why adaLN-Zero, the conditioning mechanism in diffusion transformers, beats the simpler adaLN variant, and finds that starting from zero is the most important factor. From that insight it proposes a Gaussian initialization and a compressed 'SE-like' variant that improve generation quality at equal training cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attribution that zero-initialized location dominates rests on adaLN-Mix, whose 'enforced update order' is unspecified; if implemented via gradient masking or freezing, it may confound initial location with optimization schedule, so the central claim is not yet fully supported.","rationale":"The paper's central contribution is an attribution claim: zero-initialized starting location, not the SE-like scaling path or the staggered update order, is the dominant cause of adaLN-Zero's advantage over adaLN. This claim is exactly what the adaLN-Mix experiment is designed to establish, and that experiment is the only direct evidence separating initial location from update dynamics. The paper specifies neither how the update order is enforced nor any code, and the natural ways to enforce it with nonzero W_alpha (gradient masks, parameter freezing, per-parameter learning-rate schedules) all introduce auxiliary optimization changes. Without that specification, the comparison between adaLN-Mix and adaLN-Zero cannot cleanly attribute the remaining gap to initial location, so the central claim is not yet fully supported. I separately note that the long-horizon time-savings comparison (3800K vs 7000K) borrows baselines from another implementation, but the primary mechanism claim is the more fundamental issue. The paper has real strengths: the one-line adaLN-Gaussian change is simple, the 400K apples-to-apples comparison improves FID from 20.02 to 17.86, and the method generalizes across several DiT variants and datasets. These results justify conditional acceptance pending the adaLN-Mix specification and code release, which is the same verdict the reader reached; no verdict change is needed.","tokens_in":25159,"tokens_out":10490,"duration_ms":105428,"concrete_test":"Request the exact adaLN-Mix implementation and released code from the authors, then re-run the 50K-step ImageNet 256x256 DiT-XL/2 comparison under two conditions: (i) the authors' reported enforcement, and (ii) an independent control that uses adaLN-Step1 initialization with per-parameter learning-rate freezing for exactly the weight groups and iterations that Table I says should not update, with no other changes. Compare both FID trajectories against the paper's adaLN-Mix curve. If both match the reported curve, the ablation isolates update order as claimed; if either differs, the attribution of the performance gap to initial location is confounded and a revised experiment is needed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B2 introduces adaLN-Mix to separate 'zero-initialized location' from 'gradual update order': it uses adaLN-Step1's initialization (nonzero W_alpha) while 'enforce the update order of adaLN-Zero simultaneously.' The paper provides no mechanism, pseudo-code, or code for this enforcement. In adaLN-Zero, the staggered updates in Table I arise because W_alpha=0: gradients to W_ffm, W_att, W_gamma2, W_beta2, W_gamma1, and W_beta1 vanish at iterations 1-2 precisely because alpha is zero. To reproduce that schedule with nonzero W_alpha, one must intervene, for example by freezing the affected parameters, masking their gradients, or setting per-parameter learning rates to zero for the first two iterations. Any such intervention changes the optimization trajectory and the way the nonzero W_alpha interacts with the rest of the network, so the FID gap between adaLN-Mix and adaLN-Zero would reflect not only initial location but also the side effects of the enforcement mechanism. Since the paper concludes from this gap that 'it is the zero-initialized location that accounts for the remarkable performance difference' (Section III-B2), the central attribution is unverified without the adaLN-Mix implementation. The Table III result that adaLN-Step1-Gaussian matches adaLN-Zero provides supporting evidence that small scale is beneficial, but it does not by itself separate update order from location: with W_alpha ~ 1e-3 the second-iteration gradients are tiny but nonzero, giving only a soft version of the staggered schedule. The missing adaLN-Mix specification is therefore the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates why the adaLN-Zero conditioning mechanism outperforms plain adaLN in diffusion transformers. It decomposes the difference into three candidate causes: an SE-like scaling structure, the zero-initialized starting location of the modulation weights, and the staggered 'gradual' update order induced by zero initialization. A simplified gradient derivation is given to explain the update order, and an adaLN-Mix experiment is introduced to separate location from update order. The paper then observes that the modulation weights W_alpha, W_gamma, and W_beta converge to Gaussian-like distributions and proposes adaLN-Gaussian (Gaussian initialization with std 1e-3), adaLN-Gaussian-v2 with per-module stds, and an SE-adaLN-Zero architecture with a compression ratio. Extensive experiments on ImageNet, additional datasets, and text-to-image generation report FID improvements at moderate training budgets and claim parity with the converged adaLN-Zero baseline at substantially fewer steps.","tokens_in":25499,"tokens_out":5866,"duration_ms":58130,"significance":"If the central attribution is accepted, the paper offers a simple, practical, and analysis-guided initialization that improves training efficiency without changing the architecture or learning algorithm. Strengths include the one-line PyTorch change, the breadth of experiments across model sizes, DiT variants, datasets, and text-to-image tasks, and the explicit falsifiable claim about initialization location. However, the main causal claim depends on the underspecified adaLN-Mix experiment, several long-horizon comparisons mix implementations, and the Gaussian std is tuned on the same benchmark used for the headline improvement. The empirical method may be useful, but the paper's central mechanistic conclusion needs stronger support before publication.","major_comments":[{"comment":"The central conclusion that 'it is the zero-initialized location that accounts for the remarkable performance difference' rests on the adaLN-Mix experiment. The text says adaLN-Mix adopts adaLN-Step1's initialization while enforcing adaLN-Zero's update order 'simultaneously,' but no mechanism, pseudocode, or code is provided. Since adaLN-Step1's W_alpha is nonzero, the vanishing-gradient pattern in Table I does not occur naturally; the order must be imposed via freezing, gradient masking, or per-parameter learning-rate schedules. Any such intervention changes the optimization trajectory and can interact with the nonzero W_alpha, so the FID gap between adaLN-Mix and adaLN-Zero does not cleanly separate initial location from update dynamics. The Table III result that adaLN-Step1-Gaussian matches adaLN-Zero is supportive but does not by itself separate the two factors. Please specify the exact implementation and provide at least one alternative enforcement mechanism to show the conclusion is mechanism-independent.","section":"Section III-B2, Fig. 1, Table III"},{"comment":"The Gaussian std for adaLN-Gaussian (and the per-module stds for v2) is selected by grid search on ImageNet 256x256 at 50K/400K steps, and the same benchmark and training length is then used as the headline evidence of improvement. This makes the comparison at 400K partially self-referential: the hyperparameter has been tuned on the testbed used for the claim. The paper should report performance on a held-out dataset or training budget without re-tuning, or provide sensitivity analysis demonstrating that the conclusion is robust across a range of std values. It would also help to report multiple seeds or confidence intervals; margins such as DiT-B/2 (42.72 vs 42.55) and U-DiT-L (10.87 vs 10.47) are small enough that run-to-run noise may affect the conclusion.","section":"Section IV, Tables II and IV"},{"comment":"The long-horizon comparison is not apples-to-apples. The paper states that all experiments in Table IV use the fast-DiT implementation with gradient checkpointing, mixed precision, and pre-extracted VAE features, and that results 'may be slightly different from that of the original paper.' Yet the 2,352K and 7,000K adaLN-Zero baselines are borrowed from previous work and were not trained with this implementation. The headline claim that adaLN-Gaussian reaches 2.27 FID at 3,800K, matching adaLN-Zero at 7,000K, therefore mixes implementations across the comparison. The authors should either run the adaLN-Zero baseline under the identical fast-DiT pipeline for the long horizons, or clearly restrict the claim to the matched portions of the table.","section":"Section IV, Table IV and footnote 9"},{"comment":"The entropy/thermodynamics argument is not a valid theoretical justification. The statement that systems evolve toward higher entropy (the second law) is applied to neural-network weight distributions without a closed system, a defined ensemble, or a connection between entropy increase and optimization ease. Moreover, the 'Gaussian-like' evidence in footnote 6 computes KL divergence by fitting a Gaussian to the same adaLN-Zero weights used for the comparison, so it measures the quality of a Gaussian fit rather than an independent target distribution. Please reframe this as a heuristic empirical observation, or provide a rigorous statement of the assumed dynamics.","section":"Section III-C, Fig. 5, footnote 6"}],"minor_comments":[{"comment":"There is a typo: 'adaLN-Gaussin' should be 'adaLN-Gaussian'.","section":"Section IV, text-to-image paragraph"},{"comment":"The sentence listing zero gradients repeats W_gamma2 and W_beta2: 'dL/dW_gamma2, dL/dW_beta2, dL/dW_gamma2, and dL/dW_beta2' should list each weight once.","section":"Section III-B1, text after Table I"},{"comment":"The identity map I in 'Sigmoid(I*c)' is not defined; aligning the notation with the element-wise multiplication and the 1-vector in Eq. (2) would make the SE analogy clearer.","section":"Eq. (1)"},{"comment":"There are typos: 'ensensially' should be 'essentially' and 'provid' should be 'provide'.","section":"Discussion section"},{"comment":"The footnote should state which rows of Table IV use the fast-DiT implementation; the current phrasing 'all of which are employed in experiments of Tab. IV if not specified' leaves ambiguity about the 400K and 800K baseline rows.","section":"Footnote 9"},{"comment":"The sentence 'Our SE-like v2 user=4' should read 'uses r=4'.","section":"Section V, SE-like variants"}],"recommendation":"major_revision","confidential_remarks":"Dear Editor, this is a useful empirical study, but the central mechanistic claim depends on the underspecified adaLN-Mix experiment. I would ask the authors to provide the exact implementation and matched baselines. If the missing implementation reveals that the update-order enforcement changes gradient dynamics in a more invasive way, the central attribution may need to be weakened. I also note the std selection on the same benchmark; asking for holdout evidence is appropriate. No concerns about novelty disclosure or citation practice beyond the cross-implementation baseline borrowing noted in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the practical result is likely right, and the analysis is genuinely useful. The paper decomposes adaLN-Zero into three candidate factors—the SE-like scaling path, zero initialization, and the staggered update order—and argues the initial location dominates. It also shows that initializing the modulation weights from a Gaussian with std 1e-3 (adaLN-Gaussian) consistently improves FID at the same step count across DiT variants, SiT, FasterDiT, U-DiT, LlamaVision, and text-to-image. That is a one-line code change that saves real compute. The gradient table (Table I) showing which weights update first is a nice, concrete contribution, and the weight-distribution plots are suggestive. Credit where due: the empirical breadth is unusually wide for an analysis paper, and the method is simple enough to be immediately adoptable.\n\nThe soft spots are real but mostly addressable. The central attribution claim—that zero-initialized location, not the staggered update order, drives the gap—depends on the adaLN-Mix experiment in Sec. III-B2. The paper says it uses adaLN-Step1's initialization while 'enforce the update order of adaLN-Zero simultaneously,' but never says how. There is no mechanism, no pseudo-code, no code. If enforcement is done by masking or freezing gradients, it alters the loss landscape and the interaction with the nonzero W_alpha, so the FID gap between adaLN-Mix and adaLN-Zero does not cleanly separate location from order. The stress-test is right: this is the load-bearing gap. The supporting experiment adaLN-Step1-Gaussian matches adaLN-Zero, which shows small scale helps, but it does not separate the two factors.\n\nOther concerns are minor by comparison: no error bars or multiple seeds; the Gaussian std is tuned on the same 50K-step benchmark used to report gains; and the long-horizon comparison borrows baselines from the original DiT paper while the method runs use the fast-DiT implementation (the authors note this may cause slight differences). None of this sinks the practical result—the gains are consistent and repeated across many settings—but it does mean the 'why' is not yet nailed.\n\nVerdict: worth a serious referee. The practical contribution deserves publication even if the causal story needs to be softened. For a revision, I'd ask for a precise description of adaLN-Mix, at least one additional seed for the headline numbers, and apples-to-apples baselines for the long runs. I'd also ask the authors to stop claiming the Gaussian-like distribution is validated by a KL divergence computed against a Gaussian fitted to the same weights; that is circular, though the plots stand on their own.","headline":"The adaLN-Gaussian initialization trick is a real, cheap win for DiT training, but the causal story about why zero-init matters rests on an ablation that is never specified.","tokens_in":26095,"tokens_out":2700,"would_cite":true,"duration_ms":24741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that zero-initialization in DiT's adaLN-Zero wins because it puts the modulation weights at a good starting location, and that a small Gaussian start ($\\mathrm{std}=10^{-3}$) works even better.","keywords":["diffusion transformer","adaLN-Zero","zero initialization","Gaussian initialization","conditioning mechanism","image generation","Squeeze-and-Excitation","training efficiency"],"falsifier":"Train DiT-XL/2 with adaLN-Step1 initialization but with the update order enforced exactly as specified by the paper's own gradient table—only the final layer updates on step 1, then patch embedding, final-layer modulations, and $\\alpha$ producers on step 2, then everything—implemented by freezing the parameter groups that the table marks as having zero gradient, with no other changes. If this transparently enforced schedule closes most of the gap to adaLN-Zero's FID 20.02 at 400K, the paper's claim that the starting value dominates would be wrong; if the gap stays near the adaLN-Mix level, the attribution holds. A second check: initialize only $W_\\alpha$ from $\\mathcal{N}(0,10^{-3})$ and keep $W_\\gamma, W_\\beta$ at zero; if the result does not match adaLN-Zero's 400K FID in the paper's setting, the claim that $W_\\alpha$'s initial value is the decisive factor would be falsified.","tokens_in":24906,"feed_emoji":"🖼️","tokens_out":8334,"duration_ms":78937,"temperature":0.7,"pith_summary":"The paper tries to explain why adaLN-Zero outperforms plain adaLN in diffusion transformers: it examines three differences between the two mechanisms—an added scaling path, zero-initialization of the weights that produce the scale, and a stagger in which weights update first. Through an ablation that decouples these factors, it argues that the zero-initialized starting location itself is the dominant factor, with the extra scaling path helping second and the staggered update order contributing least. Building on that diagnosis, it shows that initializing those scale-producing weights from a small Gaussian ($\\mathcal{N}(0, 10^{-3})$, called adaLN-Gaussian) reaches the same image quality faster: FID 17.86 versus 20.02 at 400K steps on ImageNet 256×256, and FID 2.27 at roughly 3,800K steps, matching what adaLN-Zero reaches at 7,000K. A sympathetic reader would care because this turns an unexplained architectural trick into a transferable initialization principle, and it also yields a parameter-reduced variant, SE-adaLN-Zero, that improves FID while using fewer parameters.","feed_headline":"Zero-init's DiT secret: starting location, not slow updates","feed_subtitle":"A tiny Gaussian start for modulation weights reaches the same FID in about half the training steps on ImageNet 256.","key_machinery":"The load-bearing object is the adaLN-Zero conditioning block: the adaptive layer-norm style modulation in which a small MLP reads the conditioning embedding and outputs scale ($\\gamma$), shift ($\\beta$), and an extra per-block scaling factor ($\\alpha$), with the linear layer producing $\\alpha$ (and also $\\gamma,\\beta$ in DiT) zero-initialized. The paper's analysis instrument is the gradient table for the first iterations of a simplified DiT block, which shows that zero initialization makes weights update gradually—first the final layer, then patch embedding and $\\alpha$ producers, then everything—and an experiment dubbed adaLN-Mix that tries to isolate this update order from the initial value. The proposed method replaces the zero start with $\\mathcal{N}(0,10^{-3})$ for the modulation weights, matching the Gaussian-like distribution those weights converge to, which is what carries the claimed training-efficiency gain.","core_discovery":"The central discovery is that adaLN-Zero's advantage comes from where its modulation weights start, not from the residual shortcut or the delayed update pattern it introduces. By comparing adaLN, adaLN-Step1 (scaling added, default initialization), adaLN-Mix (default initialization but with adaLN-Zero's update order enforced), and adaLN-Zero, the paper separates three candidate causes and finds that the near-zero initial value of the scale weights ($W_\\alpha$, and similarly $W_\\gamma, W_\\beta$) accounts for most of the performance gap. It further observes that as training proceeds these weights converge to Gaussian-like distributions, so starting them from $\\mathcal{N}(0, 10^{-3})$ (adaLN-Gaussian) places the model closer to where it is heading, which speeds convergence and improves FID at matched step counts. The paper claims this is evidence that zero-initialization is \"a well-optimized location,\" not a special inductive bias of exact zero, and that a suitably chosen Gaussian is a better starting point.","pith_inferences":["A testable extension suggested by the appendix is applying the same distribution-matching idea to ControlNet's zero convolutions, which also converge to Gaussian-like distributions; initializing those convolutions with small Gaussian noise could be measured for faster conditioning control.","The paper's entropy reasoning suggests a general recipe: estimate the converged weight distribution of a modulation path early in training and initialize at that mean and standard deviation; the optimal std would then be architecture-specific rather than universally $10^{-3}$.","Because adaLN-Gaussian does not change model capacity, the paper's own long-run tables imply the payoff is compute savings (reaching the same FID in fewer steps) rather than a higher final quality ceiling; in very long training regimes, the gap narrows.","The block-wise initialization experiments in the appendix hint that the optimal Gaussian std varies across blocks and across $W_\\alpha, W_\\gamma, W_\\beta$, so a per-block, per-modulation initialization schedule is a natural next test of the same principle."],"forward_implications":["Initializing the modulation weights of DiT with $\\mathcal{N}(0,10^{-3})$ consistently improves FID at matched training steps across DiT-B/2, DiT-L/2, DiT-L/4, and 512×512 ImageNet.","At roughly 3,800K steps, adaLN-Gaussian reaches FID 2.27 that adaLN-Zero reaches at 7,000K, a roughly 46% saving in training steps at matched quality.","The benefit transfers to training systems built on DiT (SiT, FasterDiT), to DiT-based models (LlamaVision, U-DiT), and to text-to-image generation, where adaLN-Gaussian lowers COCO FID-30K from 71.41 to 65.51 at 50K steps.","SE-adaLN-Zero, an SE-inspired compression of the conditioning MLP, reduces parameters from 676M to 582M and improves FID from 20.02 to 19.13 at 400K steps; combining it with Gaussian initialization gives FID 18.76.","The paper's diagnosis implies that exact zero is not special: any initialization that lands the modulation weights near their converged distribution should give similar or better early-training behavior."],"supporting_citations":[{"why":"Defines DiT and the adaLN-Zero versus adaLN comparison that is the paper's object of study, and supplies the training recipe and baseline FIDs.","marker":"[1]"},{"why":"Provides the Squeeze-and-Excitation module whose structural similarity to adaLN-Step1 motivates the SE-like-structure factor and the later SE-adaLN-Zero design.","marker":"[20]"},{"why":"Historical reference for zero-initialization as a way to nullify output pathways and ease early optimization, which the paper tests and partially overturns.","marker":"[40]"},{"why":"Xavier initialization, the default init used by adaLN-Step1 and a comparison point for the effect of initialization scale.","marker":"[36]"},{"why":"Kaiming initialization, the standard variance-aware alternative that the paper contrasts with zero and Gaussian initialization.","marker":"[39]"},{"why":"PixArt-α, whose conditioning design inspired the SE-like v3 variant considered in the SE-adaLN-Zero search.","marker":"[13]"},{"why":"ControlNet's zero convolution, examined in the appendix to check whether the Gaussian-like convergence pattern extends beyond DiT.","marker":"[45]"}],"fun_headline_variants":["DiT's adaLN-Zero edge is its start location, not slow updates","Zero-init's real DiT advantage: starting point, not update order","Gaussian start for DiT modulation weights speeds training","Why adaLN-Zero wins: initial weight location, not mechanism"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole attribution to \"initial location\" rests on the adaLN-Mix experiment, which the paper describes as using adaLN-Step1's initialization while enforcing adaLN-Zero's update order, but gives no mechanism, pseudocode, or code for how that enforcement is done; if the enforcement changes gradients on other weights (for example by masking or rescaling), the separation between starting point and update dynamics is not clean.","fun_headline_variants_meta":{"raw":{"variants":["DiT's adaLN-Zero edge is its start location, not slow updates","Zero-init's real DiT advantage: starting point, not update order","Gaussian start for DiT modulation weights speeds training","Why adaLN-Zero wins: initial weight location, not mechanism"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3256,"prompt_tokens":974,"completion_tokens":2282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2204}},"tokens_in":590,"tokens_out":2282,"duration_ms":19405,"temperature":1.0,"reasoning_tokens":2204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:38.617859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DiT-XL/2 with adaLN-Step1 initialization but with the update order enforced exactly as specified by the paper's own gradient table—only the final layer updates on step 1, then patch embedding, final-layer modulations, and $\\alpha$ producers on step 2, then everything—implemented by freezing the parameter groups that the table marks as having zero gradient, with no other changes. If this transparently enforced schedule closes most of the gap to adaLN-Zero's FID 20.02 at 400K, the paper's claim that the starting value dominates would be wrong; if the gap stays near the adaLN-Mix level, the attribution holds. A second check: initialize only $W_\\alpha$ from $\\mathcal{N}(0,10^{-3})$ and keep $W_\\gamma, W_\\beta$ at zero; if the result does not match adaLN-Zero's 400K FID in the paper's setting, the claim that $W_\\alpha$'s initial value is the decisive factor would be falsified.","supporting_citations":[],"review_version":1}