{"id":"cb8bcfbb-33e0-49c7-a3c2-6a330162139b","arxiv_id":"2412.12444","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LazyDiT learns small gates that decide when to reuse cached layer outputs, cutting diffusion transformer compute by up to half while matching or beating DDIM quality.","lead":"LazyDiT speeds up diffusion transformer image generation by reusing cached intermediate results from the previous denoising step, skipping up to half the computation in attention and feedforward modules. The method achieves better or comparable image quality than the DDIM sampler at matched compute, including on a smartphone GPU.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Similarity lower bound is not proven: Theorem 18 uses an unconstrained R2, and Lemma 12 constructs scalings outside DiT's AdaLN parameterization.","rationale":"The reader's weakest_assumption targets the AdaLN mismatch in Lemma 12/Theorem 13, and I agree that is a real gap. I add two related gaps: Theorem 18's proof never uses the scaling bound and instead bounds similarity using an unconstrained R2, making the claimed high similarity vacuous; and Theorem 19's constructed W is time-dependent and not the learned gate, so the linear-approximation story is not what is trained. However, the central empirical claim — matched-TMAC/latency FID improvements over DDIM — is credible: the paper provides matched TMACs in Table 3, matched mobile latency, and a comparison to Learn2Cache. The soft/hard switch mismatch and missing DeepCache/Block Caching baselines are weaker concerns. I therefore agree with the CONDITIONAL verdict: the theoretical claims should be fixed or removed, and the empirical evaluation should add the missing baselines, but the paper's main empirical finding is not overturned by this critique.","tokens_in":29052,"tokens_out":11753,"duration_ms":109805,"concrete_test":"Run the released DiT-XL/2 (256x256) with the paper's DDIM 50-step schedule and, for each of the 28 layers, record Y_attn and Y_ffn at every step. Compute the true cosine similarity f(Y_{t-1}, Y_t) for each module and compare with the trained LazyDiT gate s_{l,t} at the same t. If the median true similarity is not close to 1 (e.g., <0.8) or if the rank correlation between s and the true similarity is low (e.g., <0.3), then the theoretical premise that high similarity is predictable by the learned linear layer is empirically false, and the method's success must be attributed to the end-to-end training of the gates rather than to the stated similarity theory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's theoretical demonstration that consecutive-step outputs are highly similar is not valid as written. Theorem 18 (Appendix C.4) defines alpha = 0.5 C^2 R2^2 min{N,D}, where R2 is an assumed upper bound on ||X^Phi_{l,t-1} - X^Phi_{l,t}||. No argument shows R2 is small; the only small-distance result, Theorem 13, applies to Z after scaling and is never used in the proof. In fact Theorem 18's bound is vacuous unless R2 is controlled, and the informal Theorem 2 in the main text silently replaces the distance in X with the distance in Z, changing the statement. Second, Lemma 12 constructs scaling vectors a = 1_D * 0.5 eta/(N M1) with all coordinates equal, and b similarly, with C = 0. DiT's AdaLN uses at = W_{l,a} y_t + v_{l,a} with y_t = SiLU(emd(t)+emd(c)); the proof never shows there exist timestep/class embeddings y_t, y_{t-1} whose linear projections realize these a, b, nor that a single y_t simultaneously controls both scaling and shifting. So Theorem 13 does not apply to the actual model. Third, Theorem 19's linear approximation defines W_Phi_l = (Y^Phi_{l,t-1})^T J, which depends on t and has shape N x D; the implemented gate is a learned D x 1 vector shared across all timesteps, and the O(1) Taylor error is dropped without quantification. These gaps do not disprove the empirical matched-TMAC/latency results, but they remove the stated theoretical support: the paper has not shown that the linear layer approximates cosine similarity, nor that similarity is high.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"LazyDiT inserts lightweight trained linear gates before each MHSA and feedforward module of DiT/Large-DiT; at inference, a module is skipped and replaced by its previous-timestep cache whenever the gate's predicted similarity exceeds 0.5. The paper claims a theoretical guarantee that consecutive module outputs are highly similar and that the similarity is linearly approximable, and supports this with a lazy-loss training procedure. Empirically, on ImageNet with cfg=1.5 and 50k generated images, LazyDiT at 20-50% lazy ratios achieves FID/IS better than DDIM at similar or lower TMACs across DiT-XL/2 (256/512), DiT-L/2, Large-DiT-3B, and Large-DiT-7B, and mobile latency is profiled on a Snapdragon 8 Gen 3 device.","tokens_in":29454,"tokens_out":6769,"duration_ms":62002,"significance":"If the empirical results are taken at face value, the contribution is practically significant: it is a cheap, post-hoc cache-skip method that improves the FID-vs-compute tradeoff of strong diffusion transformers, and the authors provide unusually detailed mobile latency measurements, a comparison against Learn2Cache, and public code. The evaluation is anchored to the external ImageNet-FID benchmark and the lazy ratios are measured after training, so the central system claim is not circular. However, the theoretical support in Section 3.2 and Appendix C is not established: the existence proofs are not tied to DiT's AdaLN parameterization, the formal similarity bound is vacuous without control of R2, and the implemented linear gate differs from the Taylor-derived weight. The paper would be stronger if the theory were either repaired or repositioned as empirical motivation rather than as proof.","major_comments":[{"comment":"The proof constructs uniform scaling vectors a and b from the hidden states X1 and X2, with c = 0, but DiT's actual scaling and shifting are at = W_{l,a} y_t + v_{l,a} and bt = W_{l,b} y_t + v_{l,b}, where y_t = SiLU(emb(t)+emb(c)) depends only on the timestep and class embedding (Section 3.2). The paper never shows that the pretrained AdaLN projection matrices and embeddings can realize these constructed a, b, or that a single y_t can simultaneously satisfy the scaling and shifting conditions. Therefore Theorem 1, and through Eq. (4) also Theorem 2, does not apply to the actual model as written.","section":"Appendix C.2, Lemma 12 and Theorem 13"},{"comment":"The formal similarity bound defines α := 0.5 C^2 R2^2 min{N,D}, where R2 is an assumed upper bound on ||X^Φ_{l,t-1} - X^Φ_{l,t}||, but no argument shows R2 is small. Moreover, Eq. (4) bounds Dist by C||Z^Φ_{l,t-1} - Z^Φ_{l,t}||, whereas Theorem 18 bounds the distance in terms of ||X^Φ_{l,t-1} - X^Φ_{l,t}||; the informal Theorem 2 in the main text combines the Z-distance bound with Theorem 1, so the formal and informal statements do not match. As written, the theorem is vacuous unless an additional control on R2 is supplied.","section":"Appendix C.4, Theorem 18"},{"comment":"Both Theorem 18, Eq. (8), and Theorem 19, second equality, use the assumption ||Y^Φ_{l,t}||_F = 1 for every t. This normalization is not established for the actual DiT modules, and the implementation does not appear to enforce it. Without this assumption the cosine similarity is not equal to 1 - 0.5||Y_{t-1} - Y_t||^2, so the derived lower bound and the linear approximation do not follow.","section":"Appendix C.4 and C.5, unit-norm assumption"},{"comment":"The proof defines W^Φ_l := (Y^Φ_{l,t-1})^T J, which is an N x D matrix that depends on the timestep t, while the implemented gate is a D x 1 weight vector shared across all timesteps (Section 3.3, s^Φ_{l,t} = sigmoid((Z^Φ_{l,t} · W^Φ_l) · 1_N)). In addition, the Taylor expansion in Eq. (9) contains an O(1) error term that is dropped in the main-text Theorem 3 equality. Consequently, the paper has not demonstrated that the implemented linear layer approximates cosine similarity.","section":"Appendix C.5, Theorem 19"}],"minor_comments":[{"comment":"The text says 'lazy ratio ΓΦ ∈ ZB of MSHA or Feedforward'; 'MSHA' should be 'MHSA'.","section":"Section 3.3, after Eq. (5)"},{"comment":"The statement 'f (Y_{l,t-1}, Y_{l,t}) = 1 - Dist/2' is dimensionally inconsistent; the actual relation from Fact 7 is f = 1 - 0.5||Y_{t-1} - Y_t||^2_F when both matrices have unit Frobenius norm. Please correct this in the main text.","section":"Section 3.2, Fact 7 paraphrase"},{"comment":"The citation 'LargeDiT (Zhang et al. 2023)' points to LLaMA-Adapter, which is not the source of the Large-DiT models used in the experiments; the reference appears mismatched and should be corrected.","section":"Section 4.1, Model Family"},{"comment":"The matched-compute comparison is central to the empirical claim, but Tables 1 and 2 do not report TMACs; only Table 3 reports TMACs for a subset of configurations. Adding TMAC columns to the main tables would make the comparison transparent.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is credible and the mobile latency measurements are a strong point. My recommendation is driven by the theoretical section, which contains load-bearing gaps that the authors should either repair or explicitly re-scope as heuristic motivation. I would not recommend rejection based on the theory gaps alone, because the main system claim is empirical and is benchmarked against an external ground truth."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the empirical system, not the theory. The per-module learned lazy gates with a lazy loss over cached MHSA/Feedforward outputs are genuinely new relative to DeepCache, Block Caching, and Learn2Cache, and the matched-TMAC comparisons are the right way to evaluate a caching scheme. At 50% lazy ratio with 50 steps, LazyDiT beats DDIM at 25 steps at the same TMACs (FID 2.70 vs 2.95 on DiT-XL/2 256, and similar wins on Large-DiT 3B/7B). The mobile latency numbers on a Snapdragon 8 Gen 3 are concrete, internally consistent with the TMACs, and make the deployment story plausible. Training the gates is cheap: 500 steps with frozen base weights. That is a solid, practical contribution.\n\nThe soft spots are real but mostly contained. The theoretical section is not sound as written. Lemma 12 constructs scaling vectors that depend on the hidden-state norms, but DiT's AdaLN scaling is a fixed linear function of the timestep/class embedding; the proof never shows those exact vectors are realizable by any y_t. Theorem 18's formal bound goes through an unconstrained R2 on the pre-scaling inputs, never uses the small-distance result in Z, and the informal version silently swaps X for Z. Theorem 19 defines a gate weight that is N x D and depends on t, while the implementation is a D x 1 vector shared across all timesteps; the O(1) Taylor error is dropped. None of this disproves the empirical tables, but it does mean the paper's headline claim that similarity is provably high and linearly approximable is unsupported. The theory should be excised or heavily revised.\n\nTwo smaller flags. The paper cites DeepCache and Block Caching but only benchmarks Learn2Cache, so the claim to beat caching baselines is only partly checked. And there is a soft-to-hard switch mismatch: training blends module outputs with cache via a continuous sigmoid, inference threshold-skips. That is common, but they give no analysis of how much it matters. FIDs are single-run, so the smaller gaps (0.1-0.2) are within noise, but the large low-step gains (e.g., 6.75 vs 12.05 at 20 steps 50%) are clearly beyond it.\n\nBottom line: this is a solid systems paper with a broken theory appendix. A serious referee should engage with it, but should demand that the theoretical claims either be made correct or be dropped. The empirical method deserves publication; the proofs as written do not. My call: conditional accept after major revision, with the theory trimmed and the missing baselines added.","headline":"Useful cache-based DiT acceleration with credible matched-compute FID/latency gains, but the theoretical guarantees as written do not hold up and should be cut or rewritten.","tokens_in":30005,"tokens_out":4418,"would_cite":true,"duration_ms":42945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"LazyDiT claims that about half the per-step compute of a diffusion transformer can be skipped by reusing cached outputs from the previous denoising step, with a small trained linear layer deciding when to skip.","keywords":["diffusion transformers","lazy caching","inference acceleration","DDIM sampling","similarity lower bound","linear gate approximation","ImageNet generation","mobile deployment"],"falsifier":"Measure the actual cosine similarity $f(Y^{\\Phi}_{l,t-1}, Y^{\\Phi}_{l,t})$ for every MHSA and feedforward module of a pretrained DiT-XL/2 during 50-step DDIM sampling. If the average similarity is far from $1-\\alpha$ (say, below 0.9 at any stage), or if the measured distance $\\|Z^{\\Phi}_{l,t-1} - Z^{\\Phi}_{l,t}\\|$ under the real fixed scaling and shifting parameters is not small, the theoretical premise of the paper fails even if the learned gates still work.","tokens_in":28826,"feed_emoji":"⚡","tokens_out":8449,"duration_ms":69664,"temperature":0.7,"pith_summary":"LazyDiT claims that a diffusion transformer does not need to run every attention and feedforward module at every denoising step, because the outputs of these modules at consecutive steps are very similar. The paper derives a lower bound on that similarity, shows the similarity can be approximated by a linear function of the current input, and turns this into a trainable gate: a small linear layer decides per module whether to recompute or just reuse the previous step's cached output. At a 50 percent lazy ratio, LazyDiT matches or beats the DDIM sampler at equal compute, for example FID 2.70 versus 2.95 on DiT-XL/2 at 256 resolution. If the claim holds, roughly half the per-step compute of large diffusion transformers can be skipped without hurting sample quality, which matters for deploying these models on phones and GPUs.","feed_headline":"LazyDiT skips half the compute and beats DDIM","feed_subtitle":"Learned gates reuse cached layer outputs, improving FID at equal cost on DiT and LargeDiT models.","key_machinery":"The load-bearing object is the lazy learning layer: a linear map $W^{\\Phi}_l \\in \\mathbb{R}^{D\\times 1}$ inserted before each MHSA and feedforward module, producing a scalar gate $s^{\\Phi}_{l,t} = \\operatorname{sigmoid}((Z^{\\Phi}_{l,t} W^{\\Phi}_l)\\,1_N)$, trained with a lazy loss that rewards skipping. Its justification is the paper's three-step chain: Theorem 13 states that time- and condition-dependent scaling and shifting vectors can make consecutive-step inputs arbitrarily close; Theorem 18 uses Lipschitz continuity of attention and feedforward modules to convert that closeness into a high cosine-similarity lower bound; Theorem 19 uses a Taylor expansion around the current input to represent the similarity as an inner product with a weight matrix. During sampling, the gate replaces the module with the previous step's cached output when the similarity score exceeds 0.5.","core_discovery":"On its own terms, the paper establishes that the cosine similarity between consecutive-step outputs of an MHSA or feedforward module is bounded below by $1-\\alpha$ with small $\\alpha$ (Theorem 2 / Theorem 18), because scaling and shifting can bring consecutive inputs close and the modules are Lipschitz. It then shows that this similarity admits a linear approximation, $f(Y^{\\Phi}_{l,t-1}, Y^{\\Phi}_{l,t}) = \\langle W^{\\Phi}_l, Z^{\\Phi}_{l,t}\\rangle$ (Theorem 3 / Theorem 19), so a linear layer can predict when a module's output would be redundant. Feeding that prediction through a sigmoid yields a per-module skip decision: recompute when $s \\le 0.5$, reuse the cached previous-step output when $s > 0.5$. With the original model frozen and only these small linear layers trained for 500 steps, LazyDiT improves FID over DDIM at matched or lower TMACs (2.70 vs 2.95 at 2.87 TMACs on DiT-XL/2), and similar gains appear on Large-DiT-3B/7B and on mobile latency.","pith_inferences":["One implicit consequence the paper does not develop: if consecutive-step similarity is as high as claimed, lazy skipping should compose with step-distillation and fast ODE solvers, potentially cutting the 10-step regime further.","The learned gate could be analyzed as a function of timestep and layer; a testable extension is to see whether gate decisions form a stable schedule that transfers across resolutions and classes without retraining.","The theory's construction of scaling and shifting vectors from the hidden states themselves suggests the empirical similarity should be measured directly on pretrained models; if it is high, a non-learned threshold schedule could obtain similar speedups without any training.","Because the linear approximation carries a Taylor remainder, the strongest practical headroom may sit at moderate lazy ratios, and the uniform 0.5 threshold may be suboptimal; per-layer thresholds could squeeze out more skips."],"forward_implications":["At a 50% lazy ratio, LazyDiT reaches better FID than DDIM at the same TMACs (2.70 vs 2.95 on DiT-XL/2 at 256x256), meaning users can keep 50 sampling steps but pay the compute of a 25-step run.","In few-step regimes the gap is larger: a 10-step LazyDiT run with 30% lazy ratio beats a 7-step DDIM at similar or less compute (FID 17.05 vs 34.14 at 256x256).","The gains transfer across model families and resolutions: DiT-XL/2, DiT-L/2, Large-DiT-3B and Large-DiT-7B all match or beat DDIM at equal cost, with more redundancy in the billion-parameter models.","On a Snapdragon 8 Gen 3 phone, LazyDiT delivers a better inception score than DDIM at nearly identical latency, e.g. 136.81 vs 91.67 at 0.80 TMACs, so the method is directly deployable on mobile.","Layer-wise lazy-ratio patterns show no layer is ever fully skipped, and MHSA redundancy is concentrated in early layers while feedforward redundancy is concentrated in later layers, implying that layer pruning would be a worse strategy than per-step caching."],"supporting_citations":[{"why":"Supplies the DDIM sampler that LazyDiT accelerates and the step-count baseline every comparison is measured against.","marker":"Song, Meng, and Ermon 2020"},{"why":"Supplies the DiT architecture and pretrained DiT-XL/LargeDiT models, including the per-timestep scaling and shifting mechanism the theory analyzes.","marker":"Peebles and Xie 2023"},{"why":"Provides Lemma H.5, the attention Lipschitz bound that carries the similarity lower bound in Theorem 18.","marker":"Deng et al. 2023"},{"why":"Provides the standard diffusion training objective that LazyDiT keeps as its base loss beside the lazy loss.","marker":"Ho, Jain, and Abbeel 2020"},{"why":"Establishes the latent diffusion formulation whose VAE latent space underlies the evaluated DiT models.","marker":"Rombach et al. 2022b"},{"why":"The cache-based Learn2Cache baseline that LazyDiT is compared against in Table 7 and beats at similar training cost.","marker":"Ma et al. 2024"}],"fun_headline_variants":["LazyDiT: Skip redundant diffusion compute, beat DDIM","LazyDiT: Lazily reuse steps to run diffusion faster","Caching cuts diffusion compute — LazyDiT beats DDIM","LazyDiT: Skip recompute, reuse outputs, win on FID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after the model's scaling and shifting, the inputs to a module at consecutive denoising steps are close enough that the outputs are nearly identical. The proof constructs scaling and shifting vectors directly from the hidden states, but the real model obtains them as fixed linear functions of the timestep and class embedding, and the paper never shows that those real vectors satisfy the required closeness bound.","fun_headline_variants_meta":{"raw":{"variants":["LazyDiT: Skip redundant diffusion compute, beat DDIM","LazyDiT: Lazily reuse steps to run diffusion faster","Caching cuts diffusion compute — LazyDiT beats DDIM","LazyDiT: Skip recompute, reuse outputs, win on FID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0009,"raw_usage":{"total_tokens":3911,"prompt_tokens":1014,"completion_tokens":2897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2819}},"tokens_in":630,"tokens_out":2897,"duration_ms":19232,"temperature":1.0,"reasoning_tokens":2819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:05:05.460628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual cosine similarity $f(Y^{\\Phi}_{l,t-1}, Y^{\\Phi}_{l,t})$ for every MHSA and feedforward module of a pretrained DiT-XL/2 during 50-step DDIM sampling. If the average similarity is far from $1-\\alpha$ (say, below 0.9 at any stage), or if the measured distance $\\|Z^{\\Phi}_{l,t-1} - Z^{\\Phi}_{l,t}\\|$ under the real fixed scaling and shifting parameters is not small, the theoretical premise of the paper fails even if the learned gates still work.","supporting_citations":[],"review_version":1}