{"id":"d39a7534-eb9a-4250-8555-cb212d91e5fd","arxiv_id":"2505.21817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ALTER jointly learns layer pruning masks and timestep-to-expert routing for diffusion UNets, cutting MACs to 26% of the 50-step baseline while keeping FID and CLIP scores near parity.","lead":"This paper introduces ALTER, a method that prunes different layers of a diffusion model at different denoising steps, guided by a small network that learns which layers to skip and which expert sub-network to route each step to. The result is a diffusion model that runs in about a quarter of the compute of the original while producing images of similar quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 3.64x speedup is unverified: no 50-step wall-clock latency is reported, and the number is inconsistent with the paper's own 38.04T/9.89T MAC ratio (3.85x).","rationale":"The reader's weakest assumption was that MACs translate linearly into latency and that the 50-step baseline latency is missing; my check confirms this is the weakest link in the paper's central efficiency claim. I sharpen it by noting that the paper's own Table 2 numbers are inconsistent with the reported 3.64x (38.04/9.89 = 3.85), and that Table 2's speedup column also mismatches MAC ratios for DiP-GO, so the speedup basis is unclear. This does not necessarily invalidate the method: ALTER's FID and CLIP results are plausible, the ablations support the temporal-expert design, and if actual 50-step latency is ~7.6-8.0s, the realized speedup may still be close to claimed. However, the central quantitative headline cannot be checked from the paper as written, so the verdict should remain conditional pending the latency measurement. I do not endorse the reader's secondary objections (e.g., the 'single-stage' wording) as load-bearing for the central efficiency claim, since Appendix A's final fine-tuning step mainly affects framing, not the measured speed-fidelity trade-off.","tokens_in":17291,"tokens_out":6989,"duration_ms":62770,"concrete_test":"Measure end-to-end wall-clock latency on the same A100, batch size 1, same prompts and PNDM sampler for: (a) SDv2.1 50-step, (b) SDv2.1 20-step, (c) ALTER (0.65) 20-step, with at least 3 repeated runs. Compute the actual speedup as latency(a)/latency(c), and compare it to the claimed 3.64x and to the MAC ratio 38.04T/9.89T=3.85x. If the measured speedup is within 10% of 3.64x, the claim is salvageable; if it tracks 3.85x, correct the headline; if it is materially below 3.64x, the central efficiency claim is overstated and the paper should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the 3.64x speedup with 20 steps and 35% sparsity (Abstract; Table 2). The paper never reports measured wall-clock latency for the exact 50-step SDv2.1 baseline in Table 2, so the speedup column cannot be audited. Moreover, the reported numbers do not cohere: Table 2 gives SDv2.1 (50 steps) as 38.04T MACs and ALTER (0.65, 20 steps) as 9.89T MACs; 38.04/9.89 = 3.85, not 3.64. The discrepancy is not a rounding effect, and the same table shows similar mismatches for DiP-GO (e.g., 38.04/7.61 = 5.00 versus reported 3.81x), suggesting the speedup column mixes MAC ratios with externally inherited or separately measured latency numbers. Table 1's latencies (ALTER 20-step 2.1s vs SDv2.1 25-step 4.0s) imply at most ~1.9x over a 20-step baseline and roughly 3.8x over a linearly extrapolated 50-step baseline, again not 3.64x. Because MACs ignore router overhead, memory bandwidth, and non-prunable fixed layers, the headline acceleration rests on an unverified proxy unless the missing 50-step latency is supplied.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ALTER proposes a unified framework for accelerating Stable Diffusion v2.1 by combining layer-wise pruning with timestep-conditioned expert routing. A hypernetwork generates binary pruning masks for multiple expert subnetworks and a temporal router assigns each denoising timestep to an expert; the UNet and hypernetwork are trained in an alternating scheme. The authors report that ALTER achieves FID/CLIP comparable to the original 50-step SDv2.1 while using about 25.9% of the MACs with 20 inference steps, claiming a 3.64x speedup at 35% sparsity. Experiments on CC3M and MS-COCO compare against BK-SDM, APTP, and DiP-GO, with ablations on the number of experts, sparsity ratio, and the joint-training strategy.","tokens_in":17637,"tokens_out":4787,"duration_ms":44915,"significance":"The core idea of temporal expert routing for pruned diffusion models is timely and has clear practical motivation: it addresses the inflexibility of static pruning and the low parameter utilization of sample-wise dynamic pruning. The ablation results in Table 3 support the contribution of each design component, and the reported quality at reduced MACs is encouraging if confirmed. However, the paper's main quantitative claims—the speedup factor and the single-stage characterization—are not currently backed by the reported data, which is an issue that must be resolved before the significance can be assessed. The paper does not provide code or error bars, so the robustness of the empirical results remains unverified.","major_comments":[{"comment":"The headline 3.64x speedup is not consistent with the reported MACs. In Table 2, SDv2.1 (50 steps) has 38.04T MACs and ALTER (20 steps) has 9.89T MACs; 38.04/9.89 = 3.85, not 3.64. The same table shows similar mismatches for the DiP-GO baselines (38.04/11.42 = 3.33 vs. reported 3.02; 38.04/7.61 = 5.00 vs. reported 3.81). Since no 50-step wall-clock latency is reported anywhere, the speedup column cannot be audited. The authors should either (a) report measured wall-clock latencies for the exact 50-step baseline in Table 2 and recompute speedups, or (b) clearly label the column as a MACs ratio and correct the inconsistent numbers. The abstract's '3.64x speedup' should be revised accordingly.","section":"Abstract and Table 2"},{"comment":"The paper repeatedly claims 'single-stage optimization' (Abstract, Section 1, Section 3.3) and uses Table 3 to argue that joint training outperforms a two-stage approach. However, Appendix A explicitly states that 'a final fine-tuning stage for the dynamically pruned U-Net is still found to be a necessary step after T_end is reached.' This contradicts the single-stage claim and leaves the meaning of 'Joint Training' in Table 3 ambiguous. The authors must clarify whether this final fine-tuning is applied in all reported experiments, and if so, how the ablation distinguishes the alternating co-optimization from a two-stage procedure. Without this clarification, the central methodological contribution is not well-defined.","section":"Section 3.3 and Appendix A"},{"comment":"The MACs and latency figures are internally inconsistent across tables. Table 1 reports SDv2.1 with 25 steps as 1384.2G MACs (implying ~55G per step), whereas Table 2 reports SDv2.1 with 50 steps as 38.04T MACs (implying ~761G per step), a factor of about 14 difference in per-step MACs. Additionally, in Table 1 the latency for SDv2.1 is 4.0s for both 25 and 20 steps, which is implausible and suggests the measurement protocol or the definition of 'MACs' differs between tables. The authors should specify the measurement setup (resolution, batch size, whether MACs include the autoencoder/text encoder, and how latency is averaged) and ensure consistency across all tables.","section":"Tables 1 and 2"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported for any of the evaluation metrics. Many of the headline comparisons involve differences of 0.1-0.3 in FID or CLIP (e.g., ALTER 20-step vs. SDv2.1 20-step in Table 1, and ALTER vs. the 'Manual' variant in Table 3), which may be within noise. The authors should run at least three independent training runs with different random seeds and report mean and standard error for the main quantitative claims, or at minimum for the ablation study.","section":"Tables 1-3 and Section 4.2"}],"minor_comments":[{"comment":"The phrase 'We prune the models as 65%' is ambiguous: it could mean a target sparsity of 65% (i.e., 35% of weights retained) or a retained-MAC ratio of 65%. Please clarify the definition of p and the sparsity terminology used throughout the paper.","section":"Section 4.1"},{"comment":"The conclusion mentions '3.64× speedup ... with 35% sparsity,' while the abstract says 'utilizing only 25.9% of its total MACs.' These two numbers are not obviously consistent (1/0.259 ≈ 3.86), so please state explicitly how the speedup is derived from the sparsity and step-count information.","section":"Conclusion and Abstract"},{"comment":"The notation '$x_T^{total:1}$' and '$x_T^{total-1:0}$' in the figure is unclear and appears to contain typos. Please replace with a clear description of the timestep routing variable.","section":"Figure 2"},{"comment":"The word 'Abalation' is misspelled; it should be 'Ablation'.","section":"Section 4.3 heading"},{"comment":"'depedent' is a typo for 'dependent' in 'the Demb is depedent on the pre-trained official SDv2.1 model.'","section":"Appendix C"},{"comment":"The algorithm continues updating the UNet after T_end but stops updating the hypernetwork; this is effectively a second fine-tuning phase. Please make this explicit in the main text or revise the description to align with the 'single-stage' terminology.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The inconsistent speedup numbers and the late disclosure of a final fine-tuning stage are the two issues most likely to trigger reviewer skepticism. Please encourage the authors to provide raw latency numbers, a corrected Table 2, and a precise description of the training phases. Also suggest that they add error bars and release code to enable reproducibility checks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nALTER is a real step forward in diffusion pruning: the idea of timestep-level expert routing, where a hypernetwork generates both pruning masks and timestep-to-expert assignments in one co-optimization with the UNet, is a genuine addition over static pruning (BK-SDM) and prompt-wise routing (APTP). The method is clearly described and the empirical results on CC3M/MS-COCO are competitive: at 65% sparsity and 20 steps, they match or slightly beat the 25-step SDv2.1 baseline on FID/CLIP, and the ablation shows that the learned router beats fixed intervals and static masks. That is a credible result.\n\nThe soft spots are real but addressable. The headline 3.64x speedup is not supported by the paper: it is a ratio of MACs, not measured wall-clock latency. Worse, the internal numbers don't cohere—Table 2 shows 38.04T for the 50-step baseline and 9.89T for ALTER, which gives 3.85x, not 3.64x. The same table shows DiP-GO's speedup inconsistent with its MACs. So the speedup column is either mis-computed or mixes in some external latency numbers. The paper also claims 'single-stage' optimization, but Appendix A admits a final fine-tuning stage is necessary after T_end. That is a substantive mismatch between the abstract and the method as actually run. Also, no error bars, no code, no reported training cost.\n\nNone of that kills the central contribution. The routing mechanism and the quality trade-off are independent of the exact speedup number. But the paper overstates its headline, and the internal inconsistency casts doubt on the accuracy of the efficiency table. A referee would need to ask for measured latency, corrected speedup numbers, and a clear statement of the training stages.\n\nThis is a paper I would send out for peer review, not desk-reject. The core idea is worth the community's attention, and the flaws are fixable in revision. I would not cite it until the speedup number is fixed.","headline":"A genuine new combination of timestep routing and layer pruning, with competitive quality numbers, but the headline speedup is unverified and the single-stage claim needs cleanup.","tokens_in":18166,"tokens_out":2471,"would_cite":false,"duration_ms":24089,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ALTER trains one diffusion UNet into ten timestep-specialized pruned experts, matching full-model image quality at about a quarter of the compute.","keywords":["diffusion models","layer pruning","temporal expert routing","hypernetwork","mixture of experts","inference acceleration","Stable Diffusion"],"falsifier":"Measure end-to-end wall-clock latency for ALTER's 20-step model and the original 50-step SDv2.1 on the same GPU with batch size 1, including all routing and layer-skipping overhead, and compare the ratio to 3.64; a substantially smaller ratio would show the MAC-based speedup overstates real acceleration.","tokens_in":17091,"feed_emoji":"⚡","tokens_out":7394,"duration_ms":64946,"temperature":0.7,"pith_summary":"ALTER sets out to show that a diffusion model can be made much cheaper without training a separate pruned model: the same UNet is transformed into a mixture of \"temporal experts,\" each a layer-pruned subnetwork specialized to a range of denoising timesteps. A trainable hypernetwork learns both which layers each expert keeps and which expert handles each timestep, and these decisions are refined jointly with the UNet's fine-tuning in a single stage rather than pruning first and fine-tuning later. On Stable Diffusion v2.1, the paper reports that the resulting 20-step model matches the visual fidelity of the original 50-step model while using 25.9% of its total multiply-accumulate operations, giving a 3.64x speedup at 35% sparsity. The motivation is that different phases of denoising genuinely need different network capacity, so pruning decisions should be timestep-aware rather than static or prompt-only.","feed_headline":"Pruning + timestep routing matches 50-step quality at 26% compute","feed_subtitle":"A trainable hypernetwork prunes the UNet into 10 temporal experts, cutting MACs to 25.9% with no fidelity loss.","key_machinery":"The load-bearing object is the trainable hypernetwork $H_\\Phi$, which contains an Expert Generator $G$ that maps frozen orthogonal embeddings to $N_e \\times N_L$ layer-mask logits and a Temporal Router $R$ that maps timestep embeddings to routing logits over experts. Binary masks and one-hot expert selections are obtained with Gumbel-Sigmoid and Gumbel-Softmax combined with the straight-through estimator, so discrete architecture decisions remain differentiable. Layer skipping is simulated by $x_{\\mathrm{out}} = (1 - (m_t)_l)x_{\\mathrm{in}} + (m_t)_l f_l(x_{\\mathrm{in}})$, which either passes input through or executes layer $f_l$; at inference the zero-mask layers are truly skipped to save MACs. The objective couples a denoising-plus-distillation loss for the UNet with a hypernetwork loss that adds a log-ratio sparsity term and a router-balance term, in an alternating bi-level update.","core_discovery":"The paper's central claim is that temporal asymmetry in diffusion can be turned into a structural asset: instead of one pruned model for all steps, ALTER learns ten pruned expert subnetworks of a shared UNet and a router that assigns each denoising timestep to an expert. The pruning masks and routing are produced by a hypernetwork with Gumbel-Sigmoid and Gumbel-Softmax straight-through sampling, and the forward pass simulates pruning with the residual update $x_{\\mathrm{out}} = (1 - m_l)x_{\\mathrm{in}} + m_l f_l(x_{\\mathrm{in}})$, which skips a layer when its mask is zero while keeping gradients flowing. An alternating optimization updates the UNet to perform well under the current expert structures and then updates the hypernetwork to find structures that preserve denoising performance, hit a target sparsity ratio, and keep router use balanced. The paper reports that this single-stage co-adaptation beats static pruning, fixed manual timestep intervals, and two-stage training, and that at 20 inference steps ALTER matches or beats the unpruned 50-step SDv2.1 on FID, CLIP, and CMMD on CC3M and MS-COCO.","pith_inferences":["The reported 3.64x speedup is a MAC-count ratio; actual runtime gains may be lower on real hardware because the router adds a per-step forward pass and skipped layers create irregular memory access patterns, so wall-clock latency should be measured before deployment decisions are made.","The same hypernetwork-plus-timestep-router recipe could in principle be applied to transformer-based diffusion backbones, where prunable \"layers\" are attention blocks, though the paper only demonstrates it on a UNet.","One testable extension is combining ALTER's layer skipping with feature caching: the two target different sources of redundancy and could compound, but the paper does not measure that combination.","Because routing depends only on the timestep embedding, the learned experts might transfer across samplers with different step schedules, but guidance scale and text-conditioning changes could shift what each expert should do; that transfer is not tested."],"forward_implications":["At 20 inference steps, ALTER uses 9.89T MACs versus 38.04T for the 50-step SDv2.1 baseline, a 3.64x reduction by the paper's MAC-based measure, with FID-5K of 25.25 versus 27.29.","Because the router keys on timestep embeddings, the same trained ALTER model can be run at 15, 20, or 25 steps without retraining or adjusting cache rates, unlike cache-based methods that need parameter tuning per schedule.","Timestep-wise routing uses every expert subnetwork somewhere along the denoising trajectory, so it avoids the capacity under-utilization of sample-wise dynamic pruning, where one prompt commits to one subnetwork for the whole trajectory.","The single-stage joint optimization is necessary: the paper's ablation shows the two-stage variant (hypernetwork first, then UNet fine-tuning) is worse on FID, CLIP, and CMMD on both datasets.","The paper's own appendix states that a final fine-tuning stage for the dynamically pruned UNet is still needed after the hypernetwork training ends, so the joint optimization covers the pruning and routing search rather than the complete training pipeline."],"supporting_citations":[{"why":"Supplies the pretrained Stable Diffusion v2.1 backbone that ALTER compresses and the 50-step quality baseline.","marker":"[4]"},{"why":"Provides the sample-wise prompt-routing MoE pruning baseline that ALTER extends by adding timestep-level routing.","marker":"[18]"},{"why":"Provides the static pruning baseline BK-SDM that ALTER compares against and improves on in quality and efficiency.","marker":"[50]"},{"why":"Provides the cache-based acceleration baseline DiP-GO used for the COCO 2017 comparison.","marker":"[30]"},{"why":"Supplies the Gumbel-Softmax technique used for differentiable sampling of routing and mask decisions.","marker":"[61]"},{"why":"Supplies the straight-through estimator that makes binary pruning decisions differentiable during training.","marker":"[70]"},{"why":"Provides the PNDM sampler used for generating evaluation images across all compared methods.","marker":"[22]"},{"why":"Supplies the LAION-Aesthetics training subset used to fine-tune ALTER.","marker":"[64]"}],"fun_headline_variants":["Co-trained pruning and routing cut diffusion compute to 26%","Temporal expert routing enables 3.64x faster diffusion at same quality","ALTER: 3.6x faster diffusion with 10 pruned temporal experts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 3.64x speedup is reported as a ratio of total multiply-accumulate counts rather than a measured wall-clock latency for the 50-step baseline, so the claim assumes MAC reductions translate directly into runtime savings without significant overhead from the router or from irregular layer skipping.","fun_headline_variants_meta":{"raw":{"variants":["Co-trained pruning and routing cut diffusion compute to 26%","Temporal expert routing enables 3.64x faster diffusion at same quality","ALTER: 3.6x faster diffusion with 10 pruned temporal experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1754,"prompt_tokens":1053,"completion_tokens":701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":638}},"tokens_in":669,"tokens_out":701,"duration_ms":7670,"temperature":1.0,"reasoning_tokens":638,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:22:09.262797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure end-to-end wall-clock latency for ALTER's 20-step model and the original 50-step SDv2.1 on the same GPU with batch size 1, including all routing and layer-skipping overhead, and compare the ratio to 3.64; a substantially smaller ratio would show the MAC-based speedup overstates real acceleration.","supporting_citations":[{"cited_title":"Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the sample-wise prompt-routing MoE pruning baseline that ALTER extends by adding timestep-level routing."},{"cited_title":"Pseudo numerical methods for diffusion models on manifolds","cited_arxiv_id":null,"evidence_quote":"Provides the PNDM sampler used for generating evaluation images across all compared methods."}],"review_version":1}