{"id":"46b55360-b2b5-4764-8d1e-2fa069a10201","arxiv_id":"2607.06631","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Joint few-step distillation and step-specific structural pruning turns a video diffusion model into a compact Mixture-of-Models that cuts 24% extra FLOPs per step and reaches 30× speedup on Wan-14B.","lead":"Video diffusion models can be sped up by jointly learning which parts of the network to keep at each of a few denoising steps, not just by cutting the number of steps. On a 14B model this removes another 24% of compute per step after 4-step distillation, for roughly 30× overall speedup with quality that stays competitive on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The 1.2× wall-clock claim on Wan-14B rests on a single proprietary-hardware latency number without variance or FLOPs-to-latency validation.","rationale":"The paper’s joint-optimization story is solid: ablations (Fig. 3, Table 3) show that without progressive reverse-order curriculum and output rollout the sparsity wall sits at ~2%, while the full method reaches ~20% sparsity with competitive VBench. Decoupled prune-then-distill / distill-then-prune baselines collapse or give negligible extra speedup, supporting the necessity of co-optimization. Architectural findings (U-shaped allocation, deep-layer semantic freezing) are consistent and interesting. The reader correctly flags the hand-chosen 5% FLOPs stage-transition and proprietary hardware as reasons for CONDITIONAL rather than ACCEPT. I agree the verdict stays CONDITIONAL, but I locate the single most load-bearing soft spot one step later: the translation of theoretical FLOPs into the headline 1.2× wall-clock number. That number is the only quantity that turns “24% FLOPs removed” into a practical acceleration claim, yet it is supported by a solitary unreproduced measurement on a specialized engine. The curriculum concern is real for training stability, but even a perfectly stable mask set would not guarantee the wall-clock figure if the inference engine is doing heavy lifting. Hence partial agreement: same overall verdict, different primary soft spot. No evidence of internal inconsistency or fabrication; the claim simply needs portable latency confirmation.","tokens_in":19212,"tokens_out":798,"duration_ms":9217,"concrete_test":"Re-measure end-to-end DiT latency of the exported MoM versus the dense 4-step student on at least two commodity GPUs (e.g., A100 and H100) using a standard dense kernel path (no custom gather engine). If the wall-clock ratio falls below 1.10× while FLOPs remain ~76%, the 1.2× claim is hardware/engine-specific rather than a general consequence of the learned sparsity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim packages three numbers: 24% per-step FLOPs reduction, +1.2× wall-clock over 4-step distillation, and 30× end-to-end vs the 50-step teacher (Table 1, Wan-14B row). The FLOPs figure is architectural and internally consistent with the reported ~76% retention. The wall-clock figure, however, is a single measurement (40.72 s) on one Kunlun P800 XPU under an unreproduced specialized inference engine (§4.3) that gathers step-specific dense subsets from a super-model via index tables. No multi-run variance, no FLOPs-vs-latency scaling curve, and no comparison against a pure dense 4-step baseline under identical engine conditions are supplied. On the smaller Wan-1.3B the same method yields only +1.09×, already hinting that memory-bandwidth or gather overhead can erode theoretical sparsity. If the engine’s gather/index overhead or XPU-specific kernel fusion is responsible for a non-negligible fraction of the measured 1.2×, the practical acceleration claim weakens even while the FLOPs claim remains intact. The progressive curriculum and rollout (the reader’s weakest assumption) are necessary for training stability, but they are secondary to whether the exported MoM actually delivers the advertised wall-clock gain once the specialized engine is removed or ported.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a post-training framework that jointly optimizes few-step distribution-matching distillation (modified D-DMD + GAN) and step-aware tri-level structural pruning (block / head / FFN-channel gates) for video diffusion models. Continuous masks are optimized via STE under a sparsity penalty, producing a step-specific Mixture-of-Models that is exported as dense per-step sub-networks. Training instability is addressed by a reverse-order progressive curriculum (sparsity applied only to the newly introduced step, advanced when that step’s FLOPs drop 5 %) plus an output-rollout mechanism that trains the fake score network on multi-step final outputs rather than single-step clean predictions. A specialized inference engine gathers active parameters from a super-model via index tables. On Wan-14B the method reports 24 % per-step FLOPs reduction on top of 4-step distillation, 1.2× additional wall-clock speedup (40.72 s vs. 1222 s teacher), and competitive VBench scores; ablations and decoupled prune/distill baselines support the necessity of joint optimization and the progressive+rollout recipe.","tokens_in":19708,"tokens_out":1120,"duration_ms":10952,"significance":"If the joint-optimization claim holds, the work supplies a cleanly orthogonal acceleration axis—per-step parametric redundancy along the hidden-channel dimension—that is complementary to sparse attention, quantization and system-level kernels already used by frameworks such as LightX2V and TurboDiffusion. The architectural findings (U-shaped capacity allocation, deeper-layer semantic freezing) are interpretable and potentially reusable. Strengths include a full set of controlled two-stage baselines, quantitative ablations that demonstrate a hard sparsity wall without the proposed curriculum, and an explicit specialized engine that converts theoretical sparsity into measured latency. The contribution is therefore of practical interest for efficient video generation, provided the wall-clock numbers generalize beyond the proprietary XPU and engine.","major_comments":[{"comment":"Table 1 (Wan-14B row) and §5.4 package the central claim as 24 % FLOPs + 1.2× wall-clock + 30× end-to-end. The FLOPs figure is architectural and consistent with the reported retention; the wall-clock figure (40.72 s) is a single measurement on one Kunlun P800 under the unreproduced specialized engine of §4.3. No multi-run variance, no FLOPs-to-latency scaling curve, and no head-to-head comparison of the identical dense 4-step model under the same engine are supplied. On Wan-1.3B the same method yields only +1.09×, already suggesting gather/index or bandwidth overhead can erode theoretical sparsity. Without these controls the practical acceleration claim remains under-supported even while the FLOPs claim stands.","section":null},{"comment":"§4.2 and the ablation in §5.5 / Table 3 establish that the reverse-order curriculum and output-rollout are necessary to avoid collapse, yet both mechanisms introduce free parameters (5 % FLOPs stage-transition threshold, adaptive λ_t via gradient-norm EMA, FFN chunk size C=8, L2 reordering). The manuscript does not report sensitivity of the final masks or VBench scores to these choices, nor does it show that the learned U-shaped policy (Fig. 4) is stable across random seeds or modest hyper-parameter perturbations. Given that the reader’s weakest assumption is precisely the sufficiency of this recipe for genuine step-adaptivity, a modest sensitivity study is load-bearing for the claim that the masks are not training-set artifacts.","section":null}],"minor_comments":[{"comment":"Fig. 1(c) and §4.3 describe the super-model + index-table engine but give no memory-footprint numbers or comparison against naïve storage of T separate dense models; a short table would clarify the claimed memory advantage.","section":null},{"comment":"The Dynamic Degree of 80.56 on Wan-1.3B exceeds the teacher (65.19). Appendix D attributes this to GAN + real-video data and mask stochasticity, yet the main text should briefly flag the high variance of this metric and the supporting Motion Smoothness score so readers do not misread it as artifact-driven motion.","section":null},{"comment":"Eq. (4) uses a ReLU on the average retention across all steps; the interaction of this global term with the progressive schedule (sparsity applied only to t_new) is not immediately transparent and would benefit from a one-sentence clarification.","section":null},{"comment":"Several related-work citations (DyDiT, PhasedDMD, LightX2V) appear only as arXiv preprints; once camera-ready versions exist they should be updated.","section":null}],"recommendation":"major_revision","confidential_remarks":"The wall-clock numbers rest on proprietary Kunlun hardware and an unreproduced engine; if the journal expects fully reproducible latency claims, the authors may need to supply CUDA/PyTorch reference numbers or open-source the gather kernels. Otherwise the contribution remains solid once the two major points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit is that they actually make dynamic structural pruning coexist with few-step DMD-style distillation. Prior DyDiT-style work collapses or becomes useless under step reduction; the decoupled prune-then-distill and distill-then-prune baselines in Table 1 confirm that. Their joint objective plus reverse progressive curriculum (sparsity only on the newly introduced step, advance at 5% FLOPs drop) and output-rollout fake-score training is what keeps the co-optimization from arbitrary over-pruning. Ablations in Fig. 3 / Table 3 show the sparsity wall at ~2% without those pieces, versus ~20% with them. That is the real contribution.\n\nOn Wan-14B they report 24% per-step FLOPs removed on top of 4-step distillation, 1.2× wall-clock, 30× vs the 50-step teacher, with VBench still competitive. The FLOPs number and the U-shaped retention pattern (high capacity at t1/t4, aggressive prune of deep attention at intermediate steps) are clean and match the coarse-to-fine story. Decoupled baselines fail for the reasons they claim. The specialized super-model + index-table engine is a practical necessity for turning gates into actual speed.\n\nSoft spots are ordinary empirical ones, not load-bearing cracks. The 1.2× wall-clock is a single number on one Kunlun P800 under their unreproduced engine; no variance, no FLOPs-to-latency curve, and the 1.3B model only gets 1.09×, so gather overhead or XPU-specific fusion could be eating part of the gain. Free parameters (η_k, 5% threshold, adaptive λ, chunk size 8) are hand-chosen. No code. Dynamic Degree above the teacher is high-variance and partly explained by their GAN + data, but Motion Smoothness stays high so it is not pure flicker. None of this sinks the central claim that joint optimization unlocks an orthogonal axis.\n\nThis is for people shipping efficient video generators who already run 4-step distillation and want another free 15–25% without rewriting attention or quantizing. It deserves a serious referee; the engineering is careful enough and the negative baselines are honest. I would engage with it and cite the joint-training recipe if I am working on post-training acceleration.","headline":"Solid joint recipe that finally makes step-aware pruning work with 4-step distillation; the 24% FLOPs cut is real, the 1.2× wall-clock is thinner evidence.","tokens_in":20274,"tokens_out":589,"would_cite":true,"duration_ms":6421,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Jointly sparsifying structure while distilling few-step video diffusion yields a step-specific Mixture-of-Models that removes 24% of per-step FLOPs and reaches 30× speedup on Wan-14B with competitive quality.","keywords":["video generation speedup","dynamic network","efficient diffusion","few-step distillation","structural pruning","Mixture-of-Models","video diffusion transformer"],"falsifier":"Train the same joint objective on Wan-14B without the progressive schedule or the output-rollout, push average FLOPs to the reported 76% retention, and check whether VBench Imaging Quality and Dynamic Degree still match the dense 4-step distill baseline under the paper’s own evaluation protocol; a collapse would falsify the claim that those two devices make the co-optimization stable.","tokens_in":20114,"feed_emoji":"🎬","tokens_out":983,"duration_ms":23025,"temperature":0.7,"pith_summary":"Video diffusion models produce high-quality video but remain expensive because even after few-step distillation they still run the same heavy network at every remaining noise level. This paper shows that the computational demand is not uniform: early and late steps need more capacity than the middle, and that this redundancy can be exploited by learning a separate sparse architecture for each of the few remaining steps while the distillation itself is running. The result is a compact step-aware Mixture-of-Models. On a 14B model the joint method removes another 24% of FLOPs per step on top of 4-step distillation, adding a measured 1.2× wall-clock gain and a total 30× speedup over the 50-step teacher while keeping VBench scores competitive. Two training devices—a reverse-order progressive curriculum and an output-rollout objective for the fake score network—are what keep the joint search from collapsing where separate prune-then-distill or distill-then-prune pipelines fail.","feed_headline":"Few-step video models drop another 24% FLOPs per step","feed_subtitle":"Joint pruning and distillation turns a 14B teacher into a step-specific mixture without quality collapse","key_machinery":"The step-aware Mixture-of-Models produced by jointly optimizing a modified distribution-matching distillation loss with a sparsity penalty on learnable structural masks, kept stable by a reverse-order progressive curriculum and an output-rollout fake-score objective.","core_discovery":"Few-step distillation and dynamic structural sparsification can be solved as one joint optimization. Continuous structural masks are learned for blocks, attention heads and FFN channels at each of the four discrete timesteps; hard gates derived from those masks turn a pre-trained video DiT into a step-specific Mixture-of-Models. Stabilized by reverse-order progressive sparsification and by rolling the student all the way to the final clean frame for the fake-score loss, the co-optimization discovers genuine temporal redundancy and yields a practical further acceleration without quality collapse.","pith_inferences":["The same reverse-order curriculum may stabilize other joint architecture-search + distillation problems beyond video diffusion.","Semantic freezing of deep layers after the first step suggests later few-step stages could be replaced by lighter texture-only modules without full retraining.","If the U-shape is universal across DiT video models, FLOPs budgets could be pre-allocated per step before any joint training begins.","Exporting dense step-specific sub-networks makes the method immediately compatible with existing static-model deployment stacks on mobile or edge hardware."],"forward_implications":["Per-step parametric redundancy is an orthogonal acceleration axis that can be stacked with sparse attention, quantization and system-level engines.","The learned U-shaped capacity pattern (high at first and last steps, low in the middle) can guide the design of future hand-crafted few-step architectures.","Decoupled prune-then-distill or distill-then-prune pipelines are strictly weaker than joint optimization for this setting.","Once masks are fixed, a single super-model plus lightweight index tables is sufficient for dense inference without reloading separate models."],"fun_headline_variants":["Joint few-step distillation and sparsity cut 24% more FLOPs per video step","Step-specific MoM prunes Wan-14B by 24% FLOPs atop 4-step distillation","Dynamic block sparsification co-trained with 4-step video distillation","Four discrete masks turn a video DiT into a step-specific mixture of models","Progressive reverse sparsification yields 30x video speedup without collapse"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That applying the sparsity penalty only to the newly introduced step in reverse order, together with rolling the student forward under detached gradients to the final clean frame, is enough to stop the joint search from collapsing into arbitrary over-pruning.","fun_headline_variants_meta":{"raw":{"variants":["Joint few-step distillation and sparsity cut 24% more FLOPs per video step","Step-specific MoM prunes Wan-14B by 24% FLOPs atop 4-step distillation","Dynamic block sparsification co-trained with 4-step video distillation","Four discrete masks turn a video DiT into a step-specific mixture of models","Progressive reverse sparsification yields 30x video speedup without collapse"]},"model":"grok-4.5","effort":"low","cost_usd":0.003794,"raw_usage":{"total_tokens":1247,"prompt_tokens":826,"num_sources_used":0,"completion_tokens":93,"cost_in_usd_ticks":37940000,"prompt_tokens_details":{"text_tokens":826,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":328,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":826,"tokens_out":93,"duration_ms":4112,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T00:58:35.058937+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same joint objective on Wan-14B without the progressive schedule or the output-rollout, push average FLOPs to the reported 76% retention, and check whether VBench Imaging Quality and Dynamic Degree still match the dense 4-step distill baseline under the paper’s own evaluation protocol; a collapse would falsify the claim that those two devices make the co-optimization stable.","supporting_citations":[],"review_version":1}