Pith. sign in

REVIEW 4 major objections 5 minor 40 references

The paper claims that multi-node Diffusion Transformer inference can skip most remote attention partitions by reusing cached composed attention states from previous denoising steps, cutting cross-node communication and attention compute whi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:40 UTC pith:VOSD2I6E

load-bearing objection DiTango is a genuine systems contribution — partition-level, contribution-aware reuse of composed attention states — with the main risk being an unproven proportionality in the error model; still, it deserves a serious referee. the 4 major comments →

arxiv 2607.15650 v1 pith:VOSD2I6E submitted 2026-07-17 cs.CV

DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

Yuyang Chen , Runxin Zhong , Zan Zong , Hengjie Li , Yuyang Jin , Jidong Zhai This is my paper
classification cs.CV
keywords Diffusion TransformersContext Parallelismattention state reusespatial localitymulti-node inferencevideo generationerror-bounded cachingparallel attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DiTango claims that in multi-node Diffusion Transformer inference, attention contributions from context-parallel sequence partitions are strongly spatially local: nearby partitions dominate the output, while distant partitions contribute two to three orders of magnitude less. The system exploits this by computing fresh attention states only for high-contribution, cheap-to-reach partitions and reusing cached, composed attention states from recent denoising steps for distant, expensive-to-reach partitions. An anchor-guided planner predicts reuse error with an online model and decides per group whether to compute or reuse under an error budget; a state-centric runtime executes the irregular pattern with high compute-communication overlap. On open video DiTs, the authors report up to 1.9x end-to-end and 3.2x attention speedups across 32 GPUs with generation quality comparable to full attention on most metrics. If correct, this turns the scalability-quality dilemma of distributed diffusion inference into a tunable compute-reuse trade-off.

Core claim

The central discovery is that attention contribution in Diffusion Transformer inference is highly unequal across context-parallel KV partitions and decays with spatial distance: excluding the nearest partition changes attention outputs by MSE around 1e-2, while excluding distant partitions changes them by only 1e-4. DiTango builds on this by defining per-partition attention states—partial outputs paired with log-sum-exp values—that compose associatively and commutatively. It computes fresh states only for partitions with high contribution and low communication cost, and reuses previously composed states for low-contribution remote partitions, bypassing both KV transmission and attention comp

What carries the argument

The load-bearing object is the attention state: for sequence partition i at denoising step t, AS_t(i) = (OUT_t(i), LSE_t(i)), where OUT_t(i) is the partial attention output and LSE_t(i) its log-sum-exp. Because attention states compose associatively and commutatively, DiTango can merge several remote partitions into one cached state and fold it into the final output later. The anchor-guided selection planner then decides which groups to compute versus reuse using an online error model: ASE(G_i,t) = sum over partitions of w(j,t)·δ(j,t,t_c), where w is the partition importance weight refreshed only at anchor steps, and δ is extrapolated from local-state drift via a scale ratio α(j) measured at

Load-bearing premise

The planner's error predictions assume that each remote partition's attention-state drift over a few denoising steps is a fixed multiple of the local partition's drift, with that multiple measured at anchor steps and treated as constant until the next anchor.

What would settle it

Run a video-DiT generation on a prompt with strong long-range coupling—for example, two identical objects at opposite ends of the frame—and measure the actual attention-state error of the farthest partition against the predicted α·(local drift) across early denoising steps. If the correlation falls well below the reported R²≈0.98, or if reusing distant partitions under the planner degrades LPIPS/PSNR or VBench by more than the reported margins, the proportionality premise is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If spatial locality of attention contribution holds for a given DiT model, DiTango's planner will reuse progressively more remote partitions, shifting the quality-latency trade-off toward lower latency with bounded, predictable error.
  • The attention-state representation is roughly half the size of a full KV cache, and composed states can be merged into coarser groups under memory pressure, so the strategy fits within current GPU memory for the evaluated long-video workloads.
  • The approach changes the scaling regime of multi-node DiT inference: instead of cross-node communication growing with the number of context partitions, only high-contribution partitions incur remote traffic in a typical denoising step.
  • In single-node or high-bandwidth settings, the paper's own limitation discussion indicates that the benefit narrows because communication is no longer the bottleneck, so the reported speedups are specific to multi-node environments.
  • Because the planner is online and error-threshold-driven, the same mechanism could be paired with other lossy acceleration techniques—step skipping, layer caching, or sparsity—without re-designing the parallel backend.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the partition-level spatial locality measured here for video DiTs may also hold for long-context image generation and for the prefill phase of long-context LLMs, making composed-state reuse a candidate strategy there; this is a testable extension, not a paper claim.
  • Beyond the paper: the anchor-step proportionality assumption could be made more robust by updating scale ratios α(j) per layer or per attention head, or by detecting when global-structure prompts break the monotone distance-decay pattern; doing so might extend quality guarantees to cases the current planner does not explicitly cover.
  • Beyond the paper: in heterogeneous clusters with asymmetric inter-node bandwidth, DiTango's planner could be extended to incorporate measured per-link bandwidth as an explicit cost term, rather than treating all cross-node traffic as uniformly expensive.
  • Beyond the paper: the error model's validity is checked only at anchor steps; a hybrid that occasionally spot-checks a reused remote state against a freshly computed one would give a cheap, continuous falsification signal for the proportionality assumption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. DiTango is a distributed inference framework for Diffusion Transformers (DiTs) that exploits spatial locality in context-parallel attention. The paper partitions the sequence into groups, computes all attention states at anchor steps, and then reuses cached composed attention states for low-contribution remote partitions while computing fresh states for local/high-contribution partitions. The technical core is an error-propagation model (Eqs. 4-6), an online error estimator (Eqs. 7-10), and a threshold-based compute/reuse selection rule (Eq. 13). Evaluations on Wan2.1 and HunyuanVideo report up to 1.9x end-to-end and 3.2x attention speedups over baselines on up to 32 H20 GPUs, with quality measured by VBench, PSNR, SSIM, and LPIPS. The paper also describes a state-centric runtime with group-wise KV transfer, ring-based attention composition, and memory-aware state merging.

Significance. If the quality-preservation claim held, DiTango would be a practically valuable contribution to multi-node DiT inference, where communication overhead currently limits scaling. The identification of distance-decaying partition contribution, the use of composable attention states, and the topology-aware runtime design are plausible and well aligned with cluster hierarchies. The paper also provides an open-source implementation and detailed runtime breakdowns. However, the load-bearing link between the heuristic error predictor and the asserted quality guarantee is not yet adequately supported. The central derivation drops an upper-bound term without proving that the result is still conservative, and the key proportionality assumption in Eq. (10) is asserted rather than validated at the granularity where decisions are made. The evaluation also lacks error bars and uses an in-sample adaptive threshold. The system contribution is real, but the evidence for 'preserving generation quality' is currently conditional.

major comments (4)
  1. [§4.1.2, Eq. (10)] The reuse decision in Eq. (13) is only as good as the error estimate in Eq. (12), which in turn rests on Eq. (10): delta(i,t,t_a) = alpha(i) * ||AS_t(i_loc) - AS_{t_a}(i_loc)||_2. This assumes remote partition drift is a fixed multiple of local drift, with alpha measured only at anchor steps. The text states 'Our evaluation guarantees...' but provides no derivation and no direct validation of this proportionality at the partition/timestep level where the compute/reuse decision is made. The aggregate R^2=0.98 in Fig. 14 is not informative near the decision threshold: high global correlation can coexist with many mis-ordered partitions whose true error exceeds epsilon while predicted error does not. The risk is concrete in early denoising or for prompts with strong global structure, where remote states may drift independently of the local diagonal partition. Please provide per-partition, p
  2. [§4.1.1, Eqs. (4)-(6)] Eq. (5) is an upper bound, but Eq. (6) is obtained by dropping the second-order term with the informal justification that w_i w_j and |delta LSE_i - delta LSE_j| are small. This converts a rigorous bound into an approximation, and the approximation is not guaranteed to be conservative. If the second-order term is not negligible at some layers or timesteps, the accumulated Attention State Error is underestimated and the planner will over-reuse. Since the entire quality-preservation claim depends on bounding ASE, this step must be either proven under stated assumptions (e.g., a Lipschitz condition on LSE errors or a bound on the cross-term) or empirically quantified layer-by-layer. The current text does neither.
  3. [§4.2.4, Algorithm 1] The threshold epsilon is described as an 'error threshold', but in Algorithm 1 it is updated as a percentile of anchor correction errors to match a target cache/compute ratio R. Thus the 'error budget' is not an independent quality target; it is a compute target calibrated on the evaluation workloads. Moreover, w(i,t) and alpha(i) are measured from full attention at anchor steps on the same data used for evaluation. The quality/compute trade-off in Fig. 13 is therefore in-sample. Please report held-out prompt performance, sensitivity of quality to R and epsilon, and confidence intervals over seeds/videos. Without this, the claim 'maintaining generation quality' is not clearly distinguishable from fitting the operating point.
  4. [§6.2-§6.4, Table 2 and Fig. 11] All latency and quality numbers are point estimates. Figure 11 bars omit run-to-run variation, and Table 2 reports single PSNR/SSIM/LPIPS/VBench values. Given the many free parameters (epsilon, epsilon_local, R, group size g, alpha(i), w(i,t)), the reported 1.9x/3.2x speedups and the Pareto frontier in Fig. 13 could reflect favorable settings. In addition, Table 2's 'superior generation quality' claim is overstated: on Wan-14B, DiTango's VBench score is 80.79 versus 81.53 for the original lossless model, so it does not maintain original-model quality; it only beats the lossy baselines. The comparison to lossless baselines (TP, CP, xDiT) also omits PSNR/SSIM/LPIPS because those methods reproduce the original output, making the 'quality comparable' claim less direct. Please add error bars, multiple seeds, and a matched-compute comparison that reports all metrics for all methods.
minor comments (5)
  1. [Throughout] Several typos and OCR/extraction artifacts obscure the text: 'attention stare' in §4.2.1, 'manipuation' in §5.1.2, '30,GB' in §6.5, 'SGL-Diffusion' vs. 'SGLD' inconsistency, and 'glyph1197' symbols in Figures 5 and 16. Please proofread and replace these with the intended words and symbols.
  2. [§6.2] The abstract claims 'near-linear scaling', but §6.2 states that Wan-1.3B deteriorates beyond 24 GPUs. Qualify the scaling claim to specify the models/configurations for which it holds.
  3. [§6.4, Table 2] The speedup values in Table 2 compare 32-GPU DiTango to a single-GPU baseline, whereas the abstract and Fig. 11 emphasize speedups over baselines. Clarify which speedup is being reported in each place, and make the baseline definitions consistent.
  4. [§2.2.3] The memory calculation for KV caching is useful, but the formula '2×L×H×D×N×C' is dimensionally confusing (L, H, D, N, C are not all defined as counts in the same units). Define each symbol and state the bit-width assumption explicitly.
  5. [§1, footnote 1] The footnote states that this is a preprint 'accepted for publication at HPDC 26'. If this is a preprint, the acceptance claim should be verifiable; if the paper is under review, the wording should be adjusted. This does not affect the technical assessment.

Circularity Check

0 steps flagged

No significant circularity: the planner's error model is an online extrapolation with independent end-to-end quality and speedup validation, not a reduction to fitted parameters or self-citations.

full rationale

DiTango's derivation is self-contained and does not reduce to its inputs. The error predictor in Eq. 10, δ(i,t,t_a)=α(i)·||AS_t(i_loc)−AS_{t_a}(i_loc)||_2, uses α(i) measured once at an anchor step and then extrapolates remote attention-state drift from the always-computed local state. This is an extrapolation, not a fit to the quality metric being claimed: the predicted quantity is a state error, while the central claims are end-to-end latency and generation quality, which are measured against external baselines (TP, CP, xDiT, VideoSys, SGLD) and external benchmarks (VBench, PSNR/SSIM/LPIPS). The weights w(i,t) and the online threshold ε add calibration, but they do not make the speedup or the quality result true by construction; they are inputs to a planner whose output is then empirically evaluated. The statement 'Our evaluation guarantees that remote states evolve proportionally to the local state' is an unproven empirical proportionality assumption, not a circular step; if it breaks, the planner's decisions degrade, which is a correctness risk rather than a self-referential derivation. The only self-citation in the paper ([3], Jano) appears in a general related-work list about caching-based methods and is not load-bearing. No uniqueness theorem is imported from the authors, and no known result is merely renamed. Overall the central contribution is validated against independent benchmarks and baselines, so the circularity score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on a heuristic error model (proportional remote drift), a neglected second-order error term, an empirically observed spatial locality, and a threshold tuned to a target compute ratio. These are not independently verified beyond the paper's own measurements.

free parameters (6)
  • Error threshold ε (online adaptive) = Dynamic percentile of anchor correction errors
    Section 4.2.4: ε is adapted online 'as a percentile of the global anchor correction errors to match the target acceleration ratio'; the compute/reuse boundary is tuned to a target rather than derived.
  • Local threshold ε_local = Not stated
    Algorithm 1 input; triggers an anchor when local state drift exceeds it. No value or tuning procedure is given.
  • Cache/compute ratio R (target acceleration ratio) = Not stated; swept in Fig. 13
    Algorithm 1 input controlling how aggressive reuse is; the reported quality-speedup Pareto curve is a sweep over R.
  • Group size g = 4/8/16 explored
    Section 5.1.2 and Figure 15; group granularity changes communication patterns and is a design choice.
  • Partition scale ratio α(i) = ||AS_t_a(i)|| / ||AS_t_a(i_loc)|| at anchor
    Eq 9; measured from full attention at anchors and used to extrapolate remote error from local drift (Eq 10).
  • Partition importance weights w(i,t) = Softmax of LSE over partitions at anchor
    Eq 8; updated at anchor steps and assumed stable for τ≤5–10 steps.
axioms (5)
  • standard math Attention states are composable and associative (online-softmax composition).
    Eq 1-2; standard result from FlashAttention/FlashInfer, used throughout the design.
  • ad hoc to paper The second-order error term in Eq 5 is negligible.
    Section 4.1.1: 'safely ignored' because w_i w_j is small or |δLSE_i − δLSE_j| is small; no rigorous bound is given, and Eq 6 then treats an upper bound as an approximation.
  • ad hoc to paper Remote attention states drift proportionally to the local state.
    Eq 10: δ(i,t,t_a)=α(i)·||AS_t(i_loc)−AS_{t_a}(i_loc)||; the text claims 'our evaluation guarantees' proportionality but gives no derivation.
  • domain assumption LSE distributions remain stable over τ≤5–10 steps.
    Eq 8 and Figure 8; empirically observed, with anchors updating the weights. If this fails, weight staleness biases the error model.
  • domain assumption Attention contribution decays monotonically with partition distance across models and prompts.
    Figures 2 and 5; the entire selective-reuse strategy depends on this spatial locality persisting.

pith-pipeline@v1.3.0-alltime-deepseek · 20058 in / 14134 out tokens · 125308 ms · 2026-08-01T22:40:22.175439+00:00 · methodology

0 comments
read the original abstract

Recent advances in AI-generated content have driven widespread adoption of Diffusion Transformers (DiTs) for high-resolution, long-duration content generation. While parallelization techniques accelerate diffusion inference, they face significant scalability challenges due to excessive communication overhead in multi-node environments. We observe that sequence partitions in Context Parallelism (CP) exhibit distinct heterogeneity: spatially proximate partitions contribute more significantly to attention computation results. By mapping this heterogeneous pattern to hierarchical communication topology, we can access high-contribution partitions with reduced communication cost. This insight motivates our novel selective attention state mechanism that strategically balances partial attention computation and historical result reuse across denoising steps. We present DiTango, an efficient parallel framework for DiT generation. DiTango features an anchor-guided state selection planner that optimizes computation-reuse decisions for each partition, complemented by a runtime that orchestrates efficient state-centric operations. This design achieves superior system efficiency while preserving generation quality. Experimental evaluation on popular diffusion models demonstrates that DiTango achieves up to 1.9x end-to-end and 3.2x attention speedup with near-linear scaling in multi-node settings, while maintaining generation quality comparable to state-of-the-art approaches.

Figures

Figures reproduced from arXiv: 2607.15650 by Hengjie Li, Jidong Zhai, Runxin Zhong, Yuyang Chen, Yuyang Jin, Zan Zong.

Figure 1
Figure 1. Figure 1: Parallel inference performance breakdown for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Attention computation contribution is highly un [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison between full-sequence attention and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Parallel strategies for DiT inference. 2.1.3 Diffusion Modules Feature Reuse. Feature reuse has emerged as a popular optimization technique in diffusion models, leveraging the observation that consecutive denoising steps often produce similar intermediate features. This similarity enables training-free but lossy acceleration through strategic caching and reuse of fea￾tures from previous steps, effectively … view at source ↗
Figure 5
Figure 5. Figure 5: Spatial locality of CP attention in typical cluster: [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Selective attention state reuse mechanism (CP=4, [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Partition importance 𝑤(𝑖, 𝑡) with different prompts and layers. 4.1.2 Online Attention State Error Model. Based on Equation 6, we model the Attention State Error (ASE) of partition 𝑖’s cached state AS𝑡𝑐 (𝑖) at current timestep 𝑡 as: ASE(𝑖, 𝑡, 𝑡𝑐 ) = 𝑤(𝑖, 𝑡) · 𝛿 (𝑖, 𝑡, 𝑡𝑐 ) (7) where 𝑡𝑐 is the cache timestep, 𝑤(𝑖, 𝑡) captures partition impor￾tance, and 𝛿 (𝑖, 𝑡, 𝑡𝑐 ) models temporal error growth over cache a… view at source ↗
Figure 9
Figure 9. Figure 9: Runtime orchestration timeline. The computation and communication streams execute asynchronously, overlapping [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: State-centric communication operations. (a) Cross [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance evaluation of DiTango. Bars for baselines of too long running time are truncated, and their execution times are marked on the bars. The numbers above DiTango’s bars show speedups over the best baseline in multi-node inference [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Core attention computational efficiency measured [PITH_FULL_IMAGE:figures/full_fig_p010_12.png] view at source ↗
Figure 16
Figure 16. Figure 16: Ablation study showing the impact of different [PITH_FULL_IMAGE:figures/full_fig_p011_16.png] view at source ↗
Figure 15
Figure 15. Figure 15: Runtime breakdown analysis showing the propor [PITH_FULL_IMAGE:figures/full_fig_p011_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 24 linked inside Pith

  1. [1]

    Tim Brooks, Bill Peebles, Connor Holmes, Amos Storkey, Alexei A. Efros, Andreas Terzis, Abhinav Gupta, Devi Parikh, Douwe Kiela, Gabriel Synnaeve, Hannaneh Hajishirzi, Ilya Sutskever, James Zung, Joelle Pineau, Luke Metz, Mira Murati, Pranav Shyam, Rohun Kulkarni, Ruth Fong, Vedant Misra, Yufei Guo, Adrià Recasens, Alexander Papiez, Alexandre Lebrun, Arth...

    2024

  2. [2]

    Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos- Savvas Bouganis, Yiren Zhao, and Tao Chen. 2024. Delta-DiT: A Training-Free Ac- celeration Method Tailored for Diffusion Transformers. arXiv:2406.01125 [cs.CV] https://arxiv.org/abs/2406.01125

    Pith/arXiv arXiv 2024

  3. [3]

    Yuyang Chen, Linqian Zeng, Yijin ZHou, Hengjie Li, and Jidong Zhai. 2026. Jano: Adaptive Diffusion Generation with Early-stage Convergence Awareness. arXiv:2603.00519 [cs.CV] https://arxiv.org/abs/2603.00519

    arXiv 2026

  4. [4]

    Zigeng Chen, Xinyin Ma, Gongfan Fang, Zhenxiong Tan, and Xinchao Wang. 2024. AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising. InAdvances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mac...

    2024

  5. [5]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  6. [6]

    vipshop.com DefTruth. 2025. cache-dit: A PyTorch-native and Flexible In- ference Engine with Hybrid Cache Acceleration and Parallelism for DiTs. https://github.com/vipshop/cache-dit.git Open-source software available at https://github.com/vipshop/cache-dit.git

    2025

  7. [7]

    Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. CoRRabs/2411.01738 (2024). doi:10.48550/ARXIV.2411.01738 arXiv:2411.01738

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.01738 2024

  8. [8]

    Jiarui Fang and Shangchun Zhao. 2024. A Unified Sequence Parallelism Approach for Long Context Generative AI.arXiv preprint arXiv:2405.07719(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Jonathan Ho and Tim Salimans. 2022. Classifier-Free Diffusion Guidance.CoRR abs/2207.12598 (2022). doi:10.48550/ARXIV.2207.12598 arXiv:2207.12598

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.12598 2022

  10. [10]

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2024

  11. [11]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuai- wen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Trans- former Models.CoRRabs/2309.14509 (2023). doi:10.48550/ARXIV.2309.14509 arXiv:2309.14509

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.14509 2023

  12. [12]

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.03603 2024

  13. [13]

    Kuaishou Technology. 2025. Kling - Community for short video & livestream. https://kling.kuaishou.com/en Accessed: 2025-03-07

    2025

  14. [14]

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. 2024. DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, W A, USA, June 16-22, 2024. IEEE, 7183–7193. doi:10.1109/CVPR52733.2024.00686

    arXiv 2024

  15. [15]

    Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. 2024. Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model.CoRRabs/2411.19108 (2024). doi:10.48550/ARXIV.2411.19108 arXiv:2411.19108 12 DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.19108 2024

  16. [16]

    Hao Liu and Pieter Abbeel. 2023. Blockwise Parallel Transformers for Large Context Models. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levi...

    2023

  17. [17]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context.CoRRabs/2310.01889 (2023). doi:10. 48550/ARXIV.2310.01889 arXiv:2310.01889

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01889 2023

  18. [18]

    Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. 2025. From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers. arXiv preprint arXiv:2503.06923(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Qiang Liu. 2022. Rectified Flow: A Marginal Preserving Approach to Optimal Transport. arXiv:2209.14577 [stat.ML] https://arxiv.org/abs/2209.14577

    Pith/arXiv arXiv 2022

  20. [20]

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang...

    Pith/arXiv arXiv 2025

  21. [21]

    Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. 2024. Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching. arXiv:2406.01733 [cs.LG]

    Pith/arXiv arXiv 2024

  22. [22]

    Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 2022. On Distillation of Guided Diffusion Mod- els.arXiv e-prints, Article arXiv:2210.03142 (Oct. 2022), arXiv:2210.03142 pages. doi:10.48550/arXiv.2210.03142 arXiv:2210.03142 [cs.CV]

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03142 2022

  23. [23]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transform- ers. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 4172–4182. doi:10.1109/ICCV51070.2023.00387

    arXiv 2023

  24. [24]

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. 2023. Adversarial Diffusion Distillation. arXiv:2311.17042 [cs.CV] https://arxiv.org/ abs/2311.17042

    Pith/arXiv arXiv 2023

  25. [25]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parame- ter Language Models Using Model Parallelism.CoRRabs/1909.08053 (2019). arXiv:1909.08053 http://arxiv.org/abs/1909.08053

    Pith/arXiv arXiv 2019

  26. [26]

    Genmo Team. 2024. Mochi 1. https://github.com/genmoai/models

    2024

  27. [27]

    VideoSys Team. 2024. VideoSys: An Easy and Efficient System for Video Genera- tion. https://github.com/NUS-HPC-AI-Lab/VideoSys

    2024

  28. [28]

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, T...

    Pith/arXiv arXiv 2025

  29. [29]

    Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. 2024. PipeFusion: Dis- placed Patch Pipeline Parallelism for Inference of Diffusion Transformer Models. CoRRabs/2405.14430 (2024). doi:10.48550/ARXIV.2405.14430 arXiv:2405.14430

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.14430 2024

  30. [30]

    Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. 2025. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity.CoRRabs/2502.01776 (2025). doi:10. 48550/ARXIV.2502.01776 arXiv:2502.01776

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.01776 2025

  31. [31]

    Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation.arXiv preprint arXiv:2505.18875(2025)

    Pith/arXiv arXiv 2025

  32. [32]

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. 2024. CogVideoX: Text-to-Video Diffusion Models with An Ex- pert Transformer.CoRRabs/2408.06072 (2024). doi:10.48550/ARXIV...

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.06072 2024

  33. [33]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving.CoRRabs/2501.01005 (2025). doi:10.48550/ARXIV.2501.01005 arXiv:2501.01005

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.01005 2025

  34. [34]

    Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. 2024. DiTFastAttn: Attention Compression for Diffusion Transformer Models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/ forum?id=51HQpkQy3t

    2024

  35. [35]

    Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen

  36. [36]

    Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. 2025. SageAt- tention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration. In International Conference on Learning Representations (ICLR)

    2025

  37. [37]

    Xuanlei Zhao, Shenggan Cheng, Zangwei Zheng, Zheming Yang, Ziming Liu, and Yang You. 2024. DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers.CoRRabs/2403.10266 (2024). doi:10.48550/ARXIV.2403.10266 arXiv:2403.10266

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.10266 2024

  38. [38]

    Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. 2024. Real-Time Video Generation with Pyramid Attention Broadcast. arXiv:2408.12588 [cs.CV] https: //arxiv.org/abs/2408.12588

    Pith/arXiv arXiv 2024

  39. [39]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104 [cs.AI] https://arxiv.org/abs/2312. 07104 13

    Pith/arXiv arXiv 2024

  40. [2025]

    InInternational Conference on Machine Learning (ICML)

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. InInternational Conference on Machine Learning (ICML)