Pith. sign in

REVIEW 2 major objections 6 minor 57 references

RotateAttention makes INT4 FlashAttention work for 3D-RoPE video DiTs by fixing RoPE-induced Q/K outliers and reclaiming the unused half of the 4-bit range for the attention matrix P.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 16:52 UTC pith:G5XY76XM

load-bearing objection Solid systems paper: RoPE-aware orthogonal rotations plus full-range INT4 P give real 1.5–1.7× end-to-end gains on Wan2.2/HunyuanVideo; quality claim is schedule-dependent, not pure INT4. the 2 major comments →

arxiv 2607.02584 v2 pith:G5XY76XM submitted 2026-07-01 cs.CV

RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

Yaofu Liu , Wanli Lan , Jinxi Li , Binhang Yuan , Harry Yang This is my paper
classification cs.CV
keywords video generationquantized attention3D RoPEINT4 FlashAttentionDiToutlier rotationmixed-precision
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video DiTs with 3D RoPE spend most of their compute on attention over tens of thousands of tokens. Quantizing FlashAttention to INT4 would help, but two obstacles remain: common rotation tricks that tame Q and K outliers do not play well with RoPE, and the non-negative probability matrix P wastes half of every 4-bit code under symmetric quantization. The authors show that 3D RoPE partitions feature dimensions into frame, height and width segments and produces characteristic, symmetric outlier patterns inside those segments. They exploit the patterns with two lightweight, RoPE-compatible orthogonal rotations (one mergeable into RoPE itself, one almost free) and replace symmetric P quantization by a fixed-scale affine map that uses the full signed INT4 interval. With a selective FP16 fallback on the first and last few denoising steps and DiT blocks, the resulting mixed-precision kernel recovers essentially full-precision video quality while delivering up to 1.68 imes end-to-end and 2.2 imes kernel speedups.

Core claim

The outlier structure of Q and K in 3D-RoPE video DiTs is strictly segmented by the three RoPE partitions and is highly symmetric between Q and K; orthogonal 2 imes2 block rotations that respect this structure (Interleaved or Half) equalize those outliers without breaking RoPE or requiring expensive online transforms, while a fixed affine map that sends normalized P onto the full signed INT4 range doubles quantization resolution for the attention matrix.

What carries the argument

RoPE-aware Rotation: a block-diagonal matrix of 2 imes2 orthogonal blocks (Interleaved, mergeable into RoPE; or Half, pairing opposite half-segments) that redistributes RoPE-induced outliers while remaining sparse enough for element-wise execution, paired with Range-optimized P Quantization that maps [0,1] probabilities onto the full INT4 interval [−8,7] via fixed scale 15 and zero-point −8.

Load-bearing premise

The hand-chosen schedule that leaves the first and last few denoising steps and DiT blocks in FP16 is enough to keep generation quality acceptable and will transfer to other 3-D-RoPE video DiTs without retuning.

What would settle it

Run the same INT4 kernels with the identical mixed-precision schedule on a third independent 3-D-RoPE video DiT (for example CogVideoX or Open-Sora 2.0) and measure whether relative Cosine/SSIM/PSNR to the FP16 baseline stay within the ranges reported for Wan2.2 and HunyuanVideo; a clear drop would falsify the transfer claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • INT4 FlashAttention becomes a practical drop-in acceleration path for the current generation of open 3-D-RoPE video DiTs.
  • Half Rotation can be recommended as the default zero-overhead rotation for new models; Interleaved Rotation can be fused offline into RoPE weights for free.
  • Range-optimized P quantization should be enabled for moderate-to-high-entropy attention and can be disabled for strongly conditioned low-entropy patterns such as I2V.
  • The same RoPE-aware rotations and P rectification can be composed with complementary token-reordering methods for further accuracy gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the rotations are orthogonal and sparse, they may also stabilize INT8 or micro-scaled FP4 attention kernels that still suffer from the same RoPE-induced outlier segments.
  • The same segmented-outlier diagnosis could guide rotation design for other multi-axis positional encodings beyond 3-D RoPE.
  • If the mixed-precision schedule can be made adaptive per-block rather than hand-fixed, the fraction of INT4 attention could rise above 80 % without quality loss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes RotateAttention, a mixed-precision INT4 FlashAttention framework for DiT-based video generation models that use 3D RoPE. It identifies that 3D RoPE induces segmented, symmetric outlier patterns in Q and K (Fig. 2, Sec. 3), and addresses two obstacles to low-bit attention: (1) RoPE-aware orthogonal rotations (Interleaved, mergeable into RoPE; Half, low-overhead pairing across half-segments) that equalize outliers without unbalancing simultaneous Q/K quantization (Sec. 4.1, SVD argument for orthogonality); and (2) range-optimized affine quantization of the non-negative P matrix that uses the full INT4 range via fixed scale 15 and zero-point -8 (Sec. 4.2). A standard offline Hadamard is fused into V. With selective FP16 fallback on early/late denoising steps and DiT blocks (68-81% INT4), experiments on Wan2.2 (I2V/T2V) and HunyuanVideo report near-baseline relative metrics (Table 1), visual fidelity superior to SageAttention (Figs. 4-5), 2.2 imes kernel speedup, and up to 1.68 imes end-to-end speedup.

Significance. If the results hold under the stated scope, this is a solid systems contribution for accelerating compute-bound 3D-RoPE video DiTs, orthogonal to sparse attention. The RoPE-induced outlier analysis and the clean necessity-of-orthogonality argument (Sec. 4.1) usefully explain why LLM-style unconstrained or non-orthogonal rotations do not transfer when both Q and K must be quantized. Zero-overhead mergeable designs and the simple fixed-scale P map are practical. Strengths include component ablations (Table 1), visual comparisons, and explicit Limitations acknowledging model coverage and schedule hand-tuning. The work is incremental relative to SageAttention but well-targeted and reproducible in spirit.

major comments (2)
  1. [§5.1, Table 1, Limitations] The central quality claim (“nearly identical to full-precision baselines”) is schedule-dependent. §5.1 hand-selects FP16 for the first/last few sampling steps and first/last DiT blocks (yielding 68–81 % INT4). Table 1 still shows non-negligible residual gaps (Wan2.2-T2V best Cosine 0.956, SSIM ~0.66–0.67, PSNR ~18.5; I2V Cosine ~0.98). Without a sensitivity study on the number of fallback steps/blocks, or an automatic selection rule, it is unclear how much of the “nearly identical” result is due to the rotations/P map versus the FP16 safety net, and whether the schedule transfers to other 3D-RoPE DiTs without retuning. This is the load-bearing soft spot already flagged in Limitations; a short ablation or transfer experiment would strengthen the claim.
  2. [§4.2, §5.4, Table 1] Range-optimized P helps T2V but is neutral-to-harmful on Wan2.2-I2V (Table 1: Optim-P alone drops Cosine/SSIM/PSNR relative to plain INT4). The low-entropy explanation in §5.4 is plausible, yet the method still presents Optim-P as a core technique without a decision rule or per-model switch. For the claim that the full framework is robust, either disable Optim-P under strong conditioning or quantify when the affine map is beneficial.
minor comments (6)
  1. [Abstract, §4.2] Abstract and Eq. (11)–(12) write P = exp(QK − max(QK)); standard notation is QK⊤. Minor but repeated.
  2. [Fig. 2, §3.1] Fig. 2 caption and Sec. 3.1 claim “strong distributional symmetry” and “structured sparsity”; a quantitative correlation or half-segment energy ratio would make the observation sharper.
  3. [§4.1, §5.3, Fig. 6] Learned-rotation experiment (Sec. 4.1, Fig. 6) uses only 1–2 calibration samples and is labeled preliminary; either move fully to appendix or report variance over more seeds/samples so readers can judge stability.
  4. [§5.2] VBench is deferred to supplementary and described as less discriminative; a one-sentence summary of absolute scores in the main text would help readers who care about semantic/temporal metrics.
  5. [§5.4] Practical recommendation in §5.4 prefers Half Rotation by default; a short decision tree (I2V vs T2V, mergeability constraints) would make the guidance more actionable.
  6. [Throughout] Typos / polish: “DiT-basedvideogenerationmodels” spacing artifacts appear in several places; “incoherencemetric” etc. Clean for camera-ready.

Circularity Check

1 steps flagged

No significant circularity: empirical systems design from observed RoPE-induced outliers, evaluated against external FP16 baselines; only mild schedule hand-tuning on the same models.

specific steps
  1. other [§5.1 Experimental Setup (mixed-precision schedule)]
    "For Wan2.2-I2V, we preserve FP16 precision for the first 6 and last 4 sampling steps, as well as the first and last 2 DiT blocks; all remaining FlashAttention operations are executed with our INT4 attention kernel, resulting in 68% of FlashAttention operations running in INT4. For Wan2.2-T2V, FP16 is maintained for the first and last 2 sampling steps and DiT blocks, enabling 81% INT4 attention execution."

    The accuracy-sensitive blocks/steps are identified by inspecting the same models whose final quality is later reported as 'nearly identical' to FP16. This is ordinary hand-tuning of a mixed-precision schedule rather than a derivation that reduces by construction; the central rotation and P-quantization claims remain independent of the schedule choice.

full rationale

RotateAttention is an engineering paper whose load-bearing claims are (1) RoPE-aware 2x2 orthogonal rotations (Interleaved/Half) that equalize the segmental incoherence patterns measured in Fig. 2, (2) a fixed affine map that expands non-negative P onto the full INT4 range, and (3) offline-fused Hadamard on V. The SVD argument in §4.1 shows why non-orthogonal transforms unbalance simultaneous Q/K quantization; it is self-contained mathematics, not a fitted identity. Quality is measured by Cosine/MSE/SSIM/PSNR and visual inspection against an external FP16 baseline on held-out prompts (Table 1, Figs. 4–5). The sole mild circularity is that the mixed-precision fallback schedule (first/last steps and blocks, 68–81 % INT4) is chosen by inspecting the same models later evaluated; this is ordinary hyper-parameter selection, not a prediction forced by construction, and the paper itself flags limited transfer in Limitations. No self-definitional equations, no fitted-parameter-as-prediction, no load-bearing self-citation uniqueness theorems, and no renaming of known results. Score 1 reflects only that schedule choice.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper is an engineering method paper. Its free parameters are the hand-chosen mixed-precision schedule and the fixed affine map for P; its axioms are standard linear-algebra facts plus the empirical observation of RoPE-induced outlier structure. No new physical entities are postulated.

free parameters (3)
  • P quantization scale and zero-point = scale=15, zp=-8
    Fixed to scale=15, zp=−8 so that [0,1] maps onto the full signed INT4 interval [−8,7] (§4.2). Chosen by design rather than data fit, but still a free design choice that affects all reported numbers.
  • Mixed-precision fallback schedule = 68 % / 81 % / 76 % INT4
    Number of leading/trailing denoising steps and DiT blocks kept in FP16 (e.g., first 6 + last 4 steps and first/last 2 blocks for Wan2.2-I2V) is selected by hand to reach 68–81 % INT4 while preserving quality (§5.1).
  • Rotation block initialization = H2
    Non-trainable 2×2 blocks default to the Hadamard matrix H2; learned rotations use 1–2 calibration samples (§4.1, §5.3).
axioms (3)
  • standard math Orthogonal transformations preserve the relative quantization difficulty of Q and K when their outlier profiles are symmetric.
    Derived via SVD in §4.1; if singular values ≠1 the ranges of Q and K become unbalanced under simultaneous low-bit quantization.
  • domain assumption 3-D RoPE partitions feature channels into temporal/height/width segments whose outlier energy concentrates in one half of each segment and is symmetric between Q and K.
    Empirical observation on Wan2.2 (Fig. 2, §3); the rotation designs rest on this structure holding for other 3-D-RoPE DiTs.
  • ad hoc to paper Selective FP16 fallback on a small fraction of steps and blocks is sufficient to keep perceptual quality acceptable.
    Stated as the operating regime in §5.1; without it the pure-INT4 numbers would be worse.

pith-pipeline@v1.1.0-grok45 · 18858 in / 2892 out tokens · 34621 ms · 2026-07-14T16:52:31.576823+00:00 · methodology

0 comments
read the original abstract

In $\textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}$, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized $\textbf{FlashAttention}$ offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: $\textbf{1)}$ applying online rotation matrices -- a widely used technique for mitigating outliers in Queries ($Q$) and Keys ($K$) -- is difficult to reconcile with $\textbf{RoPE}$; and $\textbf{2)}$ the non-negative attention matrix $P = \exp(QK - \max(QK))$ makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of $Q$ and $K$ are strongly affected by the dimensional partitioning of $\textbf{3D RoPE}$. Based on this finding, we propose $\textbf{RotateAttention}$, an efficient $\textbf{mixed-precision INT4 FlashAttention}$ framework tailored for $\textbf{DiT-based video generation models with 3D RoPE}$, using selective $\textbf{FP16 fallback}$ for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: $\textbf{1) RoPE-aware Rotation}$, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in $Q$ and $K$; and $\textbf{2) Range-optimized $P$ Quantization}$, which uses fixed scales and zero-points to fully exploit the $\textbf{INT4 numerical range}$ with minimal computational overhead. Experiments show that $\textbf{RotateAttention}$ preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68$\times$ end-to-end speedup and 2.2$\times$ kernel-level acceleration.

Figures

Figures reproduced from arXiv: 2607.02584 by Binhang Yuan, Harry Yang, Jinxi Li, Wanli Lan, Yaofu Liu.

Figure 1
Figure 1. Figure 1: Quantization differences between LLM decoding and 3D-RoPE video [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Channel-wise incoherence profiles of Q and K for Wan2.2. Incoher￾ence is partitioned into three regions corresponding to the 3D RoPE spatiotemporal segments. We observe strong distributional symmetry between Q and K, and struc￾tured sparsity within each segment. captures the maximum deviation within each channel. For a given matrix X ∈ {Q, K}, we first compute the sequence-wise mean vector E[X] ∈ R 1×D: \m… view at source ↗
Figure 3
Figure 3. Figure 3: Rotation strategies for Q and K. Interleaved Rotation groups adjacent dimensions into 2×2 orthogonal blocks, while Half Rotation pairs each dimension with its counterpart in the other half-segment. Let R = UΣL⊤ be the SVD of R, where U,L ∈ O(D) and Σ = diag(σ1, . . . , σD). The corresponding transformation on K is \mathbf {R}^{-\top } = ((\mathbf {U}\mathbf {\Sigma }\mathbf {L}^{\top })^{-1})^{\top } = \ma… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of different rotation strategies in Wan2.2 T2V models [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of different rotation strategies in Wan2.2 I2V models (top row: FP16/Full/Half; bottom row: Interleave/W/O Rot/SageAttn) [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of effect of trained Rotation (left to right: FP16, trained with orthogonality, trained without orthogonality). The Interleaved Rotation method significantly improves Cosine Similar￾ity for Wan2.2-I2V, though gains in other metrics are marginal. However, on Wan2.2-T2V, it achieves the highest Cosine Similarity and delivers substantial improvements in MSE, SSIM, and PSNR—second only to Full Rotat… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 11 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems37, 100213–100240 (2024)

    Ashkboos, S., Mohtashami, A., Croci, M.L., Li, B., Cameron, P., Jaggi, M., Al- istarh, D., Hoefler, T., Hensman, J.: Quarot: Outlier-free 4-bit inference in ro- tated llms. Advances in Neural Information Processing Systems37, 100213–100240 (2024)

    2024

  2. [2]

    arXiv preprint arXiv:2506.04985 (2025)

    van Breugel, B., Bondarenko, Y., Whatmough, P., Nagel, M.: Fptquant: Function- preserving transforms for llm quantization. arXiv preprint arXiv:2506.04985 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    In: The Twelfth International Conference on Learning Representations (2024)

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., Jia, J.: LongloRA: Efficient fine-tuning of long-context large language models. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  4. [4]

    Advances in neural information processing systems34, 9355–9366 (2021)

    Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., Shen, C.: Twins: Revisiting the design of spatial attention in vision transformers. Advances in neural information processing systems34, 9355–9366 (2021)

    2021

  5. [5]

    In: The Twelfth International Conference on Learning Representations (2024)

    Dao, T.: Flashattention-2: Faster attention with better parallelism and work par- titioning. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  6. [6]

    Advances in neural information pro- cessing systems35, 16344–16359 (2022)

    Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information pro- cessing systems35, 16344–16359 (2022)

    2022

  7. [7]

    int8 (): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems35, 30318–30332 (2022)

    2022

  8. [8]

    arXiv preprint arXiv:2510.27684 (2025)

    Fan, X., Qiu, Z., Wu, Z., Wang, F., Lin, Z., Ren, T., Lin, D., Gong, R., Yang, L.: Phased dmd: Few-step distribution matching distillation via score matching within subintervals. arXiv preprint arXiv:2510.27684 (2025)

    arXiv 2025

  9. [9]

    arXiv preprint arXiv:2210.17323 (2022)

    Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: Gptq: Accurate post- training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    Pith/arXiv arXiv 2022

  10. [10]

    arXiv preprint arXiv:2505.11497 (2025)

    Huang, Y., Gong, R., Liu, J., Ding, Y., Lv, C., Qin, H., Zhang, J.: Qvgen: Pushing the limit of quantized video generative models. arXiv preprint arXiv:2505.11497 (2025)

    arXiv 2025

  11. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al.: Vbench: Comprehensive benchmark suite for video gener- ative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21807–21818 (2024)

    2024

  12. [12]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    Jiang, H., LI, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A.H., Li, D., Lin, C.Y., Yang, Y., Qiu, L.: MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    2024

  13. [13]

    arXiv preprint arXiv:2412.03603 (2024)

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al.: Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Korhonen, J., You, J.: Peak signal-to-noise ratio revisited: Is simple beautiful? In: 2012 Fourth international workshop on quality of multimedia experience. pp. 37–38. IEEE (2012)

    2012

  15. [15]

    arXiv preprint arXiv:2411.05007 (2024) 16 Y

    Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.Y., Han, S.: Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007 (2024) 16 Y. Liu et al

    arXiv 2024

  16. [16]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Li, S., Hu, Y., Ning, X., Liu, X., Hong, K., Jia, X., Li, X., Yan, Y., Ran, P., Dai, G., et al.: Mbq: Modality-balanced quantization for large vision-language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4167–4177 (2025)

    2025

  17. [17]

    In: Forty-first International Confer- ence on Machine Learning (2024)

    Li, S., Ning, X., Wang, L., Liu, T., Shi, X., Yan, S., Dai, G., Yang, H., Wang, Y.: Evaluating quantized large language models. In: Forty-first International Confer- ence on Machine Learning (2024)

    2024

  18. [18]

    In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision

    Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., Keutzer, K.: Q-diffusion: Quantizing diffusion models. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision. pp. 17535–17545 (2023)

    2023

  19. [19]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z., Wei, Y.: Duquant: Distributing outliers via dual transformation makes stronger quantized LLMs. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    2024

  20. [20]

    Proceedings of machine learning and systems6, 87–100 (2024)

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.M., Wang, W.C., Xiao, G., Dang, X., Gan, C., Han, S.: Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems6, 87–100 (2024)

    2024

  21. [21]

    In: The Fourteenth International Conference on Learning Repre- sentations (2026)

    Liu, T., Li, S., Yang, J., Zhao, T., Zhou, F., Song, X., Dai, G., Yan, S., Yang, H., Wang, Y.: PM-KVQ: Progressive mixed-precision KV cache quantization for long-cot LLMs. In: The Fourteenth International Conference on Learning Repre- sentations (2026)

    2026

  22. [22]

    arXiv preprint arXiv:2401.04585 (2024)

    Liu, X., Li, Z., Xiao, J., Chen, M., Li, J., Gu, Q.: Eda-dm: Enhanced distribu- tion alignment for post-training quantization of diffusion models. arXiv preprint arXiv:2401.04585 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchicalvisiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)

    2021

  24. [24]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Liu,Z.,Zhao, C.,Fedorov,I., Soran,B., Choudhary,D., Krishnamoorthi, R.,Chan- dra, V., Tian, Y., Blankevoort, T.: Spinquant: LLM quantization with learned ro- tations. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  25. [25]

    In: 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., Bianchini, R.: Splitwise: Efficient generative llm inference using phase splitting. In: 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). pp. 118–132. IEEE (2024)

    2024

  26. [26]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023)

    2023

  27. [27]

    arXiv preprint arXiv:2503.09642 (2025)

    Peng, X., Zheng, Z., Shen, C., Young, T., Guo, X., Wang, B., Xu, H., Liu, H., Jiang, M., Li, W., et al.: Open-sora 2.0: Training a commercial-level video generation model in $200 k. arXiv preprint arXiv:2503.09642 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    ACM Transactions on Storage (2024)

    Qin, R., Li, Z., He, W., Cui, J., Tang, H., Ren, F., Ma, T., Cai, S., Zhang, Y., Zhang, M., et al.: Mooncake: A kvcache-centric disaggregated architecture for llm serving. ACM Transactions on Storage (2024)

    2024

  29. [29]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems (2024) RotateAttention 17

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., Dao, T.: Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In: The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems (2024) RotateAttention 17

    2024

  30. [30]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shang, Y., Yuan, Z., Xie, B., Wu, B., Yan, Y.: Post-training quantization on dif- fusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1972–1981 (2023)

    1972

  31. [31]

    Neurocomputing568, 127063 (2024)

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing568, 127063 (2024)

    2024

  32. [32]

    Advances in Neural Information Processing Systems37, 76775–76818 (2024)

    Sui, Y., Li, Y., Kag, A., Idelbayev, Y., Cao, J., Hu, J., Sagar, D., Yuan, B., Tulyakov, S., Ren, J.: Bitsfusion: 1.99 bits weight quantization of diffusion model. Advances in Neural Information Processing Systems37, 76775–76818 (2024)

    2024

  33. [33]

    In: Forty-second International Conference on Machine Learning (2025)

    Sun,Y.,Liu,R.,Bai,H.,Bao,H.,Zhao,K.,Li,Y.,JiaxinHu,Yu,X.,Hou,L.,Yuan, C., Jiang, X., Liu, W., Yao, J.: Flatquant: Flatness matters for LLM quantization. In: Forty-second International Conference on Machine Learning (2025)

    2025

  34. [34]

    In: European Conference on Computer Vision

    Tang, S., Wang, X., Chen, H., Guan, C., Wu, Z., Tang, Y., Zhu, W.: Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. In: European Conference on Computer Vision. pp. 404–420. Springer (2024)

    2024

  35. [35]

    x-bit kv cache quan- tization for video large language models

    Tao, K., You, H., Sui, Y., Qin, C., Wang, H.: Plug-and-play 1. x-bit kv cache quan- tization for video large language models. arXiv preprint arXiv:2503.16257 (2025)

    arXiv 2025

  36. [36]

    In: The Twelfth International Conference on Learning Representations (2024)

    Venkataramanan, S., Ghodrati, A., Asano, Y.M., Porikli, F., Habibian, A.: Skip- attention: Improving vision transformers by paying less attention. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  37. [37]

    arXiv preprint arXiv:2503.20314 (2025)

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025)

    Pith/arXiv arXiv 2025

  38. [38]

    arXiv preprint arXiv:2504.17577 (2025)

    Wang, L., Cheng, Y., Shi, Y., Tang, Z., Mo, Z., Xie, W., Ma, L., Xia, Y., Xue, J., Yang, F., et al.: Tilelang: A composable tiled programming model for ai systems. arXiv preprint arXiv:2504.17577 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    IEEE transactions on image processing 13(4), 600–612 (2004)

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)

    2004

  40. [40]

    In: Forty-second International Conference on Machine Learning (2025)

    Xi, H., Yang, S., Zhao, Y., Xu, C., Li, M., Li, X., Lin, Y., Cai, H., Zhang, J., Li, D., Chen, J., Stoica, I., Keutzer, K., Han, S.: Sparse video-gen: Accelerating video dif- fusion transformers with spatial-temporal sparsity. In: Forty-second International Conference on Machine Learning (2025)

    2025

  41. [41]

    Advances in neural information processing systems37, 119638–119661 (2024)

    Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., Sun, M.: Infllm: Training-free long-context extrapolation for llms with an efficient context memory. Advances in neural information processing systems37, 119638–119661 (2024)

    2024

  42. [42]

    In: International conference on machine learning

    Xiao,G.,Lin,J.,Seznec,M.,Wu,H.,Demouth,J.,Han,S.:Smoothquant:Accurate and efficient post-training quantization for large language models. In: International conference on machine learning. pp. 38087–38099. PMLR (2023)

    2023

  43. [43]

    arXiv preprint arXiv:2309.17453 (2023)

    Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Yang, S., Xi, H., Zhao, Y., Li, M., Zhang, J., Cai, H., Lin, Y., Li, X., Xu, C., Peng, K., Chen, J., Han, S., Keutzer, K., Stoica, I.: Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  45. [45]

    arXiv preprint arXiv:2408.06072 (2024)

    Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al.: Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024)

    Pith/arXiv arXiv 2024

  46. [46]

    Advances in Neural Information Processing Systems37, 1196–1219 (2024) 18 Y

    Yuan, Z., Zhang, H., Pu, L., Ning, X., Zhang, L., Zhao, T., Yan, S., Dai, G., Wang, Y.: Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems37, 1196–1219 (2024) 18 Y. Liu et al

    2024

  47. [47]

    In: First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models (2025)

    Zhang, J., Huang, H., Zhang, P., wei, J., Zhu, J., Chen, J.: Sageattention2: Efficient attention with smoothing q and per-thread quantization. In: First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models (2025)

    2025

  48. [48]

    arXiv preprint arXiv:2509.24006 (2025)

    Zhang, J., Wang, H., Jiang, K., Yang, S., Zheng, K., Xi, H., Wang, Z., Zhu, H., Zhao, M., Stoica, I., et al.: Sla: Beyond sparsity in diffusion transformers via fine- tunable sparse-linear attention. arXiv preprint arXiv:2509.24006 (2025)

    arXiv 2025

  49. [49]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Zhang,J.,wei,J.,Wang,H.,Zhang,P.,Xu,X.,Huang,H.,Jiang,K.,Zhu,J.,Chen, J.: Sageattention3: Microscaling FP4 attention for inference and an exploration of 8-bit training. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  50. [50]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Zhang, J., wei, J., Zhang, P., Zhu, J., Chen, J.: Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  51. [51]

    In: Forty-second International Conference on Machine Learning (2025)

    Zhang, J., Xiang, C., Huang, H., wei, J., Xi, H., Zhu, J., Chen, J.: Spargeattention: Accurate and training-free sparse attention accelerating any model inference. In: Forty-second International Conference on Machine Learning (2025)

    2025

  52. [52]

    arXiv preprint arXiv:2502.04507 (2025)

    Zhang, P., Chen, Y., Su, R., Ding, H., Stoica, I., Liu, Z., Zhang, H.: Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Zhao, T., Fang, T., Huang, H., Wan, R., Soedarmadji, W., Liu, E., Li, S., Lin, Z., Dai, G., Yan, S., Yang, H., Ning, X., Wang, Y.: Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  54. [54]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Zhao, T., Hong, K., Yang, X., Xiao, X., Li, H., Ling, F., Xie, R., Chen, S., Zhu, H., Yichong, Z., Wang, Y.: PAROAttention: Pattern-aware reordering for efficient sparse and quantized attention in visual generation models. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  55. [55]

    In: European Conference on Computer Vision

    Zhao, T., Ning, X., Fang, T., Liu, E., Huang, G., Lin, Z., Yan, S., Dai, G., Wang, Y.: Mixdq: Memory-efficient few-step text-to-image diffusion models with metric- decoupled mixed precision quantization. In: European Conference on Computer Vision. pp. 285–302. Springer (2024)

    2024

  56. [56]

    arXiv preprint arXiv:2412.20404 (2024)

    Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., You, Y.: Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404 (2024)

    Pith/arXiv arXiv 2024

  57. [57]

    In: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)

    Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Zhang, H.: Dist- serve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). pp. 193–210 (2024)

    2024