Pith. sign in

REVIEW 3 major objections 6 minor 35 references

A fixed MXFP4 scaling boundary of 7.25 plus consistent softmax-path quantization recovers nearly full-precision video attention quality without calibration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 16:24 UTC pith:A3MDX3EZ

load-bearing objection Clean systems paper: closed-form MXFP4 scale boundary plus a softmax placement fix that nearly restores FP16 video quality—math is sound for what it claims, with one real caveat on the error proxy. the 3 major comments →

arxiv 2607.24377 v1 pith:A3MDX3EZ submitted 2026-07-27 cs.LG cs.AIcs.CV

MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention

Jianlin Yu , Jing Lin , Linghui Kong , Aiyue Chen , Weiyi Sun , Chenyu Zeng , Wangli Lan , Jinxi Li
show 8 more authors
Zhuo Zheng Ziyang Yue Danning Ke Fei Yi Tianchi Hu Yuan Ding Yiwu Yao Junsong Wang
This is my paper
classification cs.LG cs.AIcs.CV
keywords MXFP4attention quantizationpost-training quantizationvideo diffusionFlashAttentionmicroscalingsoftmax normalizationdata-free PTQ
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video diffusion models spend heavily on attention, and packing that math into the open MXFP4 4-bit format usually hurts image quality. The authors argue the damage comes from two concrete numerical failures: power-of-two block scales force a bad trade-off between clipping large values and under-resolving small ones, and quantizing only part of the online softmax path breaks the rule that attention weights must sum to one. MXAttention fixes both without data or training. Universal Optimal Scaling derives a single shared-scale boundary, Qmax = 7.25, that minimizes a data-free quantization-error objective for any block-maximum distribution, because power-of-two scaling makes the optimum grid-determined rather than data-determined. Pre-Normalization Quantization then feeds the same quantized softmax exponentials into both the row-sum and the output update, so normalized weights still sum to one by construction. On two large text-to-video models the method closes at least 95% of the Imaging Quality gap to FP16, stays within 0.01 of FP16 on reported VBench scores, and matches strong proprietary 4-bit baselines with almost no extra work in a fused kernel.

Core claim

Under ceiling-based power-of-two MXFP4 scaling, the data-free objective that averages relative E2M1 projection error over the normalized block-maximum interval is globally minimized at the fixed boundary Qmax = 7.25 for any absolutely continuous block-maximum law; combined with quantizing unnormalized softmax exponentials before both the row-sum and output-accumulator updates, this yields MXFP4 attention that recovers nearly FP16 generation quality on video diffusion models without calibration or search.

What carries the argument

Universal Optimal Scaling (UOS): the log-periodic density of power-of-two-normalized block maxima makes the sign of the objective derivative depend only on the E2M1 grid difference D(q)−D(q/2), which vanishes uniquely at q★ = 7.25; Pre-Normalization Quantization (PNQ) then reuses one quantized exponential tile for both FlashAttention state updates so induced row sums equal one exactly.

Load-bearing premise

The method treats a simple relative squared projection error on the 4-bit number grid as the right stand-in for what actually matters in attention and final video quality; if a different error measure were the true target, 7.25 need not be optimal.

What would settle it

Re-run the per-block scale sweep and end-to-end VBench comparison while optimizing a different objective (for example attention-output MSE or softmax KL) instead of the paper’s D(x); if the best fixed boundary moves well away from 7.25 or Imaging Quality no longer recovers ~95% of the FP16 gap, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MXFP4 attention can be deployed as pure post-training quantization with one fixed scale rule and no calibration set.
  • Fused FlashAttention-style kernels only need a changed scale-selection constant and reuse of the already-quantized softmax tile for the row sum.
  • Open MXFP4 can match or approach strong NVFP4 video-attention quality when scaling and softmax placement are handled this way.
  • The same UOS boundary applies across tensors, layers, and models because the optimum is grid-driven, not distribution-driven.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same log-periodic argument may yield closed-form optimal boundaries for other power-of-two microscaling formats beyond E2M1 MXFP4.
  • PNQ’s consistency principle is format-agnostic and could reduce row-sum bias in any low-bit online-softmax path, not only MXFP4.
  • If hardware vendors expose a programmable block-scale boundary, shipping 7.25 as the default for attention GEMMs would be a low-cost quality win.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes MXAttention, a data-free post-training quantization framework for MXFP4 attention in video diffusion models, with two components. (1) Universal Optimal Scaling (UOS): exploiting the periodized density induced by power-of-two E8M0 scales, the authors show that the derivative sign of a global quantization-error objective J(q)=E[D(X_q)], with D(x)=E(x)/x³ the normalized integrated squared E2M1 projection error, is independent of the block-maximum distribution (Lemma 1, Eqs. 31–32), and solve the resulting grid-dependent condition exactly, obtaining the closed-form boundary Q_max=29/4=7.25 with a complete global sign chart (Table 6). (2) Pre-Normalization Quantization (PNQ): quantizing the unnormalized softmax exponential tile before both the row-sum and output-accumulator updates in the FlashAttention loop, so the induced attention weights are row-normalized by construction (Proposition 1). Experiments on Wan2.2-14B and HunyuanVideo-13B show MXAttention recovering at least 95% of the VBench Imaging Quality gap between direct OCP MXFP4 and FP16 and staying within 0.01 of FP16 on all reported VBench metrics, competitive with NVFP4-based baselines. Code is integrated into MindIE-SD.

Significance. If the results hold, this is a useful contribution at the intersection of low-precision attention and open-standard formats. The UOS result is a genuinely parameter-free derivation: the constant 7.25 falls out of the E2M1 grid and a stated analytical objective via exact piecewise integration, not from fitting to VBench; the periodization argument (Lemma 1) is correct and cleanly removes the block-maximum law from the derivative sign. PNQ's row-sum identity is immediate but addresses a real and under-discussed failure mode of quantized online softmax, and the mechanism diagnostics (Fig. 7, Table 4) are more honest than typical. The end-to-end evidence spans two large, architecturally distinct video models, and the implementation is publicly available in MindIE-SD. Given native MXFP4 support across Blackwell, MI350, and Ascend 950, a calibration-free recipe with near-FP16 quality has clear practical value.

major comments (3)
  1. [§4.1, Eqs. (21)–(23), Theorem 1; §5.5.1, Table 4] The 'distribution-independent' claim of Theorem 1 is proved only with respect to the block-maximum law. The objective itself, E(x)=∫₀ˣ(v−Π(v))²dv, weights all magnitudes in [0,x] uniformly, which is an implicit within-block distributional assumption, and it is this weighting that determines the 7.25 root of Eq. (37). This is not an internal inconsistency — the theorem is correct as stated — but the abstract/§4.1 language will be read as stronger than what is proved. The empirical oracle sweep (Table 4) validates realized tensor MSE for Q, K, V only; the unnormalized softmax tile P̃, whose within-block distribution (mass near 0 plus a spike at the block max) most violates the uniform weighting, is never swept, and HunyuanVideo Q already shows 45% of blocks preferring ≤7.15. Requested: (a) scope the 'data-free / distribution-independent' phrasing explicitly; (b) add a P̃-tile MSE sweep or
  2. [§5.1 (Implementation Details), Table 2; Abstract] The headline Wan2.2 results use a hybrid-precision policy (Block 0 and denoising steps 38–39 retained in high precision), disclosed only in §5.1. Under fully 4-bit attention (Table 3), MXAttention on Wan2.2 reaches Imaging Quality 0.6842 versus OCP 0.5452 and FP16 0.7085 — roughly 85% of the gap, not the ≥95% quoted in the abstract. For an attention-quantization paper, needing a specific block and the final steps in high precision is material information. The abstract should disclose that the 95% figure is conditional on the hybrid policy, and the paper should explain why Wan2.2 requires it while HunyuanVideo (fully 4-bit, exceeding FP16) does not.
  3. [§5.6] The claims of 'negligible algorithmic overhead', fusion-friendliness, and competitiveness with NVFP4-based baselines are supported only by qualitative argument; the paper contains no latency, throughput, or kernel-timing measurements, and the claim that rotation and quantization can run on orthogonal compute units 'eliminating resource contention' is hardware-specific and unmeasured. Since the motivation of the work is attention acceleration, at least end-to-end and attention-kernel runtime numbers on one platform (or a clear softening of these claims) should be provided.
minor comments (6)
  1. [§5.1, Table 2] No prompt counts, seed counts, or variability estimates are given. The '<0.01 absolute degradation' parity claim and small margins (e.g., 0.0031 Imaging Quality below FP16 on Wan2.2; ~0.007 over NVFP4) are within plausible evaluation noise for VBench-style metrics. Please state the number of prompts/seeds and, ideally, report variance or a paired-difference confidence interval; the mechanism studies use only two prompts.
  2. [§5.4, Table 3] Two issues. (i) The 'w/o UOS' row does not state which boundary replaces 7.25 (presumably exact OCP); please make this explicit. (ii) The full configuration is not best on every metric: removing UOS raises Wan2.2 Aesthetic Quality (0.6176 vs 0.5994) and HunyuanVideo Imaging Quality (0.6423 vs 0.6380). The text's 'complementary gains' / 'strongest overall balance' phrasing should acknowledge these small regressions.
  3. [Appendix G; §5.4] Since Appendix G establishes that OAS at B=32 is equivalent to the ceiling boundary q=7, an ablation row using q=7 (or a q∈{7.0, 7.25, 7.5} comparison) would isolate how much of the UOS gain is specific to 7.25 versus any boundary near 7; as Table 4 shows 7.20 is also frequent, this would strengthen the case for the exact constant.
  4. [§5.5.2, Figure 7] The row-sum diagnostic is computed on the normalized probability matrix P, while PNQ quantizes the unnormalized tile P̃. The connection is mentioned but the quantitative transfer (how row-sum error on materialized P relates to the online-softmax mismatch of Eq. (18)) deserves one clarifying sentence.
  5. [Abstract/Introduction] Typographical: missing spaces at 'proposeMXAttention', 'First,Universal Optimal Scaling (UOS)leverages', 'we proposeMXAttention' (Conclusion). Also 'claim: less than 0.01 absolute degradation on all reported VBench metrics' is verified against Table 2 but only under the Wan2.2 hybrid policy — see Major Comment 2.
  6. [§2, Eq. (11); Figure 2] Clarify that e_elem^max=2 is the exponent of the grid value 4=2² (the wording 'corresponding to the grid value 4' is terse). In Figure 2, define the 'first harmonic', 'two harmonics', and 'concentrated' wrapped densities precisely (e.g., Fourier coefficients used) so the figure is reproducible.

Circularity Check

0 steps flagged

No significant circularity: Q_max=7.25 is a closed-form minimizer of a stated analytical objective, and PNQ’s row-sum identity is an intentional invariant, not a fitted or self-citing prediction.

full rationale

The load-bearing UOS claim (Theorem 1) is derived inside the paper from a fully specified, data-free objective J(q)=E[D(X_q)] with D(x)=E(x)/x^3 built from the fixed E2M1 grid. Log-periodicity of power-of-two microscaling cancels the block-maximum density from the sign of J'(q), and the unique admissible root of Δ(q)=0 on the relevant interval is q*=29/4=7.25 by exact piecewise integration—not by fitting to VBench, generation metrics, or per-layer search. Empirical block-MSE oracle sweeps and end-to-end video metrics are presented only as validation of that fixed boundary, not as its source. PNQ’s Proposition 1 (induced weights sum to one) follows immediately from using the same quantized exponential tile in both the row-sum and output-accumulator updates; the paper states this as preservation “by construction,” i.e., an algorithmic invariant, not an independent empirical prediction. There is no load-bearing self-citation uniqueness theorem, no parameter fitted on a subset and re-labeled as a prediction, and no renaming of an external empirical law as the central result. Concerns that D(x) embeds a uniform within-block weighting are about objective choice and external validity, not circular reduction of outputs to inputs. The derivation chain is self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 2 invented entities

The load-bearing novelty is mostly mathematical structure of power-of-two MX scaling plus a chosen analytical error functional; almost no fitted knobs. Claims rest on standard FlashAttention/MXFP4 definitions, absolute continuity of log block-maxima, representable E8M0 exponents, and the modeling choice that minimizing average D(X_q) is the right scale objective.

axioms (5)
  • domain assumption MXFP4 uses 32-element blocks with shared E8M0 power-of-two scale and E2M1 elements with finite grid {0,±0.5,...,±6} and round-to-nearest-with-saturation (OCP MX spec).
    Entire UOS derivation and overflow-rounding analysis are format-specific; invoked throughout §2 and §4.1.
  • ad hoc to paper The right data-free scale objective is J(q)=E[D(X_q)] with D(x)=E(x)/x^3 the normalized integrated squared projection error onto nonnegative E2M1.
    This functional choice delivers distribution-independent optimality at 7.25; other losses could move the optimum. Defined in Eqs. 21–23.
  • standard math Block-maximum log-magnitude U=log2 M is absolutely continuous so the periodized density H exists and endpoint measure-zero ties can be ignored.
    Used for g_q density, Lemma 1, and Theorem 1 (Appendix A).
  • domain assumption FlashAttention-2 online softmax maintains unnormalized accumulator and row sum; final O = diag(ℓ)^{-1} Õ without materializing P.
    PNQ and the direct-mismatch failure mode are defined inside this recurrence (§2, §4.2).
  • domain assumption A fixed orthogonal Hadamard rotation on Q/K is an acceptable, calibration-free outlier suppressor borrowed from prior PTQ practice.
    Used in the full pipeline and ablations; not derived here (§4.3, §5.4).
invented entities (2)
  • Universal Optimal Scaling (UOS) boundary Q_max=7.25 independent evidence
    purpose: Single fixed shared-scale selection rule claimed to globally minimize the paper’s MXFP4 quantization-error objective without calibration.
    Derived from E2M1 integrated error, not postulated as a new physical object; still a paper-specific construct defining the method.
  • Pre-Normalization Quantization (PNQ) independent evidence
    purpose: Quantization placement that forces identical quantized exponential tiles into row-sum and output updates so induced attention rows sum to one.
    Methodological construct; correctness is by algebraic construction (Proposition 1), validated by row-sum diagnostics.

pith-pipeline@v1.2.0-grok45-kimik3 · 30061 in / 3657 out tokens · 68030 ms · 2026-07-31T16:24:57.120961+00:00 · methodology

0 comments
read the original abstract

The quadratic cost of attention is a major bottleneck in diffusion-based video generation models. MXFP4 attention provides a promising path toward efficient inference, but direct MXFP4 quantization often degrades generation quality due to two numerical issues: the clipping-underflow trade-off from power-of-two scaling and the row-wise normalization error introduced in the softmax loop. We propose MXAttention, a data-free post-training quantization framework for MXFP4 attention. MXAttention introduces two components: Universal Optimal Scaling (UOS), which exploits the periodic structure of power-of-two microscaling to derive a distribution-independent optimal scaling boundary Qmax=7.25 without calibration or search, and Pre-Normalization Quantization (PNQ), which quantizes unnormalized softmax exponentials before row-wise summation to preserve normalization by construction. Experiments on Wan2.2 and HunyuanVideo show that MXAttention closes at least 95% of the VBench Imaging Quality gap between OCP MXFP4 and FP16, substantially improves frame-level similarity, and preserves FP16-level generation quality with less than 0.01 absolute degradation on all reported VBench metrics. MXAttention also achieves performance competitive with strong NVFP4-based baselines with negligible overhead when fused into the attention pipeline. The implementation is publicly available in MindIE-SD.

Figures

Figures reproduced from arXiv: 2607.24377 by Aiyue Chen, Chenyu Zeng, Danning Ke, Fei Yi, Jianlin Yu, Jing Lin, Jinxi Li, Junsong Wang, Linghui Kong, Tianchi Hu, Wangli Lan, Weiyi Sun, Yiwu Yao, Yuan Ding, Zhuo Zheng, Ziyang Yue.

Figure 1
Figure 1. Figure 1: Overview of MXAttention. MXAttention integrates Universal Optimal Scaling (UOS) and Pre-Normalization Quantization (PNQ) into the FlashAttention pipeline. A fixed Hadamard rotation is applied to Q/K for outlier suppression. UOS applies the distribution-independent MXFP4 scaling boundary Qmax = 7.25 to all MXFP4-quantized attention tensors. During the online-softmax loop, PNQ quantizes the unnormalized soft… view at source ↗
Figure 2
Figure 2. Figure 2: UOS objective under different wrapped distributions. (a) Percentage increase in the objective relative to its value at q = 7.25 under a log-uniform density. (b) The same quantity under representative nonuniform wrapped densities. Distribution-specific harmonics change the curve shapes, but all cases share the minimum q = 7.25. The first branch is negative. In the second branch, the factor 4q − 27 vanishes … view at source ↗
Figure 3
Figure 3. Figure 3: Normalized block-maximum ranges under representative MXFP4 scaling rules. The exact OCP floor rule maps block maxima to [4, 8), with normalized maxima above 7 entering the overflow-rounding region. TFS uses (3, 6] to avoid overflow of the block maximum, while UOS selects the intermediate range (3.625, 7.25]. Relative to OCP, UOS substantially narrows this interval. The optimum is not obtained by separately… view at source ↗
Figure 4
Figure 4. Figure 4: Pre-Normalization Quantization (PNQ). (A) Standard attention forms the normalized probability matrix P before computing O = P V . (B) FlashAttention avoids materializing P and updates both the row-wise sum of exponentials ℓi and the unnormalized output accumulator Oei from the same exponential tile Peij . (C) A direct placement uses the unquantized tile Peij in the ℓi update but the quantized tile Pbij in … view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of 4-bit attention methods. Each row shows frames generated with the same prompt and random seed. The left three columns correspond to a Wan2.2 video and the right three columns correspond to a HunyuanVideo video, with multiple frames sampled from the same generated video. Direct OCP MXFP4 produces noticeable changes in object identity and appearance, while MXAttention better preserv… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative ablation under fully 4-bit attention. From top to bottom: full MXAt￾tention, without UOS, without UOS and PNQ, and direct OCP MXFP4. Removing UOS and PNQ progressively increases visible artifacts and reduces generation quality. The fixed Hadamard rotation further reduces the effect of Q/K outliers. Adding it to direct OCP MXFP4 raises Imaging Quality from 0.5452 to 0.6352 on Wan2.2 and from 0.4… view at source ↗
Figure 7
Figure 7. Figure 7: Row-sum diagnostics for MXFP4 probability quantization. (A) Across 80 Wan2.2 attention tensors at denoising step 9, covering two prompts and 40 layers, the mean row sum of each quantized tensor is below one. (B) The row-sum error is decomposed into zeroing, saturation, downward rounding, and upward rounding. (C) Pre-quant global scaling can restore the average row sum of a representative tensor but does no… view at source ↗
Figure 8
Figure 8. Figure 8: Empirical block-maximum distributions in Wan2.2. The rows show the original block maxima, their logarithms, the wrapped log-domain phases, and the corresponding normalized linear-domain values for Q, K, V , and the normalized attention probability matrix P. The visible nonuniformity corresponds to distribution-specific harmonic components but does not change the UOS boundary condition. The P column is incl… view at source ↗
Figure 9
Figure 9. Figure 9: Effect of Hadamard rotation on block-maximum distributions. Rotation changes the log-domain distribution and its harmonic content; the boundary density ratio remains unchanged. Then e7 = e6 + l log2 x6 7 m = ( e6 − 1, 3 < x6 ≤ 3.5, e6, 3.5 < x6 ≤ 6. (96) Thus, q = 7 halves the TFS scale when x6 ∈ (3, 3.5], mapping this interval to (6, 7], and otherwise leaves the scale unchanged. This is the OAS scale-se… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 9 linked inside Pith

  1. [1]

    Hunyuanvideo: A systematic framework for large video generative models,

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang et al., “Hunyuanvideo: A systematic framework for large video generative models,”arXiv preprint arXiv:2412.03603, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Wan: Open and advanced large-scale video generative models,

    Wan Team, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xieet al., “Wan: Open and advanced large-scale video generative models,”arXiv preprint arXiv:2503.20314, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Cogvideox: Text-to-video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng et al., “Cogvideox: Text-to-video diffusion models with an expert transformer,”arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Latent consistency models: Synthesizing high- resolution images with few-step inference,

    S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao, “Latent consistency models: Synthesizing high- resolution images with few-step inference,” inInternational Conference on Learning Representa- tions, 2024

    2024

  5. [5]

    Phased consistency models,

    F.-Y. Wang, Z. Huang, A. W. Bergman, D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, X. Wang, and H. Li, “Phased consistency models,” inAdvances in Neural Information Processing Systems, 2024

    2024

  6. [6]

    Timestep embedding tells: It’s time to cache for video diffusion model,

    F. Liu, S. Zhang, X. Wang, Y. Wei, H. Qiu, Y. Zhao, Y. Zhang, Q. Ye, and F. Wan, “Timestep embedding tells: It’s time to cache for video diffusion model,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 7353–7363

    2025

  7. [7]

    Real-time video generation with pyramid attention broadcast,

    X. Zhao, X. Jin, K. Wang, and Y. You, “Real-time video generation with pyramid attention broadcast,”arXiv preprint arXiv:2408.12588, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Sageat- tention3: Microscaling FP4 attention for inference and an exploration of 8-bit training,

    J. Zhang, J. Wei, P. Zhang, X. Xu, H. Huang, H. Wang, K. Jiang, J. Zhu, and J. Chen, “Sageat- tention3: Microscaling FP4 attention for inference and an exploration of 8-bit training,”arXiv preprint arXiv:2505.11594, 2025

    arXiv 2025

  9. [9]

    Attn-qat: 4-bit attention with quantization-aware training,

    P. Zhang, M. Noto, W. Tan, C. Jiang, W. Lin, W. Zhou, and H. Zhang, “Attn-qat: 4-bit attention with quantization-aware training,”arXiv preprint arXiv:2603.00040, 2026

    arXiv 2026

  10. [10]

    OCP Microscaling Formats (MX) Specification Version 1.0,

    Open Compute Project, “OCP Microscaling Formats (MX) Specification Version 1.0,” https: //www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf, 2023. 18

    2023

  11. [11]

    Openai triton on nvidia blackwell boosts ai per- formance and programmability,

    NVIDIA, “Openai triton on nvidia blackwell boosts ai per- formance and programmability,” https://developer.nvidia.com/blog/ openai-triton-on-nvidia-blackwell-boosts-ai-performance-and-programmability/, 2025, demon- strates hardware-accelerated MXFP4 GEMM support on NVIDIA Blackwell

    2025

  12. [12]

    AMD Instinct MI350 Series GPUs,

    Advanced Micro Devices, Inc., “AMD Instinct MI350 Series GPUs,” https://www.amd.com/en/ products/accelerators/instinct/mi350.html, 2025, native MXFP4 and MXFP6 datatype support

    2025

  13. [13]

    Leading a new paradigm for ai infrastructure,

    Huawei, “Leading a new paradigm for ai infrastructure,” https://www.huawei.com/en/news/ 2025/9/hc-xu-keynote-speech, 2025, announced MXFP4 support for the Ascend 950 series

    2025

  14. [14]

    Introducing NVFP4 for efficient and accurate low-precision inference,

    NVIDIA, “Introducing NVFP4 for efficient and accurate low-precision inference,” https:// developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/, 2025

    2025

  15. [15]

    Using FP8 and FP4 with transformer engine,

    ——, “Using FP8 and FP4 with transformer engine,” https://docs.nvidia.com/deeplearning/ transformer-engine/user-guide/examples/fp8 primer.html, 2025

    2025

  16. [16]

    Quarot: Outlier-free 4-bit inference in rotated LLMs,

    S. Ashkboos, A. L. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “Quarot: Outlier-free 4-bit inference in rotated LLMs,” inAdvances in Neural Information Processing Systems, 2024. [Online]. Available: https://proceedings.neurips. cc/paper files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Confe...

    2024

  17. [17]

    Spinquant: LLM quantization with learned rotations,

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort, “Spinquant: LLM quantization with learned rotations,” inInternational Conference on Learning Representations, 2025

    2025

  18. [18]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration,

    J. Zhang, J. Wei, P. Zhang, J. Zhu, and J. Chen, “Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration,” inInternational Conference on Learning Representations, 2025

    2025

  19. [19]

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread INT4 quantization,

    J. Zhang, H. Huang, P. Zhang, J. Wei, J. Zhu, and J. Chen, “Sageattention2: Efficient attention with thorough outlier smoothing and per-thread INT4 quantization,” inInternational Conference on Machine Learning, 2025

    2025

  20. [20]

    Sageattention2++: A more efficient implementation of sageattention2,

    J. Zhang, X. Xu, J. Wei, H. Huang, P. Zhang, C. Xiang, J. Zhu, and J. Chen, “Sageattention2++: A more efficient implementation of sageattention2,”arXiv preprint arXiv:2505.21136, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Unveiling the potential of quantization with MXFP4: Strategies for quantization error reduction,

    J. Chhugani, G. Jeong, B.-Y. Su, Y. Pan, H. Yang, A. Ankit, J. Yu, S. Deng, Y. Chen, N. Satish, and C. Kim, “Unveiling the potential of quantization with MXFP4: Strategies for quantization error reduction,” inInternational Conference on Machine Learning, 2026

    2026

  22. [22]

    Search your block floating point scales!

    T. Gupta, H. Prairie, X. Wu, R. Abhyankar, Q. Wu, A. Silveria, P. Ponnusamy, J. Wang, B. Athi- waratkun, L. Song, T. Dao, D. Y. Fu, and C. De Sa, “Search your block floating point scales!” arXiv preprint arXiv:2605.12464, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    M 2xfp: A metadata-augmented microscaling data format for efficient low-bit quantization,

    W. Hu, C. Zhang, Z. Zhang, C. Guoet al., “M 2xfp: A metadata-augmented microscaling data format for efficient low-bit quantization,”arXiv preprint arXiv:2601.19213, 2026

    arXiv 2026

  24. [24]

    Deltaquant: 4-bit video diffusion models with spatiotemporal delta smoothing,

    X. Li, S. Tesfai, Z. Zhang, H. Xi, S. Yang, L. Zhang, Y. Sun, K. Peng, M. Agrawala, I. Stoica, K. Keutzer, J.-Y. Zhu, S. Han, Y. Lin, and M. Li, “Deltaquant: 4-bit video diffusion models with spatiotemporal delta smoothing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 43 578–43 588

    2026

  25. [25]

    Flashattention: Fast and memory-efficient ex- act attention with IO-awareness,

    T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R´ e, “Flashattention: Fast and memory-efficient ex- act attention with IO-awareness,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 10 448–10 459

    2022

  26. [26]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” in International Conference on Learning Representations, 2024. 19

    2024

  27. [27]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision,

    J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low-precision,”arXiv preprint arXiv:2407.08608, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Microscaling data formats for deep learning,

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolfet al., “Microscaling data formats for deep learning,”arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Oscillation-reduced MXFP4 training for vision transform- ers,

    Y. Chen, H. Xi, J. Zhu, and J. Chen, “Oscillation-reduced MXFP4 training for vision transform- ers,” inProceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267, 2025

    2025

  30. [30]

    K. V. Mardia and P. E. Jupp,Directional Statistics, ser. Wiley Series in Probability and Statistics. Chichester: John Wiley & Sons, 2000

    2000

  31. [31]

    A review of wrapped distributions for circular data,

    W. Bell and S. Nadarajah, “A review of wrapped distributions for circular data,”Mathematics, vol. 12, no. 16, p. 2440, 2024

    2024

  32. [32]

    E. M. Stein and R. Shakarchi,Fourier Analysis: An Introduction, ser. Princeton Lectures in Analysis. Princeton, NJ: Princeton University Press, 2003, vol. 1

    2003

  33. [33]

    Open-sora: Democratizing efficient video production for all,

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You, “Open-sora: Democratizing efficient video production for all,”arXiv preprint arXiv:2412.20404, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Vbench: Comprehensive benchmark suite for video generative models,

    Z. Huang, Y. He, J. Yu, F. Zhang, Y. Zhang, T. Li, W.-S. Zheng, Y. Qiao, and Z. Liu, “Vbench: Comprehensive benchmark suite for video generative models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  35. [35]

    Kuipers and H

    L. Kuipers and H. Niederreiter,Uniform Distribution of Sequences. New York: John Wiley & Sons, 1974. A Exact Density under Power-of-Two Scaling We analyze nonzero blocks for which the required shared-scale exponent lies within the E8M0 exponent range. The casesM= 0 and exponents outside this range are handled by the MXFP4 conversion routine and are omitte...

    1974