REVIEW 4 major objections 5 minor 4 cited by
SageAttention2++: A More Efficient Implementation of SageAttention2
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read SageAttention2++ claims a 3.9x speedup over FlashAttention by switching the P V matmul to an FP8 instruction with an FP16 accumulator, while keeping SageAttention2's accuracy through narrower quantization ranges.
desk verdict A sound, incremental attention-kernel optimization with a real speedup, but the accuracy claims need more evidence and softer wording. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the hardware instruction mma.m16n8k32 with an FP16 accumulator, taken from PTX documentation as 2x faster than the FP32-accumulator FP8 instruction. The paper couples this with a quantization-range constraint: per-block scales $\delta_P = |\max(\tilde P)|/\Pr$ and per-channel scales $\delta_V = |\max(V)|/V_r$ with $\Pr \times V_r \leq 2047/2$ keep the FP16 accumulation from overflowing. A second mechanism, delayed FP32 buffering, accumulates two consecutive mma results in FP16 before converting to FP32, halving the number of data-type conversion instructions.
What would settle it
Isolate mma.f16.f8.f8.f16 and mma.f32.f8.f8.f32 on RTX4090 and measure their throughput ratio; if the FP16-accumulator instruction is not about twice as fast, the paper's claimed 3.9x end-to-end speedup over FlashAttention2 will not hold.
Extended reading notes
Core claim
The central claim is that the attention output $P V$ can be computed with FP8 inputs and an FP16 accumulator, using the instruction mma.f16.f8.f8.f16, without losing SageAttention2's accuracy, provided the quantization scales are adjusted so that the worst-case accumulated product stays inside FP16's range. The paper's derivation bounds 32 accumulated products by $|32 \times p v| \leq 65504$, and after adding one extra FP16 accumulation before conversion the constraint becomes $\Pr \times V_r \leq 2047/2$. The paper chooses $\Pr = 224$ and $V_r = 4.5$, reports attention cosine similarity of 99.97%, and shows end-to-end metrics on Llama3.1, CogvideoX, HunyuanVideo, Wan, Flux, and Stable-Diffusion3.5 that match SageAttention2. The result is a claimed 3.9x speedup over FlashAttention2 for the INT4 Q/K variant and about 3.0x for the INT8 variant.
Load-bearing premise
The entire speedup rests on the unmeasured hardware claim that the FP8 instruction with an FP16 accumulator is exactly 2x faster than the FP8 instruction with an FP32 accumulator on RTX4090 and RTX5090, a figure the paper takes from NVIDIA documentation; if the real instruction throughput gap is smaller, the end-to-end speedup shrinks proportionally.
Editorial extensions
If this is right
- On RTX4090 and RTX5090, SageAttention2++(4+8) reaches about 3.9x and SageAttention2++(8+8) about 3.0x the speed of FlashAttention2 across sequence lengths from 1K to 32K.
- Attention accuracy, measured by cosine similarity and L1 error, stays at the same level as SageAttention2 when the quantization ranges are narrowed to $\Pr=224$, $V_r=4.5$.
- End-to-end metrics on Llama3.1, CogvideoX, HunyuanVideo, Wan, Flux, and Stable-Diffusion3.5 show negligible loss for the 8+8 variant and small loss for the 4+8 variant.
- The method is a drop-in change to the attention kernel: it keeps SageAttention2's Q/K smoothing, INT4/INT8 QK quantization, and FP8 PV quantization, altering only the accumulator type and the scale factors.
Reading between the lines
- The same $\Pr \times V_r$ bound applies to any FP8 matmul with a 32-way FP16 accumulation, so the range-narrowing trick could generalize beyond attention to MLP or convolution paths whose activations have controlled ranges.
- Because the speedup depends on a hardware instruction ratio that may differ on other GPUs, the FP16-accumulator choice should be re-benchmarked per architecture; on chips without a fast FP16-accumulator FP8 path the method's advantage shrinks.
- The global choice $\Pr=224$, $V_r=4.5$ could be tuned per layer or per tensor instead, potentially recovering accuracy in outlier-heavy layers at a small speed cost.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes SageAttention2++, a modification of SageAttention2 that replaces the FP32-accumulator FP8 MMA instruction used in the P*V matmul with the FP16-accumulator instruction mma.f16.f8.f8.f16. To keep the FP16 accumulator in range, the authors narrow the FP8 quantization ranges of P and V by setting Pr=224 and Vr=4.5, and they introduce delayed FP32 buffering to reduce data-type conversion overhead. The paper reports up to a 3.9x speedup over FlashAttention2 on RTX4090 and RTX5090 and claims that attention accuracy matches SageAttention2 with negligible end-to-end metrics loss across language, image, and video generation models.
Significance. If the underlying instruction-throughput assumption is correct, this is a useful and clean engineering contribution: the range-bound derivation in Eq. (3) is straightforward, and the speed benchmark covers two GPUs, two head dimensions, causal and non-causal settings, and a wide range of sequence lengths. The end-to-end evaluation spans six generation models, which is a reasonable breadth for an implementation paper. The main significance is identifying a practical way to exploit FP16-accumulator FP8 tensor-core instructions. However, the accuracy evidence is not strong enough to support the paper's global claim, and the key hardware speed ratio is not measured in the paper, so the central claims are not yet fully established.
major comments (4)
- [Sec. 4.3, Table 3] The end-to-end results in the HunyuanVideo row contradict the abstract's claim of 'maintaining the same attention accuracy as SageAttention2' and 'negligible' loss. For SageAttn2++(8+8), VQA-t drops from 54.878 (SageAttn2 8+8) to 51.080, and for SageAttn2++(4+8) from 55.141 to 52.258; the CogvideoX row also shows a drop for 8+8 from 74.447 to 73.165. Since no seeds or error bars are reported, the paper needs to show that these differences are within run-to-run variation, or it must qualify the accuracy claim. As written, the global claim is not supported by the paper's own data.
- [Sec. 3.1, Table 2] The attention-accuracy evidence is limited to the average CosSim/L1 over all layers of a single model, CogvideoX. Narrowing Vr from 448 to 4.5 reduces the per-channel quantization range by roughly a factor of 100, and small V entries are likely to be flushed to zero, but an average over layers can mask such underflow. The paper should report per-layer or per-channel error statistics, and ideally attention accuracy for HunyuanVideo and Wan, whose end-to-end metrics in Table 3 show the largest degradation.
- [Sec. 2.2, Table 1] The claimed 2x advantage of mma.f16.f8.f8.f16 over mma.f32.f8.f8.f32 is taken from NVIDIA documentation and is not benchmarked in the paper. The 3.9x end-to-end speedup in Figures 1-4 is directly contingent on this ratio, so the paper should include a microbenchmark of the two instructions on RTX4090 and RTX5090. Without such a measurement, the central performance claim is not fully verified.
- [Sec. 4.2, Figures 1-4] The paper consistently refers to 'SageAttention2++' achieving a 3.9x speedup, but Figures 1-4 show that this value corresponds to the SageAttn2++(4+8) variant, while SageAttn2++(8+8) reaches about 3.0x. The abstract and conclusion should either state the variant explicitly or report the range of speedups, since the two variants differ substantially.
minor comments (5)
- [Table 2] All CosSim values round to 99.97 and all L1 values to 0.01862, so the table cannot discriminate among the tested (Pr, Vr) configurations; reporting more significant digits or per-block error statistics would better support the 'negligible error' conclusion.
- [Sec. 3.2] The claimed benefit of delayed FP32 buffering is that it halves the data-type conversion overhead, but no ablation is provided. A comparison with and without delayed buffering would make the contribution of this design choice concrete.
- [Sec. 4.1] The kernel speed measurement methodology is not described: the paper does not state the number of runs, whether warm-up was used, or how TOPS are computed. Adding this information would improve reproducibility of the speed claims.
- [Fig. 5, 6, 7] The visible examples are presented without quantitative or qualitative analysis in the main text, so they do not by themselves support the accuracy claims; a sentence explaining what the figures demonstrate would help.
- [References] The SageAttention2 reference appears both as Zhang et al. 2025a and Zhang et al. 2025b with overlapping titles; this should be cleaned up.
Circularity Check
No circularity: the FP16-accumulator range constraint is derived from first principles, and the accuracy and speedup claims are validated empirically against independent baselines.
full rationale
SageAttention2++'s derivation chain is self-contained. The core constraint Pr*Vr <= 2047/2 (Eq. 3) is derived arithmetically from the FP16 maximum (65504), the 32-element dot-product accumulation per mma.m16n8k32, and the two-accumulator delay before FP32 conversion; it is not fitted to accuracy results. The choice Pr=224, Vr=4.5 is an engineering selection validated by the measured attention CosSim/L1 in Table 2, which also reports SageAttention2 under identical metrics rather than assuming equivalence by construction. The 3.9x speedup claim is supported by the authors' own kernel benchmarks (Figs. 1-4) against FlashAttention2, with the underlying instruction throughput ratio taken from external NVIDIA documentation. End-to-end accuracy claims are established by metrics in Table 3 comparing SageAttention2++ with SageAttention2 and full precision. No fitted parameter is renamed as a prediction, no load-bearing argument reduces to a self-citation, and no uniqueness claim is imported from prior work. The HunyuanVideo VQA-t drop noted by a skeptic is a correctness/robustness concern about dynamic range, not circularity: attention-level CosSim remains 99.97%, and the paper's claim of 'same attention accuracy as SageAttention2' refers to that measured quantity.
Assumptions & free parameters
free parameters (2)
- Pr =
224
- Vr =
4.5
assumptions (5)
- domain assumption The mma.f16.f8.f8.f16 instruction is 2x faster than mma.f32.f8.f8.f32 on RTX4090 and RTX5090.
- standard math FP16 representable range is [-65504, 65504].
- standard math For mma.m16n8k32, 32 product terms are accumulated; hence the sum is bounded by 32*|p|*|v|.
- standard math Accumulating two mma results in FP16 before converting to FP32 requires Pr*Vr <= 2047/2.
- domain assumption Narrowing V quantization to Vr=4.5 preserves accuracy across diverse text, image, and video models.
Cite this review
Pith. "Pith review of SageAttention2++: A More Efficient Implementation of SageAttention2." pith.science (2026). https://pith.science/paper/7NBX5D2Z
@misc{pith2026250521136,
author = {Pith},
title = {Pith review of: SageAttention2++: A More Efficient Implementation of SageAttention2},
year = {2026},
howpublished = {\url{https://pith.science/paper/7NBX5D2Z}},
note = {Machine review of arXiv:2505.21136}
}
read the original abstract
The efficiency of attention is critical because its time complexity grows quadratically with sequence length. SageAttention2 addresses this by utilizing quantization to accelerate matrix multiplications (Matmul) in attention. To further accelerate SageAttention2, we propose to utilize the faster instruction of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8 Matmul used in SageAttention2. Our experiments show that SageAttention2++ achieves a 3.9x speedup over FlashAttention while maintaining the same attention accuracy as SageAttention2. This means SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss. The code will be available at https://github.com/thu-ml/SageAttention.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 4 Pith papers
-
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
A closed-form MXFP4 scale boundary Qmax=7.25 plus pre-normalization softmax quantization recovers nearly all FP16 video quality lost by naive MXFP4 attention.
-
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
CoSA couples an order-aware sparse-mask proxy with a skip-capable attention kernel, achieving higher long-context accuracy at lower compute budgets than existing sparse-attention baselines.
-
Vidu S1: A Real-Time Interactive Video Generation Model
Vidu S1 generates voice-controlled interactive avatar video in real time at 540p/42 FPS with claimed infinite stable streams and top reported quality metrics.
-
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
A MILP scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment reports 20-41% cost-efficiency gains over homogeneous GPU clusters for LLM serving.
Reference graph
Works this paper leans on
-
[3]
K.-H., Cao, T., Yang, F., and Yang, M
Gao, Y ., Zeng, Z., Du, D., Cao, S., So, H. K.-H., Cao, T., Yang, F., and Yang, M. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276,
-
[4]
Clipscore: A reference-free evaluation metric for image captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y . Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 7514–7528,
work page 2021
-
[7]
Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pp. 740–
work page 2014
-
[8]
Milakov, M. and Gimelshein, N. Online normalizer cal- culation for softmax. arXiv preprint arXiv:1805.02867,
-
[11]
Wan: Open and advanced large-scale video generative models
Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, T., Gui, T., Weng, T., Shen, T., Lin, W., Wang, W., Wang, W., Zhou, W....
-
[12]
Z., Khabsa, M., Fang, H., and Ma, H
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768,
arXiv 2006
-
[14]
Infllm: Training-free long-context extrapolation for llms with an efficient context memory
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y ., Zhang, Z., Liu, Z., and Sun, M. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In First Workshop on Long-Context F oundation Models@ ICML 2024, 2024a. Xiao, G., Tian, Y ., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sink...
work page 2024
-
[15]
Gated delta net- works: Improving mamba2 with delta rule
7 SageAttention2++: A More Efficient Implementation of SageAttention2 Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta net- works: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464,
Show all 21 references
-
[16]
Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization
Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In Interna- tional Conference on Machine Learning (ICML) , 2025a. Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu...
-
[17]
Open-sora: Democratiz- ing efficient video production for all
Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y ., Li, T., and You, Y . Open-sora: Democratiz- ing efficient video production for all. arXiv preprint arXiv:2412.20404,
-
[18]
Text-to-video models are evaluated using the open-sora (Zheng et al.,
for contextual understanding, and Needle-in-A-Haystack (NIAH) task (Kamradt, 2023). Text-to-video models are evaluated using the open-sora (Zheng et al.,
2023
-
[19]
Text-to-image models are assessed on COCO annotations (Lin et al., 2014)
prompt sets. Text-to-image models are assessed on COCO annotations (Lin et al., 2014). End-to-end metrics. For text-to-text models, we use perplexity (Ppl.) (Jelinek et al.,
2014
-
[21]
For text-to-image models, generated images are compared with the images in three aspects: FID (Heusel et al.,
to measure the text-video alignment; VQA-a and VQA-t to assess the video aesthetic and technical quality, respectively; and Flow-score (FScore) for temporal consistency (Wu et al., 2023). For text-to-image models, generated images are compared with the images in three aspects:...
2023
-
[1977]
For text-to-video models, following Zhao et al
for WikiText, accuracy (Acc.) for LAMBADA and NIAH. For text-to-video models, following Zhao et al. (2025), we evaluate the quality of generated videos on five metrics: CLIPSIM and CLIP-Temp (CLIP-T) (Liu et al.,
2025
-
[2016]
Lightning attention-2: A free lunch for handling unlim- ited sequence lengths in large language models
Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y . Lightning attention-2: A free lunch for handling unlim- ited sequence lengths in large language models. arXiv preprint arXiv:2401.04658,
-
[2017]
Identifying sensitive weights via post- quantization integral
Hu, Y ., Huang, W., Liang, Z., Chen, C., Zhang, J., Zhu, J., and Chen, J. Identifying sensitive weights via post- quantization integral. arXiv preprint arXiv:2503.01901,
-
[2020]
Hunyuanvideo: A systematic framework for large video generative models
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., Wu, K., Lin, Q., Wang, A., Wang, A., Li, C., Huang, D., Yang, F., Tan, H., Wang, H., Song, J., Bai, J., Wu, J., Xue, J., Wang, J., Yuan, J., Wang, K., Liu, M., Li, P., Li, S., Wang...
-
[2022]
The llama 3 herd of models
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[2023]
Sparse videogen: Acceler- ating video diffusion transformers with spatial-temporal sparsity
Xi, H., Yang, S., Zhao, Y ., Xu, C., Li, M., Li, X., Lin, Y ., Cai, H., Zhang, J., Li, D., et al. Sparse videogen: Acceler- ating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776,
-
[2024]
Moa: Mixture of sparse attention for automatic large language model compression
Fu, T., Huang, H., Ning, X., Zhang, G., Chen, B., Wu, T., Wang, H., Huang, Z., Li, S., Yan, S., Dai, G., Yang, H., and Wang, Y . Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406.14909,
-
[2025]
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R
Accessed: 2025-05-16. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Asso...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.