{"id":"d3011ebe-6855-45f1-8ff8-1cf655eb2398","arxiv_id":"2505.21136","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using the FP16-accumulator FP8 matmul instruction with narrowed quantization ranges for P and V yields up to 3.9x speedup over FlashAttention2 with negligible accuracy loss.","lead":"SageAttention2++ tweaks the P-times-V multiplication inside attention to use a faster half-precision accumulate instruction, then shrinks the quantization scale to keep numbers in range. It runs large language, image, and video models faster while keeping output quality close to the earlier SageAttention2.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Narrowing V's E4M3 range to Vr=4.5 cuts per-channel dynamic range about 100x, underflowing small V entries to zero; the accuracy claim rests on averaged CogvideoX CosSim even though HunyuanVideo VQA-t drops by 3.8 points in Table 3.","rationale":"The reader's weakest_assumption is that the 2x speedup of mma.f16.f8.f8.f16 over mma.f32.f8.f8.f32 is unverified. I partially disagree: the paper's own kernel benchmarks (Figs. 1–4) measure SageAttention2++ end-to-end against FlashAttention2 and SageAttention2, so the speedup claim is empirically supported even if the instruction-level throughput is not isolated in a microbenchmark. The more load-bearing risk is the accuracy half of the central claim. Narrowing Vr from 448 to 4.5 is not accuracy-neutral by construction: per-channel scaling plus E4M3's fixed exponent range reduces representable dynamic range by ~100x, zeroing small V entries. Table 2 only reports average CosSim on one model, and Table 3 already shows a 3.8-point VQA-t drop for HunyuanVideo. This is a concrete, internally grounded mechanism that could break the 'same attention accuracy' claim. I keep CONDITIONAL because the claim might still hold; my proposed test (underflow counts and per-layer error, plus seeded reruns) would settle it. Credit where due: the FP16 range derivation (Eqs. 1–3) is sound, the kernel speed results are substantive, and the comparisons span diverse models.","tokens_in":10367,"tokens_out":20273,"duration_ms":208287,"concrete_test":"Instrument the SageAttention2++ kernel to count, for every V channel on HunyuanVideo, the fraction of entries with |v| < colmax/2304 (the Vr=4.5 underflow threshold) and to log per-layer attention CosSim/L1 against SageAttention2. If any layer has an underflow fraction >0.1% or per-layer CosSim below 99.9%, the 'same attention accuracy' claim is falsified; if not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.1, SageAttention2++ reduces Vr from 448 (SageAttention2) to 4.5 to satisfy the FP16 accumulator bound Pr*Vr<=2047/2 in Eq. 3. Because per-channel scaling normalizes each channel to [-Vr, Vr] and E4M3 has a fixed exponent range, this ~100x reduction shrinks the per-channel dynamic range from 448*2^9 (~229k) to 4.5*2^9 (~2.3k). Any V entry smaller than ~colmax/2304 within its channel now quantizes to zero. The paper only reports average attention CosSim over CogvideoX layers (Table 2), which can mask such underflow, and Table 3 shows HunyuanVideo VQA-t falling from 54.878 (SageAttn2 8+8) to 51.080 (SageAttn2++ 8+8). That 3.8-point drop is inconsistent with the abstract's 'maintaining the same attention accuracy as SageAttention2' and 'negligible' loss claims. The accuracy half of the central claim is therefore not established for models with wide per-channel V dynamic range.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SageAttention2++, a modification of SageAttention2 that replaces the FP32-accumulator FP8 MMA instruction used in the P*V matmul with the FP16-accumulator instruction mma.f16.f8.f8.f16. To keep the FP16 accumulator in range, the authors narrow the FP8 quantization ranges of P and V by setting Pr=224 and Vr=4.5, and they introduce delayed FP32 buffering to reduce data-type conversion overhead. The paper reports up to a 3.9x speedup over FlashAttention2 on RTX4090 and RTX5090 and claims that attention accuracy matches SageAttention2 with negligible end-to-end metrics loss across language, image, and video generation models.","tokens_in":10658,"tokens_out":7591,"duration_ms":82828,"significance":"If the underlying instruction-throughput assumption is correct, this is a useful and clean engineering contribution: the range-bound derivation in Eq. (3) is straightforward, and the speed benchmark covers two GPUs, two head dimensions, causal and non-causal settings, and a wide range of sequence lengths. The end-to-end evaluation spans six generation models, which is a reasonable breadth for an implementation paper. The main significance is identifying a practical way to exploit FP16-accumulator FP8 tensor-core instructions. However, the accuracy evidence is not strong enough to support the paper's global claim, and the key hardware speed ratio is not measured in the paper, so the central claims are not yet fully established.","major_comments":[{"comment":"The end-to-end results in the HunyuanVideo row contradict the abstract's claim of 'maintaining the same attention accuracy as SageAttention2' and 'negligible' loss. For SageAttn2++(8+8), VQA-t drops from 54.878 (SageAttn2 8+8) to 51.080, and for SageAttn2++(4+8) from 55.141 to 52.258; the CogvideoX row also shows a drop for 8+8 from 74.447 to 73.165. Since no seeds or error bars are reported, the paper needs to show that these differences are within run-to-run variation, or it must qualify the accuracy claim. As written, the global claim is not supported by the paper's own data.","section":"Sec. 4.3, Table 3"},{"comment":"The attention-accuracy evidence is limited to the average CosSim/L1 over all layers of a single model, CogvideoX. Narrowing Vr from 448 to 4.5 reduces the per-channel quantization range by roughly a factor of 100, and small V entries are likely to be flushed to zero, but an average over layers can mask such underflow. The paper should report per-layer or per-channel error statistics, and ideally attention accuracy for HunyuanVideo and Wan, whose end-to-end metrics in Table 3 show the largest degradation.","section":"Sec. 3.1, Table 2"},{"comment":"The claimed 2x advantage of mma.f16.f8.f8.f16 over mma.f32.f8.f8.f32 is taken from NVIDIA documentation and is not benchmarked in the paper. The 3.9x end-to-end speedup in Figures 1-4 is directly contingent on this ratio, so the paper should include a microbenchmark of the two instructions on RTX4090 and RTX5090. Without such a measurement, the central performance claim is not fully verified.","section":"Sec. 2.2, Table 1"},{"comment":"The paper consistently refers to 'SageAttention2++' achieving a 3.9x speedup, but Figures 1-4 show that this value corresponds to the SageAttn2++(4+8) variant, while SageAttn2++(8+8) reaches about 3.0x. The abstract and conclusion should either state the variant explicitly or report the range of speedups, since the two variants differ substantially.","section":"Sec. 4.2, Figures 1-4"}],"minor_comments":[{"comment":"All CosSim values round to 99.97 and all L1 values to 0.01862, so the table cannot discriminate among the tested (Pr, Vr) configurations; reporting more significant digits or per-block error statistics would better support the 'negligible error' conclusion.","section":"Table 2"},{"comment":"The claimed benefit of delayed FP32 buffering is that it halves the data-type conversion overhead, but no ablation is provided. A comparison with and without delayed buffering would make the contribution of this design choice concrete.","section":"Sec. 3.2"},{"comment":"The kernel speed measurement methodology is not described: the paper does not state the number of runs, whether warm-up was used, or how TOPS are computed. Adding this information would improve reproducibility of the speed claims.","section":"Sec. 4.1"},{"comment":"The visible examples are presented without quantitative or qualitative analysis in the main text, so they do not by themselves support the accuracy claims; a sentence explaining what the figures demonstrate would help.","section":"Fig. 5, 6, 7"},{"comment":"The SageAttention2 reference appears both as Zhang et al. 2025a and Zhang et al. 2025b with overlapping titles; this should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is quite short for a journal submission, and the accuracy evidence is thin relative to the strength of the claims in the abstract. The authors should be encouraged to add either error-bar experiments or a clear qualification of the accuracy claim, and to include a microbenchmark of the instruction throughput. The range-bound derivation is sound and the idea is simple and potentially useful, so the paper is worth another round if these load-bearing points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean incremental optimization on top of SageAttention2, not a new capability. The idea is to use the FP16-accumulator FP8 MMA for the P*V matmul and shrink the quantization ranges so the accumulator doesn't overflow. The bound Pr*Vr <= 2047/2 is correct, and the speedup is actually measured in their kernels, not inferred from the instruction spec. On RTX4090/5090, their (4+8) variant hits about 3.9x over FlashAttention2, a useful constant-factor win for consumer-GPU inference.\n\nWhat the paper does well: the range-narrowing derivation is simple and sound; the kernel benchmarking covers two head dimensions, causal and non-causal, and sequence lengths up to 32K; the end-to-end evaluation spans six models across text, image, and video generation. Comparing against SageAttention2 is the right baseline, since the goal is to make it faster without losing accuracy. The citation pattern is fine, as they build directly on their own prior work.\n\nWhere it gets soft: the accuracy claim is stronger than the evidence. Table 2 reports attention CosSim on only one model (CogvideoX), and the end-to-end numbers have no error bars. HunyuanVideo's VQA-t drops from 54.88 (SageAttn2 8+8) to 51.08 (SageAttn2++ 8+8), a real gap that is not \"negligible\" on that metric. The stress-test's underflow mechanism, shrinking Vr from 448 to 4.5 and zeroing small per-channel entries, is plausible for models with wide per-channel dynamic ranges, though the evidence is mixed because other Hunyuan metrics improve and the (4+8) variant's metrics often match SageAttention2 exactly. Without repeated runs or error bars, I can't tell if the VQA-t drop is systematic or noise. The fix is easy: report attention accuracy on more models, include variance, and drop the word \"negligible.\"\n\nThe other soft spot is that code is not actually released yet. For an implementation paper, that matters; a referee should be able to run the kernels and check the measured speedups.\n\nBottom line: an honest, engineering-focused contribution with a sound core idea. It deserves peer review if the code is released and the accuracy claims are qualified. This is the kind of paper a systems-for-ML venue should accept with revisions rather than desk reject, but I would want the code and error bars before trusting either the 3.9x or the \"same accuracy\" claim.","headline":"A sound, incremental attention-kernel optimization with a real speedup, but the accuracy claims need more evidence and softer wording.","tokens_in":11194,"tokens_out":5912,"would_cite":false,"duration_ms":66238,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SageAttention2++ claims a 3.9x speedup over FlashAttention by switching the P V matmul to an FP8 instruction with an FP16 accumulator, while keeping SageAttention2's accuracy through narrower quantization ranges.","keywords":["attention acceleration","FP8 quantization","FP16 accumulator","tensor cores","FlashAttention","SageAttention2","quantization range","inference acceleration"],"falsifier":"Isolate mma.f16.f8.f8.f16 and mma.f32.f8.f8.f32 on RTX4090 and measure their throughput ratio; if the FP16-accumulator instruction is not about twice as fast, the paper's claimed 3.9x end-to-end speedup over FlashAttention2 will not hold.","tokens_in":10198,"feed_emoji":"⚡","tokens_out":8254,"duration_ms":67471,"temperature":0.7,"pith_summary":"SageAttention2++ extends SageAttention2 by changing the second attention matmul, the $P V$ product, from an FP8 tensor-core instruction with an FP32 accumulator to the faster FP8 instruction with an FP16 accumulator, which the paper takes as 2x faster on RTX4090 and RTX5090. To keep the FP16 accumulator from overflowing, it narrows the FP8 quantization ranges of $P$ and $V$ so that $\\Pr \\times V_r \\leq 2047/2$, and it delays half of the FP16-to-FP32 conversions to cut overhead. The paper reports that this preserves SageAttention2's attention accuracy while delivering up to 3.9x speedup over FlashAttention2 on language, image, and video generation models. If the claim holds, low-bit attention can run much closer to tensor-core throughput without giving up the accuracy that SageAttention2 achieved.","feed_headline":"FP8 matmul with FP16 accumulator speeds attention 3.9x","feed_subtitle":"SageAttention2++ keeps SageAttention2's accuracy by narrowing quantization ranges so the FP16 accumulator never overflows.","key_machinery":"The load-bearing mechanism is the hardware instruction mma.m16n8k32 with an FP16 accumulator, taken from PTX documentation as 2x faster than the FP32-accumulator FP8 instruction. The paper couples this with a quantization-range constraint: per-block scales $\\delta_P = |\\max(\\tilde P)|/\\Pr$ and per-channel scales $\\delta_V = |\\max(V)|/V_r$ with $\\Pr \\times V_r \\leq 2047/2$ keep the FP16 accumulation from overflowing. A second mechanism, delayed FP32 buffering, accumulates two consecutive mma results in FP16 before converting to FP32, halving the number of data-type conversion instructions.","core_discovery":"The central claim is that the attention output $P V$ can be computed with FP8 inputs and an FP16 accumulator, using the instruction mma.f16.f8.f8.f16, without losing SageAttention2's accuracy, provided the quantization scales are adjusted so that the worst-case accumulated product stays inside FP16's range. The paper's derivation bounds 32 accumulated products by $|32 \\times p v| \\leq 65504$, and after adding one extra FP16 accumulation before conversion the constraint becomes $\\Pr \\times V_r \\leq 2047/2$. The paper chooses $\\Pr = 224$ and $V_r = 4.5$, reports attention cosine similarity of 99.97%, and shows end-to-end metrics on Llama3.1, CogvideoX, HunyuanVideo, Wan, Flux, and Stable-Diffusion3.5 that match SageAttention2. The result is a claimed 3.9x speedup over FlashAttention2 for the INT4 Q/K variant and about 3.0x for the INT8 variant.","pith_inferences":["The same $\\Pr \\times V_r$ bound applies to any FP8 matmul with a 32-way FP16 accumulation, so the range-narrowing trick could generalize beyond attention to MLP or convolution paths whose activations have controlled ranges.","Because the speedup depends on a hardware instruction ratio that may differ on other GPUs, the FP16-accumulator choice should be re-benchmarked per architecture; on chips without a fast FP16-accumulator FP8 path the method's advantage shrinks.","The global choice $\\Pr=224$, $V_r=4.5$ could be tuned per layer or per tensor instead, potentially recovering accuracy in outlier-heavy layers at a small speed cost."],"forward_implications":["On RTX4090 and RTX5090, SageAttention2++(4+8) reaches about 3.9x and SageAttention2++(8+8) about 3.0x the speed of FlashAttention2 across sequence lengths from 1K to 32K.","Attention accuracy, measured by cosine similarity and L1 error, stays at the same level as SageAttention2 when the quantization ranges are narrowed to $\\Pr=224$, $V_r=4.5$.","End-to-end metrics on Llama3.1, CogvideoX, HunyuanVideo, Wan, Flux, and Stable-Diffusion3.5 show negligible loss for the 8+8 variant and small loss for the 4+8 variant.","The method is a drop-in change to the attention kernel: it keeps SageAttention2's Q/K smoothing, INT4/INT8 QK quantization, and FP8 PV quantization, altering only the accumulator type and the scale factors."],"supporting_citations":[{"why":"supplies SageAttention2, the quantization scheme and accuracy baseline that this paper extends and must match.","marker":"[Zhang et al., 2025a]"},{"why":"provides FlashAttention's tiling and IO-aware attention structure that SageAttention2 and the new kernel build on.","marker":"[Dao et al., 2022]"},{"why":"FlashAttention2 is the baseline over which the 3.9x speedup is measured.","marker":"[Dao, 2024]"},{"why":"source of the claimed instruction speedups in Table 1: FP8 with FP16 accumulator at 4x over FP16.","marker":"[NVIDIA, 2022]"},{"why":"documents the mma.m16n8k32 instruction and the FP16-to-FP32 conversion overhead that motivates delayed FP32 buffering.","marker":"[NVIDIA, 2025]"},{"why":"online softmax is the underlying algorithm for tiled attention that the kernel reuses.","marker":"[Milakov & Gimelshein, 2018]"}],"fun_headline_variants":["FP8 matmul + FP16 accumulator gives 3.9x attention speedup","3.9x faster attention using FP8 matmul and FP16 accumulator","FP16 accumulator enables FP8 matmul for 3.9x faster attention","SageAttention2++: FP8 matmul with FP16 accumulator for 3.9x boost","FP8 matmul with FP16 accumulator accelerates attention 3.9x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire speedup rests on the unmeasured hardware claim that the FP8 instruction with an FP16 accumulator is exactly 2x faster than the FP8 instruction with an FP32 accumulator on RTX4090 and RTX5090, a figure the paper takes from NVIDIA documentation; if the real instruction throughput gap is smaller, the end-to-end speedup shrinks proportionally.","fun_headline_variants_meta":{"raw":{"variants":["FP8 matmul + FP16 accumulator gives 3.9x attention speedup","3.9x faster attention using FP8 matmul and FP16 accumulator","FP16 accumulator enables FP8 matmul for 3.9x faster attention","SageAttention2++: FP8 matmul with FP16 accumulator for 3.9x boost","FP8 matmul with FP16 accumulator accelerates attention 3.9x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3618,"prompt_tokens":899,"completion_tokens":2719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2609}},"tokens_in":515,"tokens_out":2719,"duration_ms":19431,"temperature":1.0,"reasoning_tokens":2609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:33:55.135746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Isolate mma.f16.f8.f8.f16 and mma.f32.f8.f8.f32 on RTX4090 and measure their throughput ratio; if the FP16-accumulator instruction is not about twice as fast, the paper's claimed 3.9x end-to-end speedup over FlashAttention2 will not hold.","supporting_citations":[],"review_version":1}