{"id":"f1eeb53f-402e-47ca-814f-abbafcf5a09d","arxiv_id":"2607.02584","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"RoPE-aware orthogonal rotations plus full-range INT4 quantization of attention probabilities deliver mixed-precision FlashAttention that matches FP16 video quality at 1.5–1.68× end-to-end speedup.","lead":"RotateAttention makes INT4 FlashAttention work for DiT video models that use 3D RoPE by rotating Q and K in a RoPE-compatible way and using the full signed INT4 range for the non-negative attention matrix P. The result is up to 1.68× faster end-to-end video generation with quality close to FP16 on leading open models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection beyond the reader's already-identified schedule-transfer risk; the central claim holds under the paper's stated scope.","rationale":"The paper is a solid systems contribution that correctly diagnoses the RoPE–rotation incompatibility and supplies two low-overhead orthogonal fixes plus a simple full-range P map. Experiments on two leading models with ablations support the reported speedups and quality. The only material caveat is the hand-tuned mixed-precision schedule that underpins the quality claim; the reader already flags this, and no stronger mathematical or experimental flaw is present. Code release and one additional model family would convert CONDITIONAL to ACCEPT, but that does not alter the current verdict.","tokens_in":14713,"tokens_out":479,"duration_ms":6099,"concrete_test":"On a third 3D-RoPE DiT (e.g., CogVideoX or Open-Sora 2.0) apply the exact same fixed schedule fractions used for Wan2.2/HunyuanVideo without retuning; if Cosine Similarity falls below ~0.94 or visual artifacts become obvious relative to FP16, the transferability premise fails and the 'nearly identical' claim must be qualified as schedule-tuned.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (hand-chosen FP16 fallback schedule of first/last steps and blocks, 68–81% INT4) is the genuine soft spot for the 'nearly identical quality' claim. Table 1 and Figs. 4–5 already show residual pixel-level gaps (e.g., Wan2.2-T2V Cosine 0.956 vs 1.0, PSNR ~18.5), so the claim is schedule-dependent rather than pure INT4. No deeper internal inconsistency appears: the orthogonality argument (§4.1, SVD analysis) correctly motivates why non-orthogonal LLM-style rotations fail for simultaneous Q/K INT4, the RoPE-aware 2×2 designs match the observed segmental incoherence (Fig. 2), and range-optimized P is a low-risk affine map. The paper itself flags limited model coverage and schedule hand-tuning in Limitations. Thus the load-bearing concern is already correctly identified; nothing stronger overturns the systems contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes RotateAttention, a mixed-precision INT4 FlashAttention framework for DiT-based video generation models that use 3D RoPE. It identifies that 3D RoPE induces segmented, symmetric outlier patterns in Q and K (Fig. 2, Sec. 3), and addresses two obstacles to low-bit attention: (1) RoPE-aware orthogonal rotations (Interleaved, mergeable into RoPE; Half, low-overhead pairing across half-segments) that equalize outliers without unbalancing simultaneous Q/K quantization (Sec. 4.1, SVD argument for orthogonality); and (2) range-optimized affine quantization of the non-negative P matrix that uses the full INT4 range via fixed scale 15 and zero-point -8 (Sec. 4.2). A standard offline Hadamard is fused into V. With selective FP16 fallback on early/late denoising steps and DiT blocks (68-81% INT4), experiments on Wan2.2 (I2V/T2V) and HunyuanVideo report near-baseline relative metrics (Table 1), visual fidelity superior to SageAttention (Figs. 4-5), 2.2\times kernel speedup, and up to 1.68\times end-to-end speedup.","tokens_in":14989,"tokens_out":1276,"duration_ms":23107,"significance":"If the results hold under the stated scope, this is a solid systems contribution for accelerating compute-bound 3D-RoPE video DiTs, orthogonal to sparse attention. The RoPE-induced outlier analysis and the clean necessity-of-orthogonality argument (Sec. 4.1) usefully explain why LLM-style unconstrained or non-orthogonal rotations do not transfer when both Q and K must be quantized. Zero-overhead mergeable designs and the simple fixed-scale P map are practical. Strengths include component ablations (Table 1), visual comparisons, and explicit Limitations acknowledging model coverage and schedule hand-tuning. The work is incremental relative to SageAttention but well-targeted and reproducible in spirit.","major_comments":[{"comment":"The central quality claim (“nearly identical to full-precision baselines”) is schedule-dependent. §5.1 hand-selects FP16 for the first/last few sampling steps and first/last DiT blocks (yielding 68–81 % INT4). Table 1 still shows non-negligible residual gaps (Wan2.2-T2V best Cosine 0.956, SSIM ~0.66–0.67, PSNR ~18.5; I2V Cosine ~0.98). Without a sensitivity study on the number of fallback steps/blocks, or an automatic selection rule, it is unclear how much of the “nearly identical” result is due to the rotations/P map versus the FP16 safety net, and whether the schedule transfers to other 3D-RoPE DiTs without retuning. This is the load-bearing soft spot already flagged in Limitations; a short ablation or transfer experiment would strengthen the claim.","section":"§5.1, Table 1, Limitations"},{"comment":"Range-optimized P helps T2V but is neutral-to-harmful on Wan2.2-I2V (Table 1: Optim-P alone drops Cosine/SSIM/PSNR relative to plain INT4). The low-entropy explanation in §5.4 is plausible, yet the method still presents Optim-P as a core technique without a decision rule or per-model switch. For the claim that the full framework is robust, either disable Optim-P under strong conditioning or quantify when the affine map is beneficial.","section":"§4.2, §5.4, Table 1"}],"minor_comments":[{"comment":"Abstract and Eq. (11)–(12) write P = exp(QK − max(QK)); standard notation is QK⊤. Minor but repeated.","section":"Abstract, §4.2"},{"comment":"Fig. 2 caption and Sec. 3.1 claim “strong distributional symmetry” and “structured sparsity”; a quantitative correlation or half-segment energy ratio would make the observation sharper.","section":"Fig. 2, §3.1"},{"comment":"Learned-rotation experiment (Sec. 4.1, Fig. 6) uses only 1–2 calibration samples and is labeled preliminary; either move fully to appendix or report variance over more seeds/samples so readers can judge stability.","section":"§4.1, §5.3, Fig. 6"},{"comment":"VBench is deferred to supplementary and described as less discriminative; a one-sentence summary of absolute scores in the main text would help readers who care about semantic/temporal metrics.","section":"§5.2"},{"comment":"Practical recommendation in §5.4 prefers Half Rotation by default; a short decision tree (I2V vs T2V, mergeability constraints) would make the guidance more actionable.","section":"§5.4"},{"comment":"Typos / polish: “DiT-basedvideogenerationmodels” spacing artifacts appear in several places; “incoherencemetric” etc. Clean for camera-ready.","section":"Throughout"}],"recommendation":"minor_revision","confidential_remarks":"Fit is good for a systems/CV venue that values practical acceleration of video DiTs. Novelty is incremental on the SageAttention line but the RoPE-specific analysis and orthogonality argument are clean contributions. No citation or authorship concerns. The schedule dependence is the only real risk to over-claiming; once addressed with a short ablation the paper is ready."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit is the observation that 3-D RoPE carves Q/K into three segments with half-segment outlier concentration and Q–K symmetry, then the two cheap orthogonal fixes that respect that structure: Interleaved (2×2 blocks mergeable into RoPE at zero cost) and Half (pair across half-segments, element-wise). Range-optimized P (fixed scale 15, zero-point –8) is almost trivial but correctly doubles the used INT4 levels for non-negative probabilities. Together they beat SageAttention on the reported metrics and give 2.2× kernel / up to 1.68× end-to-end on A10/H20 for the two leading open 3-D-RoPE DiTs.\n\nWhat they do well: the SVD argument in §4.1 that non-orthogonal transforms unbalance simultaneous Q/K INT4 is clean and explains why LLM-style unconstrained rotations fail here (Fig. 6). Ablations isolate rotation type, Optim-P, and learned-vs-fixed (Table 1). Hadamard-on-V is correctly treated as free offline fusion. They flag their own limited model coverage and hand-tuned schedule in Limitations. Citations to SageAttention, QuaRot, SpinQuant, FlatQuant, PARO are accurate; they correctly note PARO is complementary token reordering.\n\nSoft spots, in proportion: the “nearly identical” claim rests on a hand-chosen mixed-precision schedule (FP16 first/last steps and blocks, 68–81 % INT4). Table 1 still shows residual gaps (Wan2.2-T2V cosine 0.956, PSNR ~18.5). That schedule is the real transfer risk; pure INT4 would look worse. No code/kernels released, so the 2.2× number is hard to verify. Optim-P helps T2V more than I2V (low-entropy P under image conditioning). None of this is load-bearing math error; the systems contribution holds under the stated scope.\n\nThis is for people shipping or accelerating open video DiTs on commodity GPUs. Worth a serious referee; I would bring it to reading group and cite the rotation designs if I work on quantized video attention.","headline":"Solid systems paper: RoPE-aware orthogonal rotations plus full-range INT4 P give real 1.5–1.7× end-to-end gains on Wan2.2/HunyuanVideo; quality claim is schedule-dependent, not pure INT4.","tokens_in":15603,"tokens_out":571,"would_cite":true,"duration_ms":5814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"RotateAttention makes INT4 FlashAttention work for 3D-RoPE video DiTs by fixing RoPE-induced Q/K outliers and reclaiming the unused half of the 4-bit range for the attention matrix P.","keywords":["video generation","quantized attention","3D RoPE","INT4 FlashAttention","DiT","outlier rotation","mixed-precision"],"falsifier":"Run the same INT4 kernels with the identical mixed-precision schedule on a third independent 3-D-RoPE video DiT (for example CogVideoX or Open-Sora 2.0) and measure whether relative Cosine/SSIM/PSNR to the FP16 baseline stay within the ranges reported for Wan2.2 and HunyuanVideo; a clear drop would falsify the transfer claim.","tokens_in":15625,"feed_emoji":"🎬","tokens_out":942,"duration_ms":10749,"temperature":0.7,"pith_summary":"Video DiTs with 3D RoPE spend most of their compute on attention over tens of thousands of tokens. Quantizing FlashAttention to INT4 would help, but two obstacles remain: common rotation tricks that tame Q and K outliers do not play well with RoPE, and the non-negative probability matrix P wastes half of every 4-bit code under symmetric quantization. The authors show that 3D RoPE partitions feature dimensions into frame, height and width segments and produces characteristic, symmetric outlier patterns inside those segments. They exploit the patterns with two lightweight, RoPE-compatible orthogonal rotations (one mergeable into RoPE itself, one almost free) and replace symmetric P quantization by a fixed-scale affine map that uses the full signed INT4 interval. With a selective FP16 fallback on the first and last few denoising steps and DiT blocks, the resulting mixed-precision kernel recovers essentially full-precision video quality while delivering up to 1.68\times end-to-end and 2.2\times kernel speedups.","feed_headline":"INT4 attention for video DiTs without quality loss","feed_subtitle":"RoPE-aware rotations and full-range P quantization yield up to 1.68× end-to-end speedup","key_machinery":"RoPE-aware Rotation: a block-diagonal matrix of 2\times2 orthogonal blocks (Interleaved, mergeable into RoPE; or Half, pairing opposite half-segments) that redistributes RoPE-induced outliers while remaining sparse enough for element-wise execution, paired with Range-optimized P Quantization that maps [0,1] probabilities onto the full INT4 interval [−8,7] via fixed scale 15 and zero-point −8.","core_discovery":"The outlier structure of Q and K in 3D-RoPE video DiTs is strictly segmented by the three RoPE partitions and is highly symmetric between Q and K; orthogonal 2\times2 block rotations that respect this structure (Interleaved or Half) equalize those outliers without breaking RoPE or requiring expensive online transforms, while a fixed affine map that sends normalized P onto the full signed INT4 range doubles quantization resolution for the attention matrix.","pith_inferences":["Because the rotations are orthogonal and sparse, they may also stabilize INT8 or micro-scaled FP4 attention kernels that still suffer from the same RoPE-induced outlier segments.","The same segmented-outlier diagnosis could guide rotation design for other multi-axis positional encodings beyond 3-D RoPE.","If the mixed-precision schedule can be made adaptive per-block rather than hand-fixed, the fraction of INT4 attention could rise above 80 % without quality loss."],"forward_implications":["INT4 FlashAttention becomes a practical drop-in acceleration path for the current generation of open 3-D-RoPE video DiTs.","Half Rotation can be recommended as the default zero-overhead rotation for new models; Interleaved Rotation can be fused offline into RoPE weights for free.","Range-optimized P quantization should be enabled for moderate-to-high-entropy attention and can be disabled for strongly conditioned low-entropy patterns such as I2V.","The same RoPE-aware rotations and P rectification can be composed with complementary token-reordering methods for further accuracy gains."],"fun_headline_variants":["RoPE-aware rotations enable INT4 attention for video DiTs","INT4 FlashAttention for 3D-RoPE DiTs with full-range P maps","RotateAttention: RoPE-respecting INT4 for video generation","Segmented Q/K outliers fix enables 1.68× faster video DiTs","Fixed-scale INT4 P and RoPE-aware spins cut DiT attention cost"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The hand-chosen schedule that leaves the first and last few denoising steps and DiT blocks in FP16 is enough to keep generation quality acceptable and will transfer to other 3-D-RoPE video DiTs without retuning.","fun_headline_variants_meta":{"raw":{"variants":["RoPE-aware rotations enable INT4 attention for video DiTs","INT4 FlashAttention for 3D-RoPE DiTs with full-range P maps","RotateAttention: RoPE-respecting INT4 for video generation","Segmented Q/K outliers fix enables 1.68× faster video DiTs","Fixed-scale INT4 P and RoPE-aware spins cut DiT attention cost"]},"model":"grok-4.5","effort":"low","cost_usd":0.006674,"raw_usage":{"total_tokens":1809,"prompt_tokens":949,"num_sources_used":0,"completion_tokens":107,"cost_in_usd_ticks":66740000,"prompt_tokens_details":{"text_tokens":949,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":753,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":949,"tokens_out":107,"duration_ms":7280,"temperature":1.0,"reasoning_tokens":753,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T16:52:31.576823+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same INT4 kernels with the identical mixed-precision schedule on a third independent 3-D-RoPE video DiT (for example CogVideoX or Open-Sora 2.0) and measure whether relative Cosine/SSIM/PSNR to the FP16 baseline stay within the ranges reported for Wan2.2 and HunyuanVideo; a clear drop would falsify the transfer claim.","supporting_citations":[],"review_version":2}