{"id":"bb8a9a63-5878-44c2-b3b0-7c1c37ac12f6","arxiv_id":"2506.16054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.","lead":"A new method rearranges the order of image and video tokens before attention, turning scattered attention patterns into hardware-friendly blocks, then applies sparse and low-bit quantized attention. It reports near-full-precision generation quality with 20-30% density and INT8/INT4 quantization, and 1.9-2.7x end-to-end speedups on diffusion transformers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Static per-head permutation chosen from 1-2 prompts is the load-bearing premise; the paper's evidence for cross-prompt/timestep/CFG stability is qualitative, and a failure would invalidate the reported lossless-quality claim.","rationale":"The reader's weakest_assumption correctly identifies the static per-head permutation as the load-bearing premise. The paper's own evidence for permutation stability is largely qualitative: Sec. 4.1 asserts consistency without a quantitative table, and Appendix F shows visualizations with a note that detailed patterns vary over timesteps. The cosine-similarity statement in Sec. 4.2 is underspecified and does not directly establish that Eq. (3)'s argmin is invariant. If the permutation is not robust, the unified block-wise pattern dissolves, and the sparse masks plus block quantization lose their justification; the reported speedups then come at unquantified quality cost. This concern is more central than the 'lossless' wording issue, because correcting the wording would not change the mechanism, whereas a failed permutation would invalidate the mechanism itself. It is also distinct from the missing SparseVideoGen2 comparison, which affects contextual novelty more than the internal validity of the reported gains. I agree with the reader's conditional verdict: the paper warrants acceptance conditional on addressing this generalization risk, along with releasing artifacts and adding the missing baseline. No change to the reader's verdict is needed.","tokens_in":22379,"tokens_out":9403,"duration_ms":124661,"concrete_test":"Construct a held-out stress set of, e.g., 50 VBench prompts covering all subtasks plus prompts with strong camera motion and scene changes, and evaluate with 3 CFG scales including the unconditional branch, all disjoint from the 2 calibration prompts. For each attention head and each timestep, recompute the paper's Eq. (3) metric M(Theta) for all six permutations using the FP16 model's post-softmax attention maps, and record the fraction of head-timestep pairs where the calibration-time argmin permutation is not the held-out argmin. Then generate videos with (a) the original fixed permutation and (b) an oracle that re-selects each head's permutation per prompt/timestep while recomputing masks accordingly, both at 30% density.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of PAROAttention is that a per-head token permutation determined offline (Sec. 4.1) makes attention maps uniformly block-wise, so that static masks and block-wise INT8/INT4 PV quantization can be applied without quality loss. For this to work, the argmin of the permutation metric M in Eq. (3) must be stable across unseen prompts, all sampling timesteps, CFG branches, and the altered attention distribution induced by sparsity and quantization themselves. The paper asserts this stability in Sec. 4.1 ('We verify that the optimal permutation remains consistent across different timesteps and prompts') and Appendix F, but the quantitative evidence is thin: Sec. 4.2 reports a cosine similarity >= 0.99 across prompts for image-token attention without stating over how many prompts, heads, timesteps, or which similarity (raw attention maps vs. block-sparsity patterns vs. the M metric). Appendix F is primarily visual and says only that the 'type' of pattern remains consistent while 'detailed attention pattern may vary,' which is not the same as showing the selected permutation remains optimal. If the permutation order drifts for even a fraction of heads or timesteps, the offline-generated sparse blocks and the fused RoPE permutation misalign with actual attention, and the reported PSNR 22.89 at 30% density, PSNR 19.39 at 20% density, and PAROAttn(0.5+INT4) results are not representative of general use. This is a generalization gap, not a disagreement with consensus, and it is the weakest link between the measured numbers and the headline 'lossless metrics' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAROAttention, a post-training compression method for diffusion transformers in visual generation. It observes that visual attention maps have diverse, dispersed patterns that make block sparsity and PV quantization difficult, and proposes an offline per-head token permutation chosen from the six axis orders to unify these patterns into hardware-friendly block-wise forms. On top of this reordering, it applies static timestep-aware sparse masks and block-wise INT8/INT4 quantization, and implements fused CUDA kernels. Experiments on CogVideoX, Wan 2.1, and Flux.1.Dev report near-baseline quality at 20-50% density and INT8/INT4 bitwidth, with 1.9-2.7x end-to-end and up to 9.28x attention speedups. Appendices provide additional results on Wan, an ablation of SparseVideoGen's skipping scheme, overhead measurements, and visualizations of attention stability across timesteps and prompts.","tokens_in":22690,"tokens_out":5146,"duration_ms":56514,"significance":"If the stability premise holds, the contribution is significant: it converts irregular visual attention patterns into a simple block-wise form, making both static sparsity and block-wise quantization easier and more hardware-friendly, and the permutation idea is compatible with dynamic sparsity methods. The paper's strengths include the analysis of why visual attention patterns hinder sparsity and quantization, the combined sparse/quantization permutation metric in Eq. (3), detailed overhead measurements (permutation 0.03%, prefetch 0.33%), and consistent empirical trends across three models. However, the core generalization claim for offline-chosen permutations is supported mainly by qualitative or underspecified evidence, the 'lossless' wording overstates the reported FP-difference metrics, and the efficiency-accuracy trade-off is not accompanied by error bars or released artifacts. The idea is novel and likely useful, but the paper needs a quantitative stability analysis and more careful claims before acceptance.","major_comments":[{"comment":"The central premise that one offline-determined permutation per head remains optimal across unseen prompts, all timesteps, CFG branches, and the altered attention distribution is not quantitatively established. Sec. 4.2 reports cosine similarity >= 0.99 for image-token attention across prompts, but does not state the number of prompts, heads, or timesteps, nor whether the similarity is computed on raw attention maps, block-sparsity patterns, or the M metric of Eq. (3). Appendix F says only that the 'type' of pattern remains consistent while 'detailed attention pattern may vary,' which does not show that the argmin permutation is stable. If the optimal permutation drifts for even a fraction of heads or timesteps, the static masks and fused permutation misalign with actual attention, and the reported PSNR values in Tables 1 and 4 would not transfer. I request a quantitative analysis: for held-out prompts, report the fraction of heads and timesteps where the selected permutation is not the argmin of M, and the resulting quality drop when the static permutation is enforced.","section":"Sec. 4.1, Eq. (3); Appendix F"},{"comment":"The term 'lossless metrics' overstates the reported results. In Table 1, PAROAttn(0.3) has PSNR 22.89, SSIM 0.829, CosSim 0.984, and FVD-FP16 0.153 relative to the FP16 baseline, while PAROAttn(0.2) has PSNR 19.39, SSIM 0.744, and CosSim 0.962. These are substantial pixel-level differences, not lossless. The quality metrics (CLIPSIM, VQA) are nearly unchanged, but the abstract's 'lossless metrics' and Sec. 1's 'generation with lossless metrics' should be qualified as 'no significant degradation on quality metrics' with the FP-difference metrics reported explicitly and with error bars or multiple seeds.","section":"Abstract; Sec. 5.2, Tables 1 and 2"},{"comment":"The main comparison handicaps SparseVideoGen by disabling its skipping scheme. Appendix B shows that SparseVideoGen(0.5) with skipping achieves PSNR 25.37, while the main table reports PSNR 18.50 without skipping. Disabling skipping may be a legitimate fair-comparison choice given that PAROAttn applies to all timesteps, but the main text should present both configurations and explicitly justify the choice, because the claimed superiority over SparseVideoGen at 50% density partly reflects this ablation rather than the sparse-mask design itself. As written, a reader comparing Table 1 with the SparseVideoGen paper could draw a misleading conclusion about relative performance.","section":"Sec. 5.2, Table 1; Appendix B, Fig. 10"},{"comment":"Key hyperparameters are under-specified, preventing reproduction and robustness assessment. The paper gives example values for epsilon (1e-3) and sigma (90%) in Sec. 4.2, but does not report the exact values used for each model, the weighting coefficient alpha in Eq. (3), the block size b (stated only as 'chosen as 64'), or the precise timestep split for mask sharing (described only as 'first half'). The calibration set is the first 2 prompts of the CogVideo example dataset, and the evaluation uses a subset of VBench prompts and 1024 COCO prompts, with no error bars across prompts or seeds. Please provide a hyperparameter table and report variance across evaluation prompts.","section":"Sec. 4.1, Eq. (3); Sec. 5.1"},{"comment":"No code, kernels, masks, or permutation files are released, and the CUDA kernel description is high-level. Given that the central speedup claims depend on a custom fused permutation, prefetching, and block-skipping scheme, the absence of artifacts makes the efficiency numbers difficult to verify. At minimum, the per-head permutation orders and sparse masks for each model should be released, or the kernel pseudocode should be detailed enough to reimplement independently.","section":"Sec. 5.1; App. C"}],"minor_comments":[{"comment":"The word 'subpotimal' in the third paragraph should be 'suboptimal'.","section":"Sec. 1"},{"comment":"The notation n^{<epsilon}_{ij} is introduced without explaining the superscript; please clarify that this counts entries below epsilon and connect it to the block-sum threshold used for static mask generation.","section":"Eq. (1)"},{"comment":"LPIPS appears in the ablation table but is not defined in Sec. 5.1; please add a definition or citation for it.","section":"Table 3"},{"comment":"The omission of SpargeAttn results for Wan 2.1 due to NaN outputs should be noted in the main table or in a footnote to Table 4, not only in the appendix, so that readers do not misread the comparison as more complete than it is.","section":"Appendix D, Table 4"},{"comment":"The qualitative claims of 'nearly identical' frames would be strengthened by difference maps or per-image error metrics; the current visual comparisons are not quantified.","section":"Fig. 4 and Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical systems paper but appears to be an early preprint. The main risks are reproducibility (no code or artifacts) and the generalization of static permutations and masks; the requested quantitative stability analysis is necessary to justify the central claim. The self-citations to Vidit-Q and DiTFastAttn are appropriate given that the quantization and sparsity metrics build directly on those works. The paper fits the journal's scope as an attention-efficiency method for visual generation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper on a useful trick. Instead of designing sparse masks or quantization schemes to fit messy visual attention patterns, it permutes tokens so patterns become block-wise, then applies static block sparsity and INT8/INT4 PV quantization. The permutation is restricted to the six 3D axis orders and chosen offline using a metric that balances sparsity and quantization error. That combination is new enough, and the evaluation across CogVideoX, Wan, and Flux is consistent: PAROAttention clearly beats DiTFastAttn, MInference, SpargeAttn, and SparseVideoGen at matched density, and the ablations confirm the permutation is the active ingredient.\n\nWhat the paper does well: the experiments are careful, the design space is explored systematically, and the authors are transparent enough to include the SparseVideoGen skipping-scheme ablation in the appendix (PSNR 18.5 without skipping vs 25.4 with). The kernel overhead numbers (0.03% for permutation, 0.33% for prefetch) are plausible and the speedups approach theoretical limits.\n\nNow the soft spots, in rough order of severity.\n\nFirst, no comparison with SparseVideoGen2 [54], which already uses semantic-aware permutation for sparse video attention. The paper cites it but doesn't benchmark it. That's the most glaring omission, because it directly bears on novelty.\n\nSecond, the main-table comparison with SparseVideoGen uses the no-skip configuration. The authors do disclose this and provide the with-skip result in the appendix, but the main table presents the handicapped version as the headline comparison. It should be front and center.\n\nThird, “lossless metrics” is an overstatement. At 30% density, PSNR is 22.9 vs infinity for the FP16 baseline; at 20%, 19.4. Quality metrics like CLIPSIM and VQA are nearly identical, so maybe they mean “lossless on quality metrics,” but the relative-difference metrics clearly degrade. The wording should match what the numbers show.\n\nFourth, the load-bearing assumption that a per-head permutation chosen offline stays optimal across prompts, timesteps, and CFG settings is supported by one cosine similarity number (>=0.99) and visualizations in Appendix F. That may be true, and the held-out VBench results suggest it is, but the evidence is thin: they don't say how many prompts/heads/timesteps the similarity covers, or whether it's the block-sparsity pattern or the argmin of Eq. (3) that is stable. A short quantitative table would settle it.\n\nMinor: Appendix A contains a sentence that says PAROAttn at 0.5 density introduces noticeable degradation, which contradicts the main text. Probably a typo, but it should be fixed. No code or kernels are released, which matters for a systems paper.\n\nBottom line: this deserves a serious referee. The core idea is sensible, the results are consistent across three models, and the issues are fixable. I'd send it out, with the expectation of major revision around the missing baseline and the lossless wording. For a reading group, it's worth one session.","headline":"A practical, well-tested method for unifying visual attention patterns via offline-chosen axis permutations — worth reviewing, but the missing SparseVideoGen2 comparison and “lossless” wording need fixing.","tokens_in":23280,"tokens_out":3781,"would_cite":true,"duration_ms":34398,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that reordering tokens to make attention maps block-wise lets sparse and quantized attention match full-precision generation quality while cutting latency by 1.9-2.7x.","keywords":["attention sparsification","attention quantization","token reordering","diffusion transformers","video generation","image generation","block sparsity","integer quantization"],"falsifier":"Use a held-out prompt set deliberately different from the calibration prompts (e.g., text-heavy scenes or unusual camera motion), run PAROAttention at 20-30% density or INT4, and compare outputs to the FP16 baseline. If the attention maps under the static permutation no longer concentrate into block-sparse regions (measured by a drop in the sparse-block fraction or a rise in block incoherence), and PSNR or cosine similarity degrade far below the values reported in the paper's tables, the generalization claim is falsified.","tokens_in":22113,"feed_emoji":"⚡","tokens_out":9600,"duration_ms":101057,"temperature":0.7,"pith_summary":"This paper argues that the real obstacle to compressing attention in visual generation models is the layout of the attention map, not the absence of clever sparse masks or quantizers. In diffusion transformers, 3D image and video features are flattened into 1D token sequences, so heads that aggregate along different spatial axes produce multi-diagonal and dispersed attention patterns that are hard to sparsify or quantize. The proposed method, PARO, reorders tokens per head so that all these patterns collapse into a single block-wise structure, chosen offline from six axis permutations; the resulting static block mask and block-wise INT8/INT4 quantization then operate at 20-30% density without the quality collapse seen in prior baselines. On CogVideoX and Flux the authors report outputs nearly identical to the FP16 full-attention baseline, with end-to-end latency speedups of 1.9-2.7x. A sympathetic reader would care because it converts a hard pattern-matching engineering problem into a cheap layout change that any sparsity or quantization scheme can exploit.","feed_headline":"Reordering tokens lets sparse+quantized attention match full precision","feed_subtitle":"At 20-30% attention density and INT8/INT4 bits, generation quality holds while end-to-end latency drops 1.9-2.7x.","key_machinery":"The load-bearing mechanism is PARO (Pattern-Aware token ReOrdering), a per-head permutation of the flattened token sequence restricted to the six axis orders of $[F,H,W]$, chosen offline by a metric that balances block sparsifiability (fraction of $64 \\times 64$ blocks that are mostly below a small threshold) against quantization incoherence (max over mean absolute value inside each block). It converts multi-diagonal attention into block-wise attention; the companion machinery is a static, timestep-aware sparse mask (block-sum thresholding, shared after early timesteps, stored as bitmasks with prefetch) and block-wise INT8/INT4 quantization aligned to the FlashAttention block size. The permutation is fused into preceding kernels so its runtime overhead stays below 1%.","core_discovery":"The central discovery is that the apparently diverse attention patterns in visual generation models—block-wise, multi-diagonal, diagonal-in-block—are the same phenomenon: local aggregation of neighboring pixels along different axes of a feature volume that has been flattened into a 1D sequence. By permuting the token order so that the aggregation axis becomes contiguous, the paper unifies all patterns into a regular block-wise form. This one reorganization makes both compression techniques simple: a static block mask selected by block-sum thresholding (shared across prompts, and across later timesteps), and block-wise INT8/INT4 quantization of the post-softmax attention map, whose block incoherence falls from roughly 200-1200 to 12-20 after reordering. The paper's headline evidence is that on CogVideoX-5B and Flux, PAROAttention keeps quality metrics and pixel/feature-space similarity close to the FP16 dense baseline at 20-30% density and INT8/INT4 bitwidth, while the CUDA kernels deliver 1.9-2.7x end-to-end speedups.","pith_inferences":["An immediate extension is adaptive permutation: a small predictor on prompt or timestep features could pick the order per head, extending the same gains to distributions where pattern type varies more than the calibration suggests.","Because the pattern diversity is partly an artifact of token layout, training models with axis-aligned attention heads might produce attention that is natively block-sparse and low-incoherence, without any post-training permutation.","The reported speedups are tied to 64x64 block alignment on the target GPU; on other hardware or tile sizes the optimal density and bitwidth would need re-tuning, and end-to-end gains could differ.","The incoherence reduction from roughly 200-1200 to 12-20 suggests a general recipe: any compression scheme struggling with outliers inside quantization groups can first permute data to group similar values, which may transfer to KV-cache quantization in other models."],"forward_implications":["Sparse and quantized attention for visual DiTs can be designed around one unified block pattern rather than per-head specialized masks; the change is a layout swap, not a new compression algorithm.","At 20-30% density, PAROAttn matches or beats the quality of 50%-density baselines, so practical video generation moves to much lower density.","Quantizing the attention map to INT8/INT4 becomes viable, letting attention run on integer tensor cores and non-GPU accelerators rather than staying at FP8/FP16.","Because permutation also helps dynamic sparsity (SpargeAttn with PARO at 30% matches its 50% quality), reordering is a drop-in preprocessing step for other compression methods.","End-to-end inference of CogVideoX and Flux gains 1.9-2.7x with under 1% overhead and no metric loss, making long-video and high-resolution generation more deployable."],"supporting_citations":[{"why":"Target video model: its 3D full attention over 17K tokens is the setting where PAROAttention is evaluated.","marker":"[56]"},{"why":"Target image model: Flux at 1024x1024 is the second evaluation setting.","marker":"[19]"},{"why":"Defines the diffusion transformer (DiT) architecture that PAROAttention compresses.","marker":"[35]"},{"why":"Supplies the local-aggregation view of visual attention that motivates token reordering.","marker":"[33]"},{"why":"Provides the incoherence metric used in the permutation selection and quantization analysis.","marker":"[69]"},{"why":"Quantization baseline that quantizes QK to INT8 while keeping PV in FP16; PAROAttention extends PV to INT8/INT4.","marker":"[61]"},{"why":"SageAttention2 kernel is the base implementation PAROAttention's CUDA kernels build on.","marker":"[59]"},{"why":"Dynamic sparse-attention baseline; combining PARO with it shows the reorder helps existing methods.","marker":"[62]"},{"why":"Static sparse baseline for video generation that PAROAttention outperforms at equal or lower density.","marker":"[45]"},{"why":"VBench prompt set used to evaluate text-to-video generation quality.","marker":"[17]"}],"fun_headline_variants":["Token reorder turns scattered attention into block-friendly sparse+quantized","PAROAttention: One reorder unifies sparse and quantized attention","Reordering attention tokens makes sparsity+quantization lossless","Token reordering unifies attention patterns for 2.7x faster generation","Sparse+quantized attention matches full precision via token reorder"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the offline choice of one permutation per head staying correct for prompts, timesteps, and CFG settings that were not seen at calibration time; if that permutation—and the static masks built on it—stops matching the true attention, the reported quality and speedup numbers do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Token reorder turns scattered attention into block-friendly sparse+quantized","PAROAttention: One reorder unifies sparse and quantized attention","Reordering attention tokens makes sparsity+quantization lossless","Token reordering unifies attention patterns for 2.7x faster generation","Sparse+quantized attention matches full precision via token reorder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2746,"prompt_tokens":1038,"completion_tokens":1708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":1615}},"tokens_in":654,"tokens_out":1708,"duration_ms":12641,"temperature":1.0,"reasoning_tokens":1615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:34.251247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a held-out prompt set deliberately different from the calibration prompts (e.g., text-heavy scenes or unusual camera motion), run PAROAttention at 20-30% density or INT4, and compare outputs to the FP16 baseline. If the attention maps under the static permutation no longer concentrate into block-sparse regions (measured by a drop in the sparse-block fraction or a rise in block incoherence), and PSNR or cosine similarity degrade far below the values reported in the paper's tables, the generalization claim is falsified.","supporting_citations":[{"cited_title":"Flux.1: A high-quality text-to-image model","cited_arxiv_id":null,"evidence_quote":"Target image model: Flux at 1024x1024 is the second evaluation setting."},{"cited_title":"Scalable diffusion models with transformers, 2023","cited_arxiv_id":null,"evidence_quote":"Defines the diffusion transformer (DiT) architecture that PAROAttention compresses."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Supplies the local-aggregation view of visual attention that motivates token reordering."},{"cited_title":"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the incoherence metric used in the permutation selection and quantization analysis."},{"cited_title":"Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration","cited_arxiv_id":null,"evidence_quote":"Quantization baseline that quantizes QK to INT8 while keeping PV in FP16; PAROAttention extends PV to INT8/INT4."},{"cited_title":"Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024","cited_arxiv_id":null,"evidence_quote":"SageAttention2 kernel is the base implementation PAROAttention's CUDA kernels build on."},{"cited_title":"Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025","cited_arxiv_id":null,"evidence_quote":"Static sparse baseline for video generation that PAROAttention outperforms at equal or lower density."}],"review_version":1}