{"id":"170f79cc-3160-4327-9b89-41a7473483f9","arxiv_id":"2502.04507","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sliding tile attention (STA) replaces full 3D attention in video diffusion transformers with dense tile-local windows, achieving 1.89x training-free and up to 3.53x fine-tuned end-to-end speedups on HunyuanVideo with small quality loss.","lead":"This paper introduces sliding tile attention, a sparse attention kernel for video diffusion models where each query tile attends only to nearby key tiles instead of the full clip. On HunyuanVideo it cuts a 945-second generation to 501 seconds without training and to 268 seconds after a short fine-tune, with small quality trade-offs the paper calls near-parity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 1.89x/3.53x speedups are not full-pipeline end-to-end numbers: the paper excludes VAE and text-encoder time from measured latencies while the abstract calls 945s \"total inference time,\" so the headline speedup magnitude is unverified.","rationale":"I focused on the headline speedups because they are the paper's most important practical contribution. The paper's own methodology excludes VAE and text-encoder time from measured latencies, while the abstract and Figure 1 describe 945s as total inference time. That inconsistency makes the central speed claim unverifiable without full-pipeline timing. This is not an accusation of fraud: Section 4 is transparent about the DiT-only scope, but the abstract and the term \"end-to-end\" overstate what is actually measured. The reader's identified weakness, overfitting of Algorithm 1's 16-prompt mask search, is real but secondary, because the paper does provide human evaluation on 200 unseen prompts and VBench scores; those results can be inspected for quality degradation directly. By contrast, no experiment in the paper reports the VAE/text-encoder time needed to validate the end-to-end speedup. I would keep the verdict CONDITIONAL: the kernel design and quality evidence are plausible and largely reproducible in principle, but the headline speedup should be accepted only after full-pipeline timings confirm or correct the reported ratios.","tokens_in":19876,"tokens_out":14349,"duration_ms":150951,"concrete_test":"Reproduce Tables 3 and 4 with the released FastVideo/STA code on the same H100 and the paper's 720p/5s Hunyuan settings. For FA3, STA-training-free, and STA-fine-tuned, measure (a) full wall-clock generation including text encoding and VAE encode/decode, and (b) DiT-only latency as defined in the paper. If the full-pipeline speedups from (a) are within 10% of 1.89x and 3.53x, the timing concern is resolved. If they are materially lower, the headline and abstract must be restated as DiT-only speedups, and the missing VAE-plus-text-encoder time E=(a)-(b) should be reported explicitly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central practical claim is that STA cuts end-to-end HunyuanVideo latency from 945s (FA3) to 501s training-free and 268s fine-tuned. Section 4, however, states that reported wall-clock latency explicitly excludes time spent on the VAE and text encoder, whereas the abstract describes the 945s baseline as \"total inference time\" and Figure 1 says attention takes 800s of that total. If the baseline includes components that the STA timings exclude, the speedups compare different pipeline scopes and are inflated. Even if 945s is already DiT-only, the user-facing speedup is (945+E)/(501+E) training-free and (945+E)/(268+E) fine-tuned, where E is the combined VAE and text-encoder time; for any E>0 these ratios are strictly below 1.89x and 3.53x. No value of E is reported, so the headline speedup is unverifiable as stated. For example, E=150s would reduce the claimed speedups to roughly 1.63x and 2.62x. This is a measurement-scope issue rather than a kernel-correctness issue, but it is load-bearing because the paper's main contribution is expressed in end-to-end latency terms.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sliding Tile Attention (STA), a tile-based sparse attention mechanism for video diffusion transformers. STA groups tokens into 3D tiles so that all queries in a tile attend to the same set of key tiles, turning the irregular sliding-window mask into a mix of dense and empty FlashAttention blocks. The authors implement STA with a ThunderKittens/FlashAttention-3-style kernel using asynchronous data loading and report large kernel speedups (up to 10.45x over FA3 at 91% sparsity, 58.79% MFU). On HunyuanVideo they report end-to-end latency reductions from 945s (FA3) to 501s training-free and 268s with fine-tuning, and quality evaluations via VBench, SSIM/PSNR/CD-FVD, and pairwise human evaluation. They also apply STA to FLUX for image super-resolution.","tokens_in":20110,"tokens_out":9139,"duration_ms":87734,"significance":"The kernel contribution is significant. The paper identifies a real gap: existing 2D/3D sliding-window attention implementations (NATTEN, CLEAR) fail to convert FLOP reductions into wall-clock speedups, and STA addresses this with a clean tile-based formulation and a carefully engineered kernel. The benchmarks are detailed and the code is promised to be public. If the measurement-scope and quality-claim issues are corrected, this would be a useful systems contribution for accelerating video diffusion models. The paper also gives credit for the disclosed mask-search design choice and provides a transparent kernel-efficiency table.","major_comments":[{"comment":"The end-to-end latency comparison is ambiguous in scope. Section 4 states, 'For end-to-end speedup on DiT, we report measured wall-clock latency, excluding time spent on VAE and text encoder,' while the abstract and Figure 1 describe 945s as 'total inference time' with attention taking 800s of it. If 945s is total pipeline time and the STA timings of 501s and 268s exclude VAE and text encoder, then the reported speedups of 1.89x and 3.53x compare different measurement scopes and are inflated. If 945s is intended to be DiT-only, then the abstract's wording is incorrect, and for any added VAE+text-encoder time E the user-facing speedups become (945+E)/(501+E) and (945+E)/(268+E), which are strictly smaller than the headline values. The paper should either report full-pipeline latencies including VAE and text encoder or explicitly state the value of E and recompute the speedups.","section":"Section 4 and Abstract"},{"comment":"The claim of 'without quality degradation' is contradicted by the paper's own data. The abstract states that STA reduces latency to 501s 'without quality degradation,' but Table 4 reports VBench Total 80.58 for the training-free STA at 91% sparsity versus 82.71 for FA3, and Section 4.2 reports that STA-tf-1.89x has a 7.0-percentage-point lower win rate than loss rate in the human pairwise evaluation. These results indicate measurable quality loss, even if modest. The abstract should be softened to 'minimal quality degradation' or accompanied by a statistical test showing the human-eval deficit is not significant. Additionally, the two abstracts in the manuscript disagree on the training-free latency: the arXiv-style abstract says 685s while the paper's abstract says 501s; this inconsistency must be resolved.","section":"Abstract and Section 4.2 / Table 4"},{"comment":"The formulas in Theorem 3.1 do not reproduce the numbers in Table 1. For the stated configuration (tile size 4, window size 11, video size 48), the theorem gives Ndense = (max(2*floor(12/8)-1,0))^3 * 12^3 = 1728 and Nmix = (2*floor(10/8)+1)^3 * 12^3 - 1728 = 44928. Since the total number of FA blocks is 12^3 * 12^3 = 2,985,984, these correspond to 0.06% dense and 1.50% mixed, but Table 1 reports 7.17% mixed. The text says 'We apply Theorem 3.1 and Theorem 3.2 to calculate the ratio of different blocks and report them in Table 1,' so this is an internal inconsistency. If boundary effects explain the difference, the theorem should state that it ignores boundaries and the table should be annotated accordingly. The qualitative claim that STA eliminates mixed blocks is not in question, but the quantitative comparison is inaccurate as presented.","section":"Theorem 3.1 and Table 1"},{"comment":"The training-free configuration search in Algorithm 1 selects per-head window sizes by minimizing MSE to full-attention outputs on only 16 prompts. The claim that this configuration transfers to unseen prompts is central to the training-free quality results. The paper should provide a sensitivity analysis, for example by running the search on different random subsets of 16 prompts and reporting the resulting VBench scores or the variation in the selected window-size distribution. Without such an analysis, the VBench and human-eval results could be dependent on the particular 16-prompt search set, and the generality of the head-specialization assumption is not fully established.","section":"Algorithm 1 and Section 3.2"}],"minor_comments":[{"comment":"The caption and text contain corrupted unicode sequences such as '/uni00000013/uni00000018' and 'uni0000002b/uni00000048'; the figure labels should be regenerated with a proper font/encoding.","section":"Figure 3 and surrounding text"},{"comment":"The phrase 'head specialization remains agonistic to prompts' should read 'agnostic'.","section":"Section 1"},{"comment":"The phrase 'wallcock speedups' should read 'wall-clock speedups'.","section":"Section 1"},{"comment":"The heading 'Visialization of 2D SWA' should read 'Visualization of 2D SWA'.","section":"Appendix A"},{"comment":"The paper notes that 'Other memory-bound operations, such as LayerNorm and modulation, likely contribute to inference overhead,' but does not quantify this. A brief breakdown of the DiT inference time would help readers reconcile the 5.76x FLOP reduction with the 3.53x latency reduction.","section":"Section 4.4, footnote 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems contribution with a well-engineered kernel and transparent benchmarks. The main issues are fixable: clarify the latency measurement scope, correct the quality claim in the abstract, reconcile Theorem 3.1 with Table 1, and add a sensitivity analysis for the mask search. The discrepancy between the 685s and 501s numbers in the two abstracts should also be cleaned up. I do not see a reason to reject the paper, but the headline speedup and 'no quality degradation' claims need to be made accurate before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is solid and worth engaging with. STA replaces token-level sliding windows with tile-level windows, so each query block attends to whole key tiles, eliminating the mixed blocks that kill the efficiency of NATTEN and CLEAR. That is a real algorithmic insight, and the kernel numbers back it up: 58.79% MFU and a 10.45x speedup over FA3 at 91% sparsity is impressive, and the wall-clock speedups over NATTEN/CLEAR are dramatic. The paper also does a useful empirical study of attention locality and head specialization in HunyuanVideo, and the code is public. Those are real contributions.\n\nThe soft spots are mostly around claims rather than the mechanism itself. The abstract says \"without quality degradation,\" but the paper's own VBench numbers show 80.58 (training-free) versus 82.71 for full attention, and the human evaluation shows a 7-point win-minus-loss deficit against the original. That is small but not zero, and the abstract overstates it. The latency accounting is also confusing: Section 4 says reported latencies exclude VAE and text encoder, while the abstract calls 945s \"total inference time.\" If both baseline and STA timings exclude those components, the ratio is fine, but the wording makes it look like the headline 1.89x/3.53x are user-facing end-to-end speedups, and they are not. That needs to be fixed with a clear statement of what is included. There is also a concrete internal inconsistency: Theorem 3.1 gives a mixed-block ratio of about 1.5% for Tiled NATTEN with window (11,11,11) on a (48,48,48) video, but Table 1 reports 7.17%. One of those is wrong, and it matters because the whole motivation is about mixed-block overhead. Algorithm 1 is under-specified—it records a mask per timestep and head but does not explain how the per-head configuration is aggregated across the 16 prompts. Human evaluation is reported without error bars.\n\nNone of these are load-bearing flaws in the mechanism. The speedup evidence is strong, the baselines are appropriate, and the code is available. But the paper as written overclaims quality preservation and conflates measurement scopes, and the Theorem 3.1/Table 1 mismatch needs a correction. With those fixed, this is a solid systems paper for a venue like ICML or MLSys. It deserves a serious referee, and I would be happy to see it in the reading group.\n\nRecommendation: send to peer review, with the expectation of major revision on the presentation of quality and latency claims.","headline":"A genuinely useful sparse-attention kernel for video DiTs with real speedups, but the paper's own quality numbers and latency accounting undercut its headline claims.","tokens_in":20722,"tokens_out":3436,"would_cite":true,"duration_ms":32913,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sliding tile attention replaces wasteful full 3D attention in video diffusion models, cutting generation time nearly in half without retraining while keeping output quality at near-parity.","keywords":["sliding tile attention","video diffusion transformer","sparse attention","3D locality","head specialization","kernel optimization","HunyuanVideo","inference acceleration"],"falsifier":"Run the training-free STA configuration on a held-out set of prompts with strong camera motion, rapid cuts, or small fast-moving objects, and compare against full attention with human evaluation; if the tie rate falls far below the reported 83% or VBench Total drops more than a point, the head-specialization assumption does not transfer. A kernel-level check: measure the STA kernel's MFU and latency on a non-Hopper GPU; the paper's hardware claim predicts the wall-clock speedup still scales with sparsity, so a machine where this fails would falsify the generality of the tile-based design.","tokens_in":36,"feed_emoji":"🎬","tokens_out":8410,"duration_ms":135223,"temperature":0.7,"pith_summary":"The paper claims that the dominant compute cost of state-of-the-art video diffusion transformers is full 3D attention, and that most of this attention is redundant: a local window covering about 15.5% of the token space carries about 70% of the attention mass. To exploit this, it introduces sliding tile attention (STA), which slides over cubes of tokens rather than individual tokens, so every computed attention block is dense and the sparse mask never touches the fast compute path. On a leading open video model, STA cuts end-to-end generation of a 5-second 720P clip from 945 seconds with FlashAttention-3 to 501 seconds with no training, with an 83% human tie rate against the original, and to 268 seconds after fine-tuning with a VBench Total of 82.62 versus 82.71. If right, this gives a plug-in speedup for existing video diffusion transformers and a new system-algorithm co-design pattern for sparse attention.","feed_headline":"Sliding-tile attention makes video DiTs 1.9x faster, no retraining","feed_subtitle":"Training-free swap cuts 5s 720p generation from ~16 min to ~8 min; fine-tuning reaches 3.5x.","key_machinery":"The load-bearing object is the tile: a contiguous group of tokens forming a spatial-temporal cube, sized so that one tile equals one FlashAttention block, with token ordering inside a tile kept consecutive. STA slides over tiles instead of tokens, so all queries in a tile share the same key-tile set and the attention map contains only dense and empty blocks. The kernel's efficiency comes from a consumer–producer split: producer warpgroups asynchronously load only the KV blocks in the window and own all mask logic, while consumer warpgroups run dense attention with no masking overhead at all. The second mechanism is head specialization, exploited by Algorithm 1, which picks per-head window sizes by profiling 16 prompts to find the mask minimizing MSE to full attention.","core_discovery":"The central discovery is that video diffusion transformers trained with full 3D attention develop strong, prompt-invariant locality: each attention head concentrates mass in a fixed local spatial–temporal window, and different heads specialize in different window sizes. The paper shows that by reordering the attention computation into tiles whose size equals the FlashAttention block size, the sliding-window pattern decomposes into dense blocks and empty blocks only, with no mixed blocks, turning a theoretically cheap but practically slow sparse attention into one whose wall-clock speedup tracks the FLOP reduction. Per-head window sizes can then be selected automatically by minimizing the MSE between masked and full attention outputs on just 16 prompts, giving training-free speedups with near-parity quality, and a short fine-tuning stage closes the remaining gap.","pith_inferences":["The prompt-invariance of head locality, if it holds, suggests that full 3D attention in video diffusion transformers is heavily overtrained, and the 16-prompt search could itself be replaced by an online adaptive search that updates windows per diffusion timestep without hurting quality.","The tile-wise decomposition may transfer to other high-dimensional attention problems with locality, such as point cloud transformers or multi-view 3D generators, where mixed-block overhead has similarly prevented sparse attention from paying off.","The reported quality numbers imply a direct cost-quality knob: because larger windows preserve more of the full-attention distribution, deployment could tune sparsity per scene complexity rather than globally."],"forward_implications":["Existing pretrained video diffusion transformers can be accelerated about 1.9x without any training by swapping in STA and a per-head window configuration.","With roughly 8 hours of fine-tuning on 2,000 generated clips, the same models run about 3.5x faster at 91% attention sparsity with essentially unchanged VBench scores.","STA outperforms cache-based acceleration on both quality and speed, including higher human win rates at comparable inference budgets.","Because STA's windows are selected per head, the method is complementary to step-distillation and caching, so the speedups compose.","The tile-based pattern also yields speedups in 2D image super-resolution, indicating the mechanism generalizes beyond video."],"supporting_citations":[{"why":"Supplies the full-attention video diffusion transformer that STA accelerates and the 945s baseline latency.","marker":"HunyuanVideo-Team, 2025"},{"why":"FlashAttention-3 is the kernel baseline STA must beat and the source of the consumer-producer design pattern.","marker":"Shah et al., 2024"},{"why":"Provides the kernel framework on which STA's optimized tile kernel is implemented.","marker":"Spector et al., 2024"},{"why":"FlexAttention is used for STA's simpler implementation and for baseline kernels, and it quantifies the mask-overhead cost that STA avoids.","marker":"Dong et al., 2024"},{"why":"Neighborhood attention (NATTEN) is the token-wise sliding-window baseline whose mixed-block inefficiency motivates tile-wise STA.","marker":"Hassani et al., 2023"},{"why":"MovieGen Bench provides the 200-prompt human evaluation protocol and the tie-rate metric for quality comparison.","marker":"Polyak et al., 2024"},{"why":"VBench supplies the automated quality scores used to compare training-free and fine-tuned STA against full attention.","marker":"Huang et al., 2024"},{"why":"Delta-DiT is the cache-based training-free acceleration baseline that STA outperforms in quality and speed.","marker":"Chen et al., 2024"}],"fun_headline_variants":["Sliding tile attention: 2-17x faster video attention","Video generation speeds up 1.4x without retraining via tile attention","Tile-based sliding window attention trims video inference time","Sliding tile attention cuts video DiT latency with no fine-tune"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The training-free speedup rests on the assumption that the per-head window sizes chosen by minimizing MSE on only 16 prompts generalize to all other prompts, rather than overfitting those sixteen examples.","fun_headline_variants_meta":{"raw":{"variants":["Sliding tile attention: 2-17x faster video attention","Video generation speeds up 1.4x without retraining via tile attention","Tile-based sliding window attention trims video inference time","Sliding tile attention cuts video DiT latency with no fine-tune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1775,"prompt_tokens":979,"completion_tokens":796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":721}},"tokens_in":595,"tokens_out":796,"duration_ms":7690,"temperature":1.0,"reasoning_tokens":721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:29:50.908004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the training-free STA configuration on a held-out set of prompts with strong camera motion, rapid cuts, or small fast-moving objects, and compare against full attention with human evaluation; if the tie rate falls far below the reported 83% or VBench Total drops more than a point, the head-specialization assumption does not transfer. A kernel-level check: measure the STA kernel's MFU and latency on a non-Hopper GPU; the paper's hardware claim predicts the wall-clock speedup still scales with sparsity, so a machine where this fails would falsify the generality of the tile-based design.","supporting_citations":[],"review_version":1}