{"id":"e8a44b6c-69ae-441f-9ce6-f7234b3f045c","arxiv_id":"2504.16083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MMInference speeds up long-context VLM prefill by up to 8.3x at 1M tokens using modality-aware permutation sparse attention while keeping accuracy close to full attention.","lead":"A new sparse-attention system speeds up the slow pre-filling stage of long-context vision-language models by up to 8.3x at one million tokens. It exploits the grid-like structure of video attention and handles text-video boundaries without retraining, keeping accuracy close to full attention.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration dependence on EgoSchema and absence of an xAttention comparison leave the accuracy-retention half of the central claim insufficiently secured at 1M tokens.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing premise: offline per-head pattern assignment calibrated on a single EgoSchema sample transfers to arbitrary inputs, longer contexts, and unseen benchmarks. My independent stress test agrees and sharpens the concern in two ways. First, the calibration set is not merely a proxy domain; it is the same benchmark as a headline evaluation target, so the EgoSchema result is partially explained by selection. Second, the only accuracy evidence at the claimed 1M-token scale is synthetic needle retrieval, which is a narrow probe of attention locality and does not substantiate accuracy retention on realistic long-context multimodal workloads. The missing xAttention comparison is also a genuine evidence gap because xAttention is the closest published dynamic sparse attention baseline for VLMs. None of these issues invalidate the kernel-level speedup measurements or the engineering contribution, and the small accuracy deltas on the tested benchmarks are credible. They do, however, mean the abstract's accuracy claim is broader than the evidence. The existing CONDITIONAL verdict is appropriate; these concerns do not warrant rejection, but they should be resolved or explicitly scoped before the claim is taken at full strength.","tokens_in":22214,"tokens_out":1522,"duration_ms":15841,"concrete_test":"Re-run the offline pattern search of Algorithm 4 using a non-EgoSchema calibration set, e.g., a small sample drawn from VideoMME or Next-QA, keeping all other settings identical, and re-evaluate the Table 1 benchmarks. If the averaged accuracy on EgoSchema or other tasks shifts materially relative to the published configuration, the pattern assignment is overfit to the EgoSchema calibration sample. Separately, run xAttention on the same LongVILA-1M V-NIAH setup at 1M tokens and report recall and latency; if xAttention matches or beats MMInference, the claimed state-of-the-art advantage is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MMInference accelerates pre-filling by up to 8.3x at 1M tokens while maintaining accuracy. The speedup half is well supported by kernel and end-to-end measurements, but the accuracy half rests on the offline per-head pattern assignment of Algorithm 4. Appendix C.5 states that this assignment is calibrated on a single EgoSchema sample of at most 25K tokens, while EgoSchema is also a headline evaluation benchmark in Table 1. This creates a calibration-to-evaluation overlap: pattern choices that score well on EgoSchema are selected, so the strong EgoSchema row is partly a product of the search objective. More importantly, the 1M-token accuracy evidence consists only of V-NIAH and MM-NIAH synthetic retrieval tasks (Figs. 5, 13, 14). These tasks test a single inserted needle and do not exercise sustained reasoning, detailed captioning, or multi-turn mixed-modality dialogue at long context. Additionally, xAttention (Xu et al., 2025b), the closest VLM dynamic block-sparse baseline, is cited but never compared, so the claimed advantage over the most relevant prior work is not directly evidenced. These are addressable gaps rather than internal inconsistencies, but they mean the 'maintaining accuracy' clause is broader than the current evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MMInference, a training-free dynamic sparse attention method for accelerating the pre-filling stage of long-context vision-language models (VLMs). The authors identify a Grid attention pattern in video inputs, characterize modality-boundary patterns (Q-Boundary, 2D-Boundary, etc.), and propose permutation-based transformations plus an offline per-head pattern search (Algorithm 4) to construct dynamic sparse masks. They provide optimized Triton/FlashAttention-based kernels. Experiments on Llava-Video, LongVILA, Qwen2.5-VL, and VideoChat-Flash report accuracy close to full attention on video QA and captioning benchmarks, with speedups up to 8.3x at roughly 1M tokens on V-NIAH and MM-NIAH retrieval tasks.","tokens_in":22457,"tokens_out":2368,"duration_ms":24032,"significance":"If the accuracy-retention claim holds at the claimed scale, MMInference is a noteworthy systems contribution: it is training-free, model-agnostic, and demonstrates that modality-aware permutation can convert irregular multimodal sparse patterns into efficient dense-block computations. The latency measurements are concrete and reproducible in spirit, and the paper ships code. The kernel-level analyses and the Grid-pattern observation are useful for follow-up work on efficient long-video inference. However, the breadth of the accuracy claim currently exceeds the evidence: the calibration-to-evaluation overlap on EgoSchema and the restriction of 1M-token validation to synthetic needle-in-a-haystack tasks leave the 'maintaining accuracy' clause less secure than the abstract suggests.","major_comments":[{"comment":"The per-head pattern assignment is calibrated on a single EgoSchema sample of at most 25K tokens, and EgoSchema is also a headline benchmark in Table 1. This creates a selection-to-evaluation overlap: the search objective (attention recall on this one sample) directly influences the choice of sparse patterns that are then reported as improved or maintained accuracy on EgoSchema. The claim in Appendix C.5 that this calibration 'exhibits strong generalization and stability' needs quantitative support, for example by re-running the offline search on a sample from a different benchmark and reporting both configurations, or by explicitly treating EgoSchema as a development set and not a held-out evaluation.","section":"Appendix C.5, Table 1"},{"comment":"The central claim of 'maintaining accuracy' at 1M tokens is supported only by V-NIAH and MM-NIAH, which test a single inserted needle at various depths in long video/text haystacks. These tasks do not exercise sustained reasoning, detailed captioning, or multi-turn mixed-modality interactions at long context, so the 1M-token evidence is narrower than the abstract's unqualified 'maintaining accuracy.' I recommend either adding long-context accuracy measurements on a broader set of tasks (for example, long-form video QA with explicit long-range reasoning or document-scale mixed-modal tasks) or explicitly qualifying the accuracy claim to retrieval-style tasks at the 1M scale.","section":"Abstract, §4.3, §4.4, Fig. 5"},{"comment":"xAttention (Xu et al., 2025b) is cited as a recent dynamic sparse attention method for long-context VLMs, but it is not included in any comparison table or latency experiment. Since the paper's contribution is precisely a dynamic sparse attention scheme for VLMs, the absence of this closest comparable baseline leaves the claimed advantage over prior dynamic sparse VLM methods unevidenced. A direct comparison on the same benchmarks and latency setup should be added, or the exclusion should be explicitly justified.","section":"§5 (Related Work), §4.1 (Baselines)"},{"comment":"The offline search uses attention recall on one calibration sample as the selection objective, but the paper does not demonstrate that this recall-based proxy transfers to downstream task accuracy across diverse inputs, context lengths, or benchmarks. Fig. 8 tests index transfer within the same modality but not the transfer of the per-head pattern configuration selected on one EgoSchema sample to the full set of evaluation inputs. A sensitivity analysis showing pattern stability across multiple calibration samples and across context lengths (e.g., 20K, 66K, 300K, 1M tokens) would strengthen the load-bearing transfer assumption.","section":"Algorithm 4 and §4.6 (Fig. 8)"}],"minor_comments":[{"comment":"There are typos: 'attenton' should be 'attention' in the sentences introducing Figs. 13 and 14.","section":"Appendix E.1 and E.2"},{"comment":"'egoschema' is written in lowercase in the text; use consistent capitalization for the benchmark name.","section":"Appendix C.5"},{"comment":"In Grid-Shape Flash Attention, the pseudocode after the softmax-like rescaling is abbreviated (the exp and normalization steps are compressed), which makes it hard to verify correctness against the FlashAttention-2 equations. Please expand the pseudocode so the rescaling and softmax updates are explicit.","section":"Algorithm 5"},{"comment":"The VideoChat-Flash row reports performance without a full-attention baseline in the same table; adding Full Attention and MInference for this model would make the comparison easier to interpret.","section":"Table 2"},{"comment":"The phrase 'offline search the optimal sparse patterns' is a grammatical artifact; consider rephrasing to 'offline search for the optimal sparse patterns.'","section":"Abstract and §3.3"}],"recommendation":"major_revision","confidential_remarks":"The calibration overlap on EgoSchema is the most serious concern and should be addressed head-on in revision. The paper's speedup evidence is solid, but the accuracy-retention claim is currently broader than the evidence at 1M tokens. The missing xAttention comparison is surprising given that it is the closest related dynamic sparse VLM work; the authors should either add it or explain the omission. If these points are resolved with additional experiments or careful claim qualification, the paper could be a solid contribution to efficient long-context VLM inference."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a credible systems paper. It reports a training-free sparse attention method for VLM prefill that achieves large speedups at 1M tokens with small accuracy deltas on the benchmarks tested. The main novelty is real: the paper notices that video inputs produce grid-like attention structure and that modality boundaries break the usual sparse patterns, and it handles both with permutations plus an offline per-head pattern search. That is a genuine extension of MInference, not a repackaging.\n\nWhat's good: the latency measurements are thorough, with kernel-level and end-to-end numbers across four VLMs, and the GPU kernels are described in enough detail to be credible. The speedup over FlashAttention-2 (8.3x at 1M) and over MInference (1.7x) is consistent with the FLOPs reduction, though note MInference is run at roughly twice the FLOPs of MMInference, so part of the win is sparser patterns, not just faster kernels. The accuracy results at 20k-66k tokens on video QA/captioning are solid, and V-NIAH/MM-NIAH at 1M show near-full-attention recall. Code is available.\n\nSoft spots:\n\n1. Appendix C.5 says the offline pattern search is calibrated on one EgoSchema sample (up to 25k tokens), and EgoSchema is a headline row in Table 1. That is a genuine calibration-to-evaluation leak. It is not huge—the search optimizes attention recall on a single sample, not EgoSchema accuracy—but the paper should disclose it clearly and ideally report results with a different calibration source. As written, the EgoSchema row is not strictly held-out.\n\n2. The 1M-token accuracy evidence is only V-NIAH and MM-NIAH, which are single-needle retrieval tasks. No sustained reasoning, captioning, or dialogue at 1M. That does not kill the claim, but the abstract's 'maintaining accuracy' is broader than the evidence.\n\n3. xAttention is cited as the closest VLM dynamic sparse baseline but never compared. Direct comparison would strengthen the paper.\n\nThese are fixable gaps. The central engineering claim—that permutation of grid and modality-boundary attention works and speeds up prefill—holds up as far as I can tell. The paper is for people building long-context VLM inference; for that audience it is useful.\n\nI'd send it to serious reviewers rather than desk-rejecting. My own verdict would be conditional on addressing the calibration leak and scoping the 1M accuracy claim.","headline":"Credible training-free VLM prefill accelerator; permutation-based grid/modality handling is a real extension of MInference, but the 1M accuracy claim needs scoping and the EgoSchema calibration leak should be fixed.","tokens_in":23082,"tokens_out":2639,"would_cite":true,"duration_ms":23323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMInference claims that modality-aware permutation sparse attention accelerates long-context VLM pre-filling by up to 8.3x at 1M tokens while matching full-attention accuracy on the benchmarks tested.","keywords":["sparse attention","long-context vision language models","pre-fill acceleration","permutation-based sparse attention","modality boundaries","video understanding","dynamic sparse indexing","needle in a haystack"],"falsifier":"Construct a long mixed-modality input in which the final 64 queries of a visual segment attend to a different key region than the earlier tokens of that segment (a scene cut or topic shift placed exactly at the segment end), and measure both the attention recall of the online-estimated masks and the needle-retrieval score. If attention recall falls far below the roughly 95% threshold the paper's own analysis treats as adequate, or retrieval drops from the reported 97.7% on V-NIAH, the last-64-queries extrapolation premise is false.","tokens_in":21966,"feed_emoji":"⚡","tokens_out":11739,"duration_ms":89200,"temperature":0.7,"pith_summary":"This paper claims that the slow pre-filling stage of long-context vision-language models, the phase before the model starts generating an answer, can be made roughly an order of magnitude faster with no retraining and no measurable accuracy loss on the benchmarks tested. The underlying observation is that video-driven attention in these models is not unstructured: certain heads show an evenly spaced grid of attended positions, and attention inside one modality behaves differently from attention that crosses a text-video boundary. MMInference exploits this by permuting queries, keys, and values so that scattered grid lines and same-modality regions become contiguous blocks, converting sparse memory access into dense tensor-core computation, while a cheap online estimator builds the sparse index from the last 64 queries of each modality segment. The paper reports up to 8.3x end-to-end prefill speedup at 1M tokens on a single A100 (and 12x at the kernel level) over FlashAttention-2, with video QA, captioning, and needle-in-a-haystack retrieval scores matching full attention.","feed_headline":"Cut prefill time for long-video models by up to 8.3x","feed_subtitle":"Modality-aware sparse attention keeps VLM accuracy on video QA and retrieval while cutting prefill cost.","key_machinery":"The load-bearing machinery is the permutation of the $Q$, $K$, and $V$ tensors guided by a dynamic sparse index. It works through three pattern families: (1) the Grid pattern for video-heavy heads, whose stride and phase are found online by scanning an approximate attention matrix built from the last 64 queries, after which the tensors are permuted so that grid lines align into contiguous blocks (Algorithm 1); (2) the Q-Boundary and 2D-Boundary heads, where row-wise or full modality-grouping permutation isolates intra-modality regions, so a sparse index built once in one visual region extrapolates across text-interrupted visual segments (Algorithms 2 and 3); and (3) the Modality-Aware Sparse Attention Search Algorithm (Algorithm 4), which assigns each head its pattern offline under a kernel-measured FLOPs budget, calibrated on a single 25K-token video sample. Computation is carried out by block-sparse FlashAttention-style kernels (built on FlashAttention-2 and the PIT dynamic sparse compilation approach) that load only the permuted blocks, keeping tensor-core compute dense.","core_discovery":"The paper's central claim is that attention in long-context VLMs is sparse in a way that is both modality-structured and permutation-friendly, so that full-attention accuracy can be recovered by computing only a fraction of the attention matrix. It identifies three pattern families: the Grid pattern, in which video tokens attend along evenly spaced horizontal and vertical lines whose stride and phase are searched online from an approximate attention matrix; the Q-Boundary and 2D-Boundary patterns, where modality boundaries split the query dimension or both query and key dimensions; and the A-shape and Vertical-Slash patterns carried over from text-only LLMs. The method's namesake move is to permute the $Q$, $K$, and $V$ tensors so these patterns become consecutive, with row-wise permutation grouping same-modality queries, column-wise permutation aligning key strides, and 2D permutation isolating each modality pair into its own block, turning scattered sparse loads into dense tensor-core computation. Permutation is realized as index-based sparse loading inside FlashAttention-style kernels, so no weights change and no fine-tuning is needed. Offline, a modality-aware search (Algorithm 4) assigns each head its optimal pattern set under a kernel-measured FLOPs budget; online, the sparse index is estimated from the last 64 queries of each modality segment and the attention is computed with block-sparse kernels. On four VLMs and inputs up to 1.1M tokens, the paper reports up to 8.3x prefill speedup over FlashAttention-2 with benchmark scores matching full attention, including 97.7% versus 98.3% retrieval recall on V-NIAH and 91.3% versus 90.9% on its new mixed-modality NIAH test.","pith_inferences":["The permutation principle should extend beyond prefill: the paper itself notes that 2D/3D sliding-window attention in video diffusion transformers can be reshaped into dense tensor-core tiles by an analogous permutation, which would bring the same speedup to generative video models rather than only to the prefill stage of VLMs.","A testable extension is per-modality index amortization: since a sparse index from one visual region extrapolates across text-separated visual segments, building the index once per modality segment and reusing it could remove even the small cost of the last-64-queries estimator at extreme context lengths.","The reported robustness rests on an unusually small calibration set, one EgoSchema sample of at most 25K tokens, so re-running the offline pattern search across diverse video/text mixtures (different frame rates, scene cuts, interleaving ratios) would map where per-head pattern assignments stay stable and where they break."],"forward_implications":["At 1M-token contexts the pre-fill stage runs up to 8.3x faster end-to-end and up to 12x faster at the kernel level, turning multi-minute waits on one A100 into a small fraction of the time.","Accuracy holds: average scores on six video understanding benchmarks stay within roughly half a point of full attention, and long-video retrieval recall remains at 97.7% versus 98.3% for full attention at 6K frames.","The method is training-free and model-agnostic, applying to LongVILA, Llava-Video, VideoChat-Flash, and Qwen2.5-VL, and it stacks with visual-token compression instead of competing with it.","Static sparse patterns degrade sharply on mixed-modality inputs, whereas MMInference keeps retrieval near full-attention levels up to roughly 1.1M tokens, evidence that handling modality boundaries, not sparsity alone, is what preserves accuracy."],"supporting_citations":[{"why":"Supplies the predecessor dynamic sparse attention framework (A-shape and Vertical-Slash patterns), the online index estimation approach, and the primary baseline; MMInference also follows its kernel-aware search-space methodology.","marker":"(Jiang et al., 2024)"},{"why":"FlashAttention-2 is the dense-attention baseline against which the 8.3x speedup is measured and the kernel foundation the sparse attention builds on.","marker":"(Dao, 2024)"},{"why":"PIT provides the sparse-loading-with-dense-computation paradigm that enables the dynamic sparse kernels to run efficiently on tensor cores.","marker":"(Zheng et al., 2023)"},{"why":"LongVILA supplies the 1M-context model used for V-NIAH, MM-NIAH, and the 8.3x latency measurement.","marker":"(Chen et al., 2025)"},{"why":"Llava-Video is the second main evaluated VLM, used for the video understanding benchmarks at 110 frames.","marker":"(Zhang et al., 2024b)"},{"why":"Defines the V-NIAH long-video retrieval task that serves as the paper's main long-context accuracy test.","marker":"(Zhang et al., 2024a)"},{"why":"StreamingLLM's attention-sink pattern is the A-shape baseline; its failure on mixed-modality inputs is a key comparison point.","marker":"(Xiao et al., 2024)"},{"why":"Sparse Transformer supplies the SF-fixed and SF-strided baselines and the original locality observation for image patches that motivates the grid pattern.","marker":"(Child et al., 2019)"}],"fun_headline_variants":["8.3x faster VLM prefill via sparse permutation attention","Modality-aware sparse attention cuts VLM prefill by 8.3x","Grid-pattern sparse attention accelerates long-video VLM prefill","Permute attention to cut VLM prefill costs by 8.3x","Sparse permutation attention boosts long-context VLM prefill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-head sparse patterns chosen offline from a single 25K-token calibration sample, plus the online sparse indices estimated from the last 64 queries of each modality segment, must transfer to arbitrary inputs, contexts up to 1M tokens, and benchmarks the calibration never saw; if the transfer fails, the masks miss true attention mass and accuracy degrades.","fun_headline_variants_meta":{"raw":{"variants":["8.3x faster VLM prefill via sparse permutation attention","Modality-aware sparse attention cuts VLM prefill by 8.3x","Grid-pattern sparse attention accelerates long-video VLM prefill","Permute attention to cut VLM prefill costs by 8.3x","Sparse permutation attention boosts long-context VLM prefill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000926,"raw_usage":{"total_tokens":4088,"prompt_tokens":1184,"completion_tokens":2904,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":800,"completion_tokens_details":{"reasoning_tokens":2808}},"tokens_in":800,"tokens_out":2904,"duration_ms":17774,"temperature":1.0,"reasoning_tokens":2808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:11.939204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a long mixed-modality input in which the final 64 queries of a visual segment attend to a different key region than the earlier tokens of that segment (a scene cut or topic shift placed exactly at the segment end), and measure both the attention recall of the online-estimated masks and the needle-retrieval score. If attention recall falls far below the roughly 95% threshold the paper's own analysis treats as adequate, or retrieval drops from the reported 97.7% on V-NIAH, the last-64-queries extrapolation premise is false.","supporting_citations":[{"cited_title":"H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L","cited_arxiv_id":null,"evidence_quote":"Supplies the predecessor dynamic sparse attention framework (A-shape and Vertical-Slash patterns), the online index estimation approach, and the primary baseline; MMInference also follows its kernel-aware search-space methodology."},{"cited_title":"Flashattention-2: Faster attention with better parallelism and work partitioning","cited_arxiv_id":null,"evidence_quote":"FlashAttention-2 is the dense-attention baseline against which the 8.3x speedup is measured and the kernel foundation the sparse attention builds on."},{"cited_title":"Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation","cited_arxiv_id":null,"evidence_quote":"PIT provides the sparse-loading-with-dense-computation paradigm that enables the dynamic sparse kernels to run efficiently on tensor cores."},{"cited_title":"Long VILA : Scaling long-context visual language models for long videos","cited_arxiv_id":null,"evidence_quote":"LongVILA supplies the 1M-context model used for V-NIAH, MM-NIAH, and the 8.3x latency measurement."}],"review_version":1}