Pith. sign in

REVIEW 3 major objections 6 minor 4 references

SPAVA is a sequence-parallel framework that accelerates long-video LMM inference by compressing KV caches locally and passing only essential blocks across GPUs, delivering up to 12.72x speedup over FlashAttn without notable accuracy loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:56 UTC pith:4VZQVU2H

load-bearing objection It is APB with better system engineering and faster speedups, but the headline "no performance loss" claim rests on a query-only KV selection rule that is never checked against an oracle, and the hyperparameters are tuned on a VNBench subset that later appears in the main accuracy table. the 3 major comments →

arxiv 2601.21444 v2 pith:4VZQVU2H submitted 2026-01-29 cs.CV cs.AIcs.CL

APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

Yuxiang Huang , Mingye Li , Xu Han , Chaojun Xiao , Weilin Zhao , Ao Sun , Ziqi Yuan , Hao Zhou
show 2 more authors
Fandong Meng Zhiyuan Liu
This is my paper
classification cs.CV cs.AIcs.CL
keywords sequence parallelismapproximate attentionlong-video understandingprefill accelerationKV cache compressionpassing blocksmultimodal LLM inferencedistributed inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SPAVA aims to break the efficiency-performance trade-off in long-video understanding: instead of pruning visual tokens or applying sparse attention on one GPU, it distributes an approximate attention computation across multiple GPUs while keeping every visual embedding. The paper's central claim is that this sequence-parallel approximate attention achieves 12.72x, 1.70x, and 1.18x speedups over FlashAttn, ZigZagRing, and APB, respectively, with accuracy close to full attention on LongVideoBench and VNBench. If correct, SPAVA shows that long-video LMMs can be scaled to more frames and higher resolutions under strict time-to-first-token constraints without compressing away fine-grained video details. The significance is that it moves the bottleneck from the LLM's quadratic attention to a tunable approximate attention whose communication and computation are balanced across hosts.

Core claim

SPAVA's core claim is that approximate attention and sequence parallelism can be combined so that the prefill stage of long-video LMMs becomes faster than prior acceleration strategies without sacrificing accuracy. Each host compresses its local context block to the top-lp key-value pairs ranked by the query block's attention scores, all-gathers these as passing blocks from earlier hosts, and computes attention over the anchor block, local context, and passing blocks. The full visual embedding sequence is preserved, while compute and communication are cut. System-level optimizations—frame parallelism in the visual encoder, ZigZag load balancing across virtual hosts, a fused context-and-query

What carries the argument

The passing block is the central mechanism: each context block is compressed to the top-lp key-value pairs ranked by query-to-context attention scores Q_qr K^T, then all-gathered across hosts so every host sees the most query-relevant KVs from earlier blocks. The anchor block, the first la embeddings, provides stable global context, and local context blocks remain local. Together they reduce attention FLOPs and communication volume while retaining cross-host visibility for essential KVs.

Load-bearing premise

The load-bearing assumption is that the top-lp key-value pairs chosen by the query block's attention scores are the only cross-host essential ones; any KV that matters for an earlier context token but not for the query is excluded from passing blocks and becomes invisible to all later hosts.

What would settle it

Measure, on a long-video input, the overlap between the set of KVs SPAVA selects using Q_qr K^T and the set selected by the union of all context-token queries; then build a QA item whose answer depends on a KV that only context queries select. If SPAVA's accuracy drops specifically on such items, the passing-block selection assumption fails; the overlap fraction is a direct quantitative test.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-video LMMs can process more frames and higher resolutions within a fixed prefill budget, because attention cost no longer grows with the full cross-host context.
  • Because no visual embeddings are pruned, SPAVA's accuracy tracks full attention more closely than token-pruning or sparse-attention baselines on retrieval, ordering, and counting tasks.
  • The approach is training-free and applies to any decoder-only Transformer; the paper reports SPAVA also accelerating long-context NLP inference, with 13.99x over FlashAttn on RULER with Llama-3.1-8B.
  • Accuracy remains stable as host count grows from 2 to 8, so the method is a candidate deployment pattern for large multi-GPU inference systems.
  • The hyperparameters la, anchor length, and lp, passing length, expose an explicit speed-accuracy dial; the chosen setting lp = n/128 gives the reported balance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The selector is query-driven, so SPAVA's passing blocks are implicitly optimized for question-answering; open-ended tasks such as summarization or multi-hop temporal reasoning, where the query does not name the evidence, could expose blind spots that the current benchmarks under-weight.
  • The reported speedup ratios are measured on an 8x A800 NVLink cluster; on slower interconnects, the communication-overlap gains will shrink, so the numbers are upper bounds for typical cloud settings.
  • SPAVA's query-score selection could be reused as a training-free alternative to learned retaining heads for KV eviction during decoding, or combined with such heads when a model already has them.
  • The frame-parallel visual encoder means the speedup applies end-to-end, not only to LLM attention; this makes the framework relevant to encoding-heavy pipelines such as surveillance or autonomous-driving video streams.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. SPAVA proposes a sequence-parallel framework for accelerating long-video LMM prefill. It keeps all visual embeddings, partitions the video context into anchor, local context, and query blocks, and uses query-block attention scores to locally compress each context block to its top-lp KV pairs. These compressed blocks are gathered across hosts as passing blocks, which are then included in the attention computation of every later host together with the anchor and local context. System-level optimizations include frame parallelism in the visual encoder, a fused context-query forward pass, ZigZag-style attention load balancing, and overlapping of communication with attention computation. Experiments on LongVideoBench and VNBench report accuracy close to FULLATTN and speedups of 12.72x, 1.70x, and 1.18x over FLASHATTN, ZIGZAGRING, and APB respectively. The paper also includes system ablations, host scalability studies, a communication-medium sensitivity test, hyperparameter analyses, and a transfer experiment to long-context NLP on RULER.

Significance. If the accuracy-preservation claim holds, SPAVA is a practically useful multi-GPU prefill accelerator for long-video LMMs: it avoids irreversible token pruning while cutting the quadratic attention cost and the communication volume of exact sequence parallelism. The paper has concrete strengths: it is training-free (in contrast to APB's trained retaining heads), it preserves all visual embeddings, code is released, and it provides extensive ablations on each system optimization, host scalability, and communication medium. Its best result on Qwen2.5VL-7B/LongVideoBench even exceeds FULLATTN. The main risk is that the KV-selection rule is query-centric while the selected blocks are consumed by context tokens, and the benchmark-level hyperparameter choice is made on a component of the same benchmark used for the headline accuracy table. Both concerns are addressable with additional experiments.

major comments (3)
  1. [§3.3, Eq. (4), Algorithm 1 lines 1–5] The passing block is built solely from Q_qr K^(h)T, yet Eq. (4) makes that passing block part of every context token's attention input. A KV pair that is important for an earlier context token but not for the query is therefore never propagated to later hosts and is invisible to them. Table 3 shows that removing P is harmful, but it only compares presence versus absence of the passing block, not the selection criterion. The case study in §5.5 only shows that query-relevant blocks are selected more often; it does not check whether context-relevant blocks are retained. To support the "without notable performance loss" claim, please add an oracle comparison, e.g., select the top-lp KVs by cumulative attention from all tokens in the context block rather than by query attention, and report both accuracy and KV-overlap with the query-based selection on VNBench. Without such a test, the suffici
  2. [§5.4, Tables 8–9 and Table 1] The default hyperparameters la=n/64 and lp=n/128 are selected using the Ordering-E subset of VNBench with InternVL3-2B, and then VNBench Overall is reported as a headline accuracy result for the same model in Table 1. This is tuning on a component of the test benchmark. Please hold out a separate validation split for hyperparameter selection, or report accuracy for both the tuned setting and an untuned default across all models, so the reader can assess the degree of contamination.
  3. [§5.3, Tables 4, 5, 7] All throughput numbers are reported as point estimates without repetitions, error bars, or a statement of measurement methodology. The headline speedup claims (12.72x/1.70x/1.18x) and the system-ablation comparisons are central to the paper, but some differences are small (e.g., Table 4 at 16 frames: SPAVA 1.846 vs -O 1.827; Table 7: -0.75%). Please report mean ± std over multiple runs, specify whether the timing includes visual encoding, prefill, decoding, or end-to-end TTFT, and state the batch size and warm-up procedure.
minor comments (6)
  1. [Table 3 and §5.4] Table 3's header says the tested model is Qwen2.5VL-3B, but the surrounding text says the ablation is run on InternVL3-2B. Please reconcile.
  2. [Algorithm 1, line 11] In the attention for Block 2, the key list appears as [Ka, K_p^(2H−1), K_p^(2H−1)], which repeats the passing-block key twice. It should presumably be [Ka, K_p^(2H−1), K^(2H−1)], matching Eq. (4). Please fix.
  3. [Table 9] The caption says "Hyperparameter analysis on l_p", but the table varies l_a. Please correct the caption.
  4. [§5.5] The case study uses a 256-frame video, while §5.1 states that the frame number is set to 64 for the benchmarks. Please clarify whether the case study uses a different configuration and why.
  5. [Global] Minor typographical issues: "degredation" in the Abstract and Tables, "Spava" in Figure 1, and inconsistent capitalization of SPAVA in figures/legends. Also, several reference entries contain "and 1 others" placeholders; please clean these up.
  6. [Figure 2 and §3.3] The relationship between physical hosts, virtual hosts, and the two context blocks per physical host is dense. A short textual example of the notation for H=2 or H=4 would make the framework much easier to follow.

Circularity Check

0 steps flagged

No significant circularity: SPAVA's speed and accuracy claims are empirical; the query-based KV selection and VNBench hyperparameter tuning are correctness/benchmarking concerns, not reductions by construction.

full rationale

SPAVA is an empirical systems paper: the headline speedups are measured against FLASHATTN, ZIGZAGRING, and APB under controlled settings, and the accuracy results are reported from benchmark evaluations rather than derived from the method's definitions. The only candidate circular mechanisms do not hold up under inspection. (i) The passing-block selector (Eq. 4, Algorithm 1) chooses top-lp KVs using Q_qr K^(h)^T and then feeds those passing blocks into context-block attention. This is a heuristic assumption about which KVs are important, not an identity or a fit to the accuracy target; the ablation in Table 3 tests presence versus absence of passing blocks, not the selection criterion. It may be a correctness risk for context-token representations, but it is not circular. (ii) The hyperparameters la and lp are chosen via ablations on VNBench Ordering-E (Tables 8-9), and VNBench Overall is later reported. This is a benchmarking/tuning concern that can inflate reported accuracy, but the reported numbers are measured outcomes, not statistically forced predictions, and the speedup claims are independent of these accuracy hyperparameters. (iii) APB (Huang et al., 2025) is cited for the passing-block idea with overlapping authors, but SPAVA implements its own training-free selector, evaluates against APB, and does not use the APB citation as a proof, uniqueness argument, or hidden ansatz. No equation in the paper reduces to its own inputs, and no fitted parameter is renamed as a prediction. The identified weaknesses are accuracy-risk and evaluation-validity issues, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced; 'anchor block', 'passing block', and 'virtual host' are software constructs. The main load-bearing premises are the KV-importance proxy and the independence/frame-parallelism assumption. The two numeric free parameters (la, lp) are tuned on the same benchmark family used for the headline results, which is the main circularity burden.

free parameters (3)
  • anchor length la = n/64 (e.g., 512 for VNBench InternVL3-2B)
    Fraction of input length kept as the anchor block; chosen from ablation in Table 9 to balance Ordering-E accuracy against throughput.
  • passing length lp = n/128 (e.g., 256 for VNBench InternVL3-2B)
    Compressed KV block length passed across hosts; chosen from ablation in Table 8 because smaller values degrade accuracy and larger values reduce throughput.
  • host count H = 8 (main experiments)
    Number of physical GPUs in the headline speedup measurements; varied in Table 5. The 12.72x speedup over FlashAttn depends on this deployment choice.
axioms (4)
  • domain assumption Only the most essential KV pairs need to be visible to subsequent tokens; non-essential KVs can remain confined to their local block.
    Stated as the observation motivating approximate attention in §3.3; if false for some videos, SPAVA's accuracy would degrade.
  • domain assumption The query block's attention scores Q_qr K^(h)^T are a sufficient proxy for the importance of each local KV to all context tokens.
    Algorithm 1 lines 1-5 use ArgTop_lp of these scores to build passing blocks; context tokens themselves never vote on which cross-host KVs are essential.
  • domain assumption Frame encoding is independent across frames, so frame parallelism does not change model outputs.
    Used in §3.1 and §4.1, inspired by LongVILA; true for the evaluated ViT-based LMMs but not universal.
  • standard math Online-softmax (FlashAttention lse) merging of partial query attention results across hosts is numerically exact.
    Used in Eq. 5 and Appendix A to merge partial query attention results; this is a standard property of FlashAttention-style online softmax.

pith-pipeline@v1.3.0-alltime-deepseek · 23104 in / 16938 out tokens · 171742 ms · 2026-08-03T06:56:20.059605+00:00 · methodology

0 comments
read the original abstract

The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose APB-V, a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, APB-V reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of APB-V, delivering speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss. Code available at https://github.com/thunlp/APB

Figures

Figures reproduced from arXiv: 2601.21444 by Ao Sun, Chaojun Xiao, Fandong Meng, Hao Zhou, Mingye Li, Weilin Zhao, Xu Han, Yuxiang Huang, Zhiyuan Liu, Ziqi Yuan.

Figure 1
Figure 1. Figure 1: SPAVA’s performance and inference speed using Qwen2.5VL-3B as the base LMM on VNBench (processing 64-frame 1440p videos). spite the potential in advancing long-video process￾ing, LMMs encounter severe efficiency bottlenecks when handling ultra-long videos. An increased number of video frames results in a larger batch size for the visual encoder, leading to a rise in both compute cost and the number of gene… view at source ↗
Figure 2
Figure 2. Figure 2: The framework of SPAVA. The anchor block and passing block are denoted as Ba and Bp, while B(h) denotes the context block on virtual host h. Bqr represents the query block. The video input is first encoded into embeddings using frame parallelism across hosts. After context splitting, each physical host (containing two virtual hosts) holds the anchor block, query block, and corresponding context blocks. In … view at source ↗
Figure 5
Figure 5. Figure 5: Overlapping communication with computa￾tion on virtual host h. “QqrK(h1,h2)⊤” estimates KV importance; “B (h1,h2) c ” are essential KVs; “query” and “merge” denote query attention and its merging; “Ba” and “B(h1,h2) ” indicate anchor and context attention. We briefly introduce the following system optimiza￾tions to alleviate these obstacles, with more details introduced in Appendix B. 4.1 Visual Load Balan… view at source ↗
Figure 6
Figure 6. Figure 6: The relative speedup of SPAVA and baselines compared to FLASHATTN under various number of frames. We use 720p for Qwen2.5-VL models. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The relative speedup of SPAVA and baselines compared to FLASHATTN under various resolutions. Method Counting E-1 E-2 I Avg. w/o A 50.00 7.33 27.33 25.33 w/o P 21.33 5.33 25.33 17.33 SPAVA 52.00 9.33 28.67 30.00 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A case study from VNBench. Yellow intensity indicates how frequently a spatial position is selected into [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith

  1. [2]

    arXiv:2504.21403

    Static or dynamic: Towards query-adaptive token selection for video question answering. arXiv:2504.21403. Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2025. Shadowkv: Kv cache in shad- ows for high-throughput long-context llm inference. Proceedings of ICML. Yunlong Tang, Jing Bi, Siting ...

    arXiv 2025

  2. [4]

    We run 20 entries on the following tasks: Single-NIAH-1/2, MultiKey-NIAH-1/2, MultiValue-NIAH, MultiQuery-NIAH, VT, CWE, FWE, and QA1/2, using 8 GPUs

    using Llama-3.1-8B-Instruct (Grattafiori 15 et al., 2024), and the results are listed be- low. We run 20 entries on the following tasks: Single-NIAH-1/2, MultiKey-NIAH-1/2, MultiValue-NIAH, MultiQuery-NIAH, VT, CWE, FWE, and QA1/2, using 8 GPUs. The input length is set to 128K. We also report the prefill through- put for reference. FULLATTNis implemented ...

    2024

  3. [2024]

    E” and “I

    Longvideobench: A benchmark for long- context interleaved video-language understanding. Proceedings of NeurIPS. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024a. Infllm: Training-free long- context extrapolation for llms with an efficient con- text memory.Proceedings of NeurIPS. Guangxuan ...

    Pith/arXiv arXiv 2021

  4. [2025]

    arXiv:2503.08689

    Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv:2503.08689. MiniCPM-Team. 2025. Minicpm4: Ultra-efficient llms on end devices.arXiv:2506.07900. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, a...

    Pith/arXiv arXiv 2025