Pith. sign in

REVIEW 3 major objections 4 minor 16 references

Think in Sets for Streaming Video Token Compression

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NovaCov reframes streaming video token compression as set selection: a token is worth keeping only for what it adds beyond the content the model already holds.

desk verdict A genuinely new formulation—streaming token compression as set selection with a bounded historical reference—with a clean submodularity proof and solid gains, but reproducibility gaps (no code, no error bars, seven hand-set hyperparameters) keep it at conditional accept. read the letter →

arxiv 2608.01169 v1 pith:KDDS6TT2 submitted 2026-08-02 cs.CV

Moxu Duan , Jingwen Fu , Yuwang Wang This is my paper
classification cs.CV
keywords streamingvideounderstandingtokencompressionsetselectionsubmodularmaximizationfacility-locationcoveragetraining-freehistoricalreferencebankgreedyapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Streaming video models must drop most visual tokens of every arriving frame to keep latency and memory bounded, but existing training-free compressors score each token in isolation and can waste the budget on near-duplicate content. The paper argues that compression should be a set-selection problem: a token is worth keeping only for what it adds beyond the tokens already retained. To do this in a causally constrained stream, NovaCov maintains a fixed-capacity, recency-weighted bank of historical prototypes and greedily selects tokens under a dual facility-location objective that covers the current frame while favoring content the history has not yet covered. On streaming benchmarks it keeps 99.6% of the uncompressed ReKV accuracy while cutting LLM prefilling latency by 46%, and it also improves over earlier training-free methods on offline long-video benchmarks. The payoff is a plug-in selection rule that makes a streaming VideoLLM nearly as accurate as an uncompressed one at a fraction of the compute.

What carries the argument

The Historical Reference Bank: a capacity-bounded set of prototypes, each an EMA-refined vector with a last-update frame and match count, updated by match-or-insert and pruned by recency-aware eviction with utility $u_j = 2^{-(t-\tau_j)/\lambda}\log(1+n_j)$. It supplies the historical term of the reference set. The dual-branch objective $F_t(S) = \lambda_c \sum_i \rho_i c_i(S) + \lambda_r \sum_i \max(c_i(S)-h_i,0)$ is a weighted sum of two facility-location functions, one over the current frame and one over the residual coverage above history, preserving monotone submodularity.

What would settle it

Construct a stream in which an object appears, disappears for longer than the bank's half-life (16 frames), then reappears; if eviction removes its prototype, the reappearing tokens are treated as novel rather than already-conveyed, so a repetition-counting task should lose accuracy relative to a bank with a much longer half-life or larger capacity. This directly tests whether the recency-weighted bank faithfully represents the historical reference.

Watch

Extended reading notes

Core claim

NovaCov's central claim is that streaming video token compression is a subset-selection problem whose reference set must include the content already forwarded to the model, and that a bounded, recency-weighted Historical Reference Bank is enough to serve as that reference. Each incoming token is scored by its historical coverage $h_i$, the similarity to the most similar bank prototype, decayed by recency. The selection objective combines a current-frame facility-location branch weighted by novelty with a residual branch that credits a token only for coverage above $h_i$; both branches are monotone submodular, so greedy selection keeps the $(1-1/e)$ approximation guarantee. The paper reports

Load-bearing premise

The Historical Reference Bank, with its fixed capacity, match threshold, EMA rate, and decay half-life, faithfully summarizes everything the model has already been shown, so the historical coverage $h_i$ correctly identifies redundant tokens; the paper checks this only empirically and gives no sensitivity analysis over these hyperparameters.

Editorial extensions

If this is right

  • At 25% retention, NovaCov exceeds the uncompressed ReKV accuracy on five of ten StreamingBench tasks, showing that removing redundant tokens can reduce noise and improve cross-frame reasoning.
  • The method is training-free and plug-and-play, so it can be dropped into any streaming VideoLLM pipeline that already uses per-frame KV-cache retrieval, without retraining.
  • The bounded bank keeps per-frame cost independent of stream length, so the latency advantage grows with stream duration.
  • The dual-branch objective and greedy selection carry a $(1-1/e)$ worst-case guarantee, which token-wise ranking cannot offer.
  • The same compressor transfers to offline long-video understanding, improving all three offline benchmarks over previous training-free methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reference-set principle suggests that other streaming components, such as KV-cache eviction or retrieval indices, could benefit from the same 'what has the model already seen' criterion, potentially unifying token and cache compression.
  • Because the bank's recency decay explicitly trades memory for recency, NovaCov's design quantifies a general forgetting strategy; one could tune the half-life to match a domain's typical reappearance timescales, which the current fixed $\lambda=16$ leaves unexamined.
  • The residual branch formulation—shifting the similarity by $h_i$—may extend to non-facility-location objectives (e.g., coverage with determinantal point processes) while preserving a similar approximation analysis.
  • A testable extension is to apply the same historical-reference idea to audio or LiDAR tokens in streaming multimodal models, where the same cross-frame redundancy pattern should appear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes NovaCov, a training-free, plug-and-play token compressor for streaming video. It maintains a capacity-bounded, recency-weighted Historical Reference Bank of prototypes, updated by a match-or-insert EMA rule and recency-aware eviction. At each frame it greedily maximizes a dual-branch submodular coverage objective: a novelty-weighted current-frame facility-location branch plus a historical-novelty residual branch that credits only coverage beyond what the bank already supplies. The paper proves that both branches are facility-location functions, so greedy selection retains the (1-1/e) approximation guarantee. Evaluations on OVO-Bench, StreamingBench, EgoSchema, MLVU-dev, and VideoMME report that NovaCov retains 99.0-99.6% of uncompressed ReKV accuracy while reducing LLM prefilling latency by ~46%, outperforming existing training-free compressors.

Significance. If the empirical claims hold, NovaCov is a useful contribution: it is, to my knowledge, the first set-wise streaming token compressor, and it cleanly reduces the problem to a weighted facility-location objective with a classical approximation guarantee. The proof in Proposition 1 is correct and the benchmark coverage is broad, spanning streaming and offline long-video understanding. The main reservation is that the central mechanism—the Historical Reference Bank—is validated only indirectly, and all headline numbers are single point estimates without sensitivity analysis or error bars. Since the residual branch and the novelty weights both depend on the bank's coverage scores h_i, the strength of the empirical SOTA claim is not yet established.

major comments (3)
  1. [Historical Reference Bank, Eq. (6), Table 4d] The Bank's fidelity is load-bearing: h_i in Eq. (6) directly controls the residual branch in Eq. (8) and the novelty weights in Eq. (7). The only ablation aimed at the bank, Table 4d, is labeled 'Unbounded', but it still uses EMA/threshold prototypes and simply never evicts; it is not the exact history R*_t of Eq. (3). Thus the paper never tests whether the bounded bank faithfully represents the content already forwarded to the LLM. With defaults C=512, λ=16, θ=0.9, α=0.10, λc=0.1, λr=0.9, ρmin=0.20 and no sensitivity analysis, the claimed 4.7-point StreamingBench gain over STC-Pruner rests on a single hand-set point in a seven-dimensional space. Please add sensitivity curves for the bank hyperparameters, and an oracle variant that computes h_i against the true history (for short streams) to validate the bank design.
  2. [Experiments, Tables 1–3] All tables report single point estimates. The headline improvements over STC-Pruner are 4.7 on StreamingBench, 2.1 on OVO-Bench, and 0.5 on the offline average; these are modest enough that run-to-run variance could change the ranking on some benchmarks. Please report means and standard deviations over multiple runs or seeds, or otherwise provide statistical evidence that the observed differences are not noise. Without this, the SOTA claim is under-supported.
  3. [Table 4d and 'Bounded vs. unbounded'] The text states that the unbounded variant's cost 'rises without bound', but no latency or memory measurements are reported for variant (d). More importantly, because (d) still uses the same EMA/threshold update, it does not isolate the effect of boundedness on bank fidelity. The accuracy drop from 68.4 to 67.4 is attributed to saturation, but an alternative explanation is that unbounded growth changes the h_i distribution without approaching the true history. Please directly compare against the exact-history oracle and measure cost for the unbounded variant.
minor comments (4)
  1. [Main Comparisons, Table 3] The claim that NovaCov 'outperforms every compression method on all five benchmarks' is too strong: on VideoMME Short, NovaCov ties with VidCom2 and STC-Pruner at 68.7, and several task-level numbers in Tables 1–2 are below a baseline. Please qualify the claim to aggregate or average performance.
  2. [Eq. (7)] The rank normalization RankNorm(1-h_i) is not specified in detail. Please define how ties are handled and how ranks are mapped to [0,1].
  3. [Implementation details] No code release or reproducibility statement is mentioned. Given the number of hand-set hyperparameters, releasing code or at least detailed per-task settings would substantially improve reproducibility.
  4. [Figure 1] The y-axis label 'Accuracy Performance (%)' is redundant; 'Accuracy (%)' would be clearer. The legend is also hard to read at small sizes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NovaCov's derivation is self-contained, with claims supported by external benchmarks and standard submodularity theory.

full rationale

The paper's central derivation is the dual-branch coverage objective in Eq. (8) and the submodularity guarantee in Proposition 1. The objective is defined over the current frame and the Historical Reference Bank, where the bank is updated from previously selected tokens. This is an algorithmic feedback loop, not a logical circularity: the bank does not define the prediction target (benchmark accuracy) and is not fitted to the evaluation data. The proof that F_t is monotone submodular is direct and relies only on the classical facility-location result (Krause and Golovin), with the residual branch correctly shown to be a shifted facility-location function. The empirical claims (e.g., 99.6% ReKV accuracy, 46% latency reduction) are measurements on external benchmarks (OVO-Bench, StreamingBench, EgoSchema, MLVU, VideoMME) using fixed hyperparameters; no parameter is fitted to the test labels and then reported as a prediction. The only self-referential aspect is the online update of the bank from selected tokens, which is a standard algorithmic mechanism and is ablated (Table 4d) against an unbounded bank. No load-bearing self-citations, uniqueness claims, or ansatz-smuggling are present. The lack of sensitivity analysis is a robustness concern, not circularity. Thus the derivation chain is substantive and independent.

Assumptions & free parameters 7 free parameters · 4 assumptions · 1 invented entities

The central claim rests on standard submodularity theorems, a domain assumption that cosine similarity in token embedding space captures redundancy, and an empirical assumption that the bounded Historical Reference Bank faithfully represents the already-forwarded context. The paper does not release code or data, and the hand-set hyperparameters are not swept, which limits external verification.

free parameters (7)
  • Bank capacity C = 512
    Hand-set; controls the boundedness of the historical reference. No sensitivity analysis.
  • Half-life λ for recency decay = 16 frames
    Hand-set; controls how quickly prototypes fade. No sensitivity analysis.
  • Match threshold θ = 0.9
    Hand-set; controls when a token matches a prototype. No sensitivity analysis.
  • EMA coefficient α = 0.10
    Hand-set; controls prototype refinement rate. No sensitivity analysis.
  • Branch weight λ_c = 0.1
    Hand-set; relative weight of current-frame branch. No sensitivity analysis.
  • Branch weight λ_r = 0.9
    Hand-set; relative weight of historical-novelty branch. No sensitivity analysis.
  • Novelty floor ρ_min = 0.20
    Hand-set; minimum demand weight for any token. No sensitivity analysis.
assumptions (4)
  • standard math Facility-location functions f(S) = Σ_i max_{s∈S} A(i,s) with non-negative A are monotone submodular, and greedy maximization under cardinality constraint achieves (1-1/e) approximation.
    Invoked in Theoretical Analysis, Proposition 1, citing Krause and Golovin 2014 and Nemhauser et al. 1978.
  • domain assumption Cosine similarity between L2-normalized visual token embeddings, clamped to [0,1], measures semantic/content redundancy.
    Used to define A and h_i in Eqs. (4)-(6); the effectiveness of coverage relies on this.
  • domain assumption The Historical Reference Bank, updated with EMA and recency-weighted eviction, faithfully summarizes the content already forwarded to the LLM.
    Central to the historical-novelty branch; validated only empirically, not formally.
  • domain assumption Compression decisions are causal and query-agnostic; only current frame and history are available.
    Problem formulation in Method section.
invented entities (1)
  • Historical Reference Bank
    purpose: Bounded, recency-weighted prototype set summarizing previously forwarded visual content; supplies the history term in the reference set for marginal-gain computation.
    Algorithmic construct introduced by this paper; its adequacy is validated only through the paper's own benchmarks, no external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Think in Sets for Streaming Video Token Compression." pith.science (2026). https://pith.science/paper/KDDS6TT2

@misc{pith2026260801169,
  author       = {Pith},
  title        = {Pith review of: Think in Sets for Streaming Video Token Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDDS6TT2}},
  note         = {Machine review of arXiv:2608.01169}
}
read the original abstract

Streaming VideoLLMs process frames causally while visual tokens grow continuously, making compression essential for controlling prefilling latency and memory. Existing training-free methods independently rank tokens, ignoring marginal-gain interactions among retained tokens. We argue that streaming video token compression should instead be formulated as set selection, where each candidate is valued by what it adds beyond the tokens already retained. Unlike existing set-wise methods designed for offline tasks, streaming makes causal, frame-by-frame pruning decisions, so modeling cross-frame interactions requires an explicit historical reference. This creates a reference-set dilemma: the reference must adequately represent previously conveyed content while remaining bounded for real-time inference. We introduce NovaCov, to our knowledge the first training-free, plug-and-play set-wise token compressor designed for streaming video. NovaCov maintains a capacity-bounded, recency-weighted Historical Reference Bank and optimizes a dual-branch submodular coverage objective that preserves representative current-frame content while prioritizing information insufficiently covered by history. Both branches are facility-location functions, so greedy selection retains the classical (1-1/e) approximation guarantee. Across streaming and offline benchmarks, NovaCov outperforms existing training-free compression methods, retaining 99.6% of ReKV accuracy while reducing LLM prefilling latency by 46%.

Figures

Figures reproduced from arXiv: 2608.01169 by the authors.

Figure 1
Figure 1. Accuracy–budget trade-off on OVO-Bench under [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visual diagnosis of the need for an explicit historical reference in streaming set-wise selection. Left: historical coverage [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of NovaCov. A per-frame cycle between token selection and bank update. Left: each current-frame token [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages

  1. [3]

    Feige, U

    Mmtok: Multimodal coverage maximization for efficientinferenceofvlms.arXivpreprintarXiv:2508.18264. Feige, U

    arXiv

  2. [6]

    InICASSP 2026-2026 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 12147–12151

    Streaming- bench: Assessing the gap for mllms to achieve streaming video understanding. InICASSP 2026-2026 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), 12147–12151. IEEE. Liu, J.; Yu, Z.; Lan, S.; Wang, S.; Fang, R.; Kautz, J.; Li, H.; and Alvarez, J. M

    work page 2026

  3. [7]

    arXiv:2412.08646

    StreamChat: Chatting with Streaming Video. arXiv:2412.08646. Liu, X.; Wang, Y.; Ma, J.; and Zhang, L

    arXiv

  4. [8]

    InProceedingsofthe2025 ConferenceonEmpiricalMethodsinNaturalLanguagePro- cessing, 1910–1924

    Video com- pression commander: Plug-and-play inference acceleration forvideolargelanguagemodels. InProceedingsofthe2025 ConferenceonEmpiricalMethodsinNaturalLanguagePro- cessing, 1910–1924. Mangalam, K.; Akshulakov, R.; and Malik, J

    work page 1910

  5. [10]

    arXiv:2505.15269

    LiveVLM: Efficient Online Video Under- standing via Streaming-Oriented KV Cache and Retrieval. arXiv:2505.15269. Niu,J.;Li,Y.;Miao,Z.;Ge,C.;Zhou,Y.;He,Q.;Dong,X.; Duan, H.; Ding, S.; Qian, R.; Zhang, P.; Zang, Y.; Cao, Y.; He, C.; and Wang, J

    arXiv

  6. [11]

    Ren,S.;Chen,S.;Li,S.;Sun,X.;andHou,L.2023

    OVO-Bench: How Far Are Your Video-LLMsfromReal-WorldOnlineVideoUnderstanding? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18902–18913. Ren,S.;Chen,S.;Li,S.;Sun,X.;andHou,L.2023. TESTA: Temporal-Spatial Token Aggregation for Long-form Video- Language Understanding. InFindings of the Association for Computational Lin...

    work page 2023

  7. [13]

    Xie, Y.; He, B.; Wang, J.; Zheng, X.; Ye, Z.; and Wu, Z

    Streamvln: Streaming vision-and-language navigation via slowfast con- text modeling.arXiv preprint arXiv:2507.05240. Xie, Y.; He, B.; Wang, J.; Zheng, X.; Ye, Z.; and Wu, Z

    arXiv

  8. [14]

    Yang,S.;Chen,Y.;Tian,Z.;Wang,C.;Li,J.;Yu,B.;andJia, J

    Fluxmem: Adaptive hierarchical memory for stream- ing video understanding.arXiv preprint arXiv:2603.02096. Yang,S.;Chen,Y.;Tian,Z.;Wang,C.;Li,J.;Yu,B.;andJia, J. 2025a. VisionZip: Longer is Better but Not Necessary in Vision Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19792–19802. Yang, Y.; ...

    arXiv

Show all 16 references
  1. [15]

    InProceedings of the 33rd ACM International Conference on Multimedia, 10807–10816

    TimeChat-Online: 80% Visual Tokens Are Naturally Redundant in Streaming Videos. InProceedings of the 33rd ACM International Conference on Multimedia, 10807–10816. Zhang, B.; Li, K.; Cheng, Z.; Hu, Z.; Yuan, Y.; Chen, G.; Leng, S.; Jiang, Y.; Zhang, H.; Li, X.; Jin, P.; Zhang, ...

    arXiv

  2. [16]

    arXiv:2410.02713

    Video Instruction Tuning with Synthetic Data. arXiv:2410.02713. Zhou, J.; Shu, Y.; Zhao, B.; Wu, B.; Liang, Z.; Xiao, S.; Qin, M.; Yang, X.; Xiong, Y.; Zhang, B.; Huang, T.; and Liu,Z.2025. MLVU:BenchmarkingMulti-TaskLongVideo Understanding. InProceedingsoftheIEEE/CVFConferenc...

    2025 arXiv

  3. [1998]

    A Threshold oflnnfor Approximating Set Cover.Journal of the ACM, 45(4): 634–652. Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; Zhang, R.; Wang, Z.; Zhou, C.; Shen, Y.; Zhang, M.; Chen, P.; Li, Y.; Lin, S.; Zhao,S.;Li,K.;Xu,T.;Zheng,X.;Chen,E.;Shan,C.;He,R.; andSun,X.2025. Video-M...

    2025

  4. [2019]

    InProceedings of the 3rd ACM SIGPLAN InternationalWorkshoponMachineLearningandProgram- ming Languages (MAPL), 10–19

    Triton: An Inter- mediate Language and Compiler for Tiled Neural Network Computations. InProceedings of the 3rd ACM SIGPLAN InternationalWorkshoponMachineLearningandProgram- ming Languages (MAPL), 10–19. Wang, S.; Chen, G.; Huang, D.-A.; Li, Z.; Li, M.; Liu, G.; Kautz, J.; Alv...

    2025 arXiv

  5. [2023]

    Nemhauser,G.L.;Wolsey,L.A.;andFisher,M.L.1978

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Infor- mation Processing Systems, 36: 46212–46244. Nemhauser,G.L.;Wolsey,L.A.;andFisher,M.L.1978. An AnalysisofApproximationsforMaximizingSubmodularSet Functions—I.Mathematical...

    1978

  6. [2024]

    InEuropean Conference on Computer Vision (ECCV), 19–35

    An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision- Language Models. InEuropean Conference on Computer Vision (ECCV), 19–35. Chen, X.; Tao, K.; Shao, K.; and Wang, H. 2025a. Stream- ingtom:Streamingtokencompressionforefficientvide...

    2026

  7. [2025]

    arXiv:2502.13923

    Qwen2.5-VL Technical Report. arXiv:2502.13923. Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J

    arXiv

  8. [2026]

    Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C

    Moving Beyond Diversity:VisualTokenPruningasSubspaceReconstruction for Efficient VLMs.arXiv preprint arXiv:2606.18681. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C. 2024a. LLaVA-OneVision: Easy Visual Task Transfer....

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.