Pith. sign in

REVIEW 3 major objections 6 minor 175 references

A 4B streaming vision-language model cuts visual tokens by ~75% via codec motion/residual patches and still matches stronger static baselines while speeding video inference up to 3.5×.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 06:15 UTC pith:72FMYQVD

load-bearing objection Solid codec-native streaming stack with real efficiency numbers; the Qwen3-VL head-to-head overclaims causal credit for the front-end because data and task SFT are confounded. the 3 major comments →

arxiv 2607.24904 v1 pith:72FMYQVD submitted 2026-07-27 cs.CV cs.CL

Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

Senqiao Yang , Kaichen Zhang , Zhaoyang Jia , Jinghao Guo , Yifei Shen , Xinjie Zhang , Xiaoyi Zhang , Haoqing Wang
show 15 more authors
Xiao Li Peng Zhang Xiang An Yin Xie Zhening Liu Xun Guo Jiahao Li Shicheng Zheng Jinglu Wang Zongyu Guo Wenxuan Xie Zihan Zheng Yuxuan Luo Bin Li Yan Lu
This is my paper
classification cs.CV cs.CL
keywords codec-native tokenizationstreaming vision-language modelsMage-ViTmotion vectors and residual energyproactive event gatevisual token efficiencyvideo spatial reasoningAI4AI data pipelines
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard vision-language models are strong at offline image reasoning but wasteful and slow on continuous video because they re-encode mostly static frames. Mage-VL argues the fix is to make the visual front-end codec-native: keep full anchor frames and, on predicted frames, keep only 16×16 patches that motion vectors and residual energy mark as informative, dropping over three-quarters of the tokens while holding spatiotemporal layout via 3D rotary positions. The authors train that encoder, Mage-ViT, from scratch on far less data than flagship image encoders yet match them on probes, then stack a five-stage curriculum and a light System-1 gate that decides when a System-2 language decoder should speak. The resulting 4B model matches a same-scale dense baseline on static tasks, improves video QA, temporal grounding, and 2D/3D spatial scores, and reports up to 3.5× wall-clock speedup. Along the way they package seven empirical recipes: small unlabeled pre-training can suffice, variable resolution scales cleanly, dense captions can replace long VideoQA SFT, motion training helps static spatial skill, closed-loop AI caption pipelines lift quality, and pure-text SFT before multimodal RL can unlock agentic gains.

Core claim

Codec-aligned sparse tokenization—selecting dynamic, entropy-rich 16×16 patches from motion vectors and residual energy on I/P-style frames—lets a from-scratch visual encoder and a 4B dual-system VLM match dense same-scale baselines on static tasks, beat them on video and spatial reasoning, and cut wall-clock inference by up to 3.5× while supporting proactive streaming with a lightweight speak/silent gate.

What carries the argument

Mage-ViT’s codec-driven patchifier: an importance map from HEVC motion-vector magnitude and residual energy (or a neural codec’s bit-allocation map) that keeps all I-frame patches and top-k P-frame patches under a fixed token budget, assembled into canvases with shared 3D rotary positions so a System-1 event gate and System-2 decoder can run on sparse rolling streams.

Load-bearing premise

Gains against the matched-size dense baseline are treated as evidence for the codec front-end, even though the training data mix, caption pipeline, and five-stage recipe also differ.

What would settle it

Train an otherwise identical Mage-VL stack on the same five-stage data but with uniform dense frame sampling at matched token budgets; if video, spatial, and wall-clock advantages over the dense twin disappear, the codec-sparsity claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-video and live perception systems can allocate tokens by temporal unpredictability instead of fixed frame grids and still keep accuracy.
  • Vision encoders for VLMs need not rely on multi-billion image-text pre-training if objectives and image+video joint training are aligned to the downstream interface.
  • Dense timestamped video captions plus short SFT can substitute for heavy long-VideoQA instruction tuning for zero-shot long-video QA.
  • A frozen backbone plus a tiny speak/silent gate is enough to add proactive streaming without a separate online architecture.
  • Cross-codec patch selection (traditional HEVC vs neural codecs) can stay training-free if only the importance map is consumed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If motion/residual importance is truly the right prior, the same selector could front-end non-VLM video stacks (detection, tracking, robotics) without retraining the compressor.
  • Confounding of data recipe and architecture suggests a clean ablate-one-factor leaderboard (codec on/off, captions on/off, gate on/off) would become the decisive community check.
  • Zero-Vision SFT before RL implies post-training compute may be better spent protecting text reasoning than on another round of visual instruction tuning.
  • Edge and always-on cameras are the natural deployment surface: 75% fewer visual tokens plus a silent-most-of-the-time gate maps directly to battery and bandwidth budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Mage-VL-4B, a streaming vision-language model whose visual front-end (Mage-ViT) tokenizes video codec-natively: I-frame patches are kept dense while P-frame patches are selected by codec-derived importance (motion-vector magnitude + residual energy in HEVC, or negative log-likelihood in the neural codec DCVC-RT), cutting visual tokens by ~75%. Mage-ViT is trained from scratch with cluster discrimination against MetaCLIP-derived K-means prototypes on ~560M images and 100M video frames. On top, a five-stage curriculum (captions → instruction → long video → codec-stream → streaming gate) builds the full model, with a System-1 event gate deciding when a frozen System-2 decoder should speak. Reported results: Mage-ViT matches SigLIP2-class encoders on linear/attentive probes (Table 1); Mage-VL-4B matches Qwen3-VL-4B on static image tasks and exceeds it on most video QA, temporal grounding, spatial, and tracking benchmarks (Tables 3–5), with up to 3.5× wall-clock speedups; streaming results on SoccerNet-Caption and OVO-Bench (Tables 6–7); plus ablations on caption-pipeline optimization and a Zero-Vision SFT + RL study. Support is broad: frozen-encoder probes, matched-budget scaling curves (Fig. 8), cross-codec transfer without retraining (Table 2), and a standardized lmms-eval protocol. The central weaknesses are causal attribution: the Table 4 claim that Mage-VL and Qwen3-VL-4B "differ only in the visual front-end" is confounded by large training-data/curriculum

Significance. If the results hold, the work makes three useful contributions: (i) evidence that a codec-aligned sparse front-end preserves representation quality at ~75% fewer tokens, supported by encoder-level probes rather than only end-task scores; (ii) a falsifiable efficiency claim (Fig. 8, Table 5) with matched-token-budget curves and wall-clock numbers on stated hardware; and (iii) an open release of encoder, model, and code, plus a set of clearly enumerated, individually testable empirical findings. The cross-codec transfer experiment and the Zero-Vision SFT result are each of independent interest to the community. Significance is tempered by the causal-attribution problems above: the headline 'front-end-only' comparison is actually a full-recipe comparison, and Finding 1 omits that the pretraining targets are distilled from a web-scale supervised teacher. These are framing/attribution defects rather than evidence that the engineering results are wrong.

major comments (3)
  1. [Table 4 caption and §5.3.2] The caption asserts Mage-VL and Qwen3-VL-4B 'differ only in the visual front-end,' but §4.2–4.3 document large training-mixture differences: ~350M AI4AI-recaptioned images, 7.95M video captions, Stage 4 codec-stream supervision, and task-specific SFT data Qwen3-VL did not use. The numbers make the confound concrete: the +18–20 J&F tracking gains (Ref-DAVIS17 25.83 vs 7.48) coincide with explicit Molmo2-VideoTrack/VideoPoint supervision in Stage 4; the +11.0 VSI-Bench gain coincides with 4M spatial samples; the Timelens gains (+7.6/+17.1/+22.5) coincide with TimeLens/VideoChat-Flash data in Stage 3. A front-end change cannot plausibly yield a 3–8× referring-segmentation jump; supervised tracking data can. The authors themselves concede in §5.3.2 (Finding 4) that single-variable ablations are infeasible — the same caveat must be applied to the Table 4 framing. Either retitle the comparison
  2. [§3.2 (Objective) and Finding 1 (§5.2.1)] Mage-ViT is trained by cluster discrimination against K-means prototypes computed over MetaCLIP features (§3.2, 'Objective'). MetaCLIP is itself trained on billions of image–text pairs, so the representation claim 'trained from scratch on ~560M unlabeled images... matches encoders trained on billions of image–text pairs' (Abstract; Finding 1, §5.2.1) describes distillation from a web-scale-supervised teacher, not label-free learning from unlabeled data alone. This materially weakens Finding 1 as stated ('web-scale pre-training is not essential'). The claim is recoverable in a weaker form (codec-aligned objectives plus prototype distillation are data-efficient), but the text should state the MetaCLIP dependence prominently — abstract, Finding 1, and §5.2.1 — and ideally report a control with self-supervised prototypes (e.g., DINOv2 features) to isolate the effect.
  3. [§5.5, Fig. 8, and Table 5] The 'matched budget' in Fig. 8 matches only encoder token workload: tc-N is built from 8N source frames while frame-N sees N frames, so the codec arm consumes 8× more raw visual information plus an HEVC encode/decode pass whose cost is not accounted for. Table 5 says Mage-VL times are 'full measured wall-clock' — please state explicitly whether codec extraction and patchification are included; if not, the '3.5× speedup' (NextQA: tc8/415s vs Qwen/1460s) is not end-to-end. Also note the flagship speedup compares tc8 (64 source frames) against Qwen at 32 frames, i.e., different information budgets in the opposite direction from Fig. 8. A per-stage timing breakdown (codec, ViT, LLM) and at least one accuracy comparison at equal raw-frame count would make Finding 5 sound.
minor comments (6)
  1. [Table 6] Table 6 mixes evaluation conditions: StreamMind numbers are taken from its publication, JoyAI is evaluated at 1 Hz with a relaxed ±1s window while Mage-VL uses zero-tolerance canvas matching. The TriggerAcc/TimVal comparison across these protocols is not apples-to-apples; please flag this in the caption or re-run baselines under a single protocol.
  2. [Table 7 (OVO-Bench)] The caption says Mage-VL runs at '1fps,' but the text says queries are answered under the SimpleStream recent-window protocol (four most recent frames). Clarify whether Mage-VL was evaluated on codec-token streams or on the 4-frame window, and how the gate/EPFE memory was used (or bypassed) in this offline-style protocol. Backward-Tracing accuracy (48.15%) lagging Real-Time (79.84%) also deserves one sentence of analysis given the streaming-memory design.
  3. [Table 3] Several regressions vs Qwen3-VL-4B are unexplained: CC-OCR Doc (32.25 vs 39.69), DUDE (46.44 vs 50.98), WebSRC (92.80 vs 95.40), TextVQA (77.28 vs 80.55), Charades in Table 5 (31.4 vs 45.9). Given the OCR/document emphasis of the caption pipeline, the CC-OCR gap is surprising and worth a remark.
  4. [Abstract / Tables 3–4] 'Comprehensively surpasses the 15B Phi-4-reasoning-vision baseline' (Abstract) should be qualified: Phi-4-R-V is a reasoning-tuned model not optimized for video/streaming, and it beats Mage-VL on some rows (MMOU 51.9 vs 39.3, ERQA, MMSI in part). The comparison is fine to include but the abstract phrasing overstates it.
  5. [Throughout] Typos/formatting: 'a16× 16patch level' (Abstract); 'Duringlatervideostages' (§4.2.1); Fig. 1 label 'VSI-Bench 64.0' vs Table 4's 64.3; double citation [72]/[100] for StreamMind with different venues; several 2026-dated arXiv references lack links. Appendix A prompt rendering has artifacts (e.g., '‘**bold**‘').
  6. [§5.4] PR-AUC of 9.30% (Table 6) is very low in absolute terms; a sentence contextualizing it against the speak-event base rate on SoccerNet-Caption would help readers calibrate the streaming claim.

Circularity Check

1 steps flagged

Empirical systems paper scored on external public benchmarks; no load-bearing prediction reduces to its inputs by construction.

specific steps
  1. fitted input called prediction [§4.2.1 Caption-prompt optimization; §6.1 Table 8]
    "a rubric-scoring agent powered by GPT-5 evaluates the generated captions along four dimensions: Completeness; Redundancy; Coherence; OCR fidelity. ... The most notable gains are observed on document- and text-heavy benchmarks, such as InfoVQA (+5.62), OCRBench (+3.80), ChartQA (+1.52), and DocVQA (+1.08). This aligns with the explicit OCR quality metric incorporated in the evaluation sub-agent"

    Mild only: the caption prompt is iteratively fitted using an OCR-fidelity rubric, then downstream OCR/document QA gains are presented as pipeline validation. The reported scores are still external benchmark accuracies after full model training, not the rubric scores themselves, so this is correlated data tuning rather than a by-construction identity. Not load-bearing for the codec-native or Mage-ViT claims.

full rationale

Mage-VL’s central claims are model-performance numbers on independent public suites (ImageNet/CIFAR linear probes, DocVQA, VideoMME, VSI-Bench, Timelens, OVO-Bench, SoccerNet-Caption, wall-clock latency on B200). Mage-ViT’s cluster-discrimination objective uses MetaCLIP prototypes as a training signal, not as the reported metric; linear/attentive probes and downstream VLM scores remain external. Codec token reduction (top-k under budget B) is an explicit design choice whose accuracy retention is measured, not assumed. AI4AI prompt optimization uses rubric dimensions (completeness, OCR fidelity) that correlate with later OCR/document gains (Table 8), which is mild data-tuning toward related capabilities but not a by-construction identity between fit and reported score. Streaming gate labels are derived from the authors’ caption timestamps, yet timing/quality are evaluated on SoccerNet and OVO-Bench under fixed protocols. Overlap with OneVision-Encoder / LLaVA-OneVision-2 citations supplies architectural lineage, not a uniqueness theorem that forces the reported results. The Table 4 “differ only in the visual front-end” framing is a causal-confound issue (custom curriculum and task SFT), not circularity: the metrics are not defined to equal the training objective. No equation or claimed first-principles derivation collapses to its fitted inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

This is an empirical systems paper. Load-bearing premises are modeling choices (codec importance as token allocator; dual-process gate; cluster discrimination via MetaCLIP prototypes) and many training hyperparameters/data filters, not a short formal axiom list. Invented pieces are named modules, not new physical entities.

free parameters (5)
  • P-frame top-k / token budget B = B=4096; tc canvases ≈8/16/32-frame workloads
    Default B=4096 tokens for 64-frame clips and tc8/16/32 operating points set the accuracy-efficiency trade-off; chosen as design knobs, not derived.
  • Codec importance weights (motion magnitude + residual energy)
    HEVC selector combines motion-vector magnitude and residual energy with unspecified relative weighting in the main text; defines which patches survive.
  • Gate threshold τ and speak/silent class weights = τ=0.5
    Inference uses τ=0.5; training uses class-weighted CE to handle speak sparsity—both directly control timing metrics.
  • ViT/pretrain hyperparameters = r=0.1; Stage3 frames=384; RoPE θ=8M
    LR schedule (1e-3 then 5e-5), negative-sample ratio r=0.1, resolution ranges, mix 50/40/10 codec/chunk/collage, RoPE base 8M, Stage-3 384 frames—recipe choices that affect reported quality.
  • K-means prototype count / MetaCLIP feature space for cluster discrimination
    Pretraining targets nearest MetaCLIP-derived clusters; cluster granularity and teacher features are free design choices that shape Mage-ViT.
axioms (5)
  • domain assumption Codec bit allocation (motion/residual or neural NLL) is a sufficient proxy for spatio-temporal importance for downstream VLM tasks.
    Stated as the codec-aligned sparsity principle (§3); entire token reduction claim rests on this proxy not dropping task-critical static detail.
  • domain assumption A still image is a degenerate one-frame video; shared 3D RoPE preserves geometry after aggressive patch drop.
    §3.1 architecture premise enabling one encoder for image/video/stream modes.
  • domain assumption Proactive streaming can be factored into a frozen System-2 VLM plus a lightweight System-1 speak/silent gate on rolling memory.
    §4.1/Fig. 3 bio-inspired dual-system design; Stage 5 freezes the base model and only trains the gate.
  • standard math Standard transformer/ViT training mathematics (AdamW, attention, CE losses) behave as usual at this scale.
    Background optimization assumptions used throughout §§3–4.
  • ad hoc to paper Dense timestamped captions can be deterministically converted into causal speak/silent supervision without future frames.
    §4.2.3 caption converter following StreamMind-style event gating; defines streaming labels used in Stage 5.
invented entities (3)
  • Mage-ViT codec-driven patchifier + canvas independent evidence
    purpose: Select and pack I-frame-all / P-frame-top-k 16×16 patches before the ViT trunk.
    Core named module; builds on OneVision-style sparsity but is the paper’s central artifact.
  • Cognition gate / System 1 event gate (+ EPFE streaming memory) independent evidence
    purpose: Predict silent vs speak on causal codec windows and trigger the frozen decoder.
    Introduced for proactive streaming; evaluated on SoccerNet-Caption and qualitative live commentary.
  • AI4AI prompt-code co-optimization loop independent evidence
    purpose: Iteratively rewrite caption prompts/code via rubric agents to raise mid-training data quality.
    Methodological entity; Table 8 shows downstream lifts after optimization.

pith-pipeline@v1.2.0-grok45-kimik3 · 47468 in / 4048 out tokens · 101813 ms · 2026-07-31T06:15:49.894823+00:00 · methodology

0 comments
read the original abstract

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

175 extracted references · 67 linked inside Pith

  1. [1]

    Claude fable 5 & claude mythos 5 system card

    Anthropic. Claude fable 5 & claude mythos 5 system card. Technical report, Anthropic, 6

  2. [2]

    Gemini 3 pro model card

    Google DeepMind. Gemini 3 pro model card. Technical report, Google Deep- Mind, 11 2025. URLhttps://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf

    2025

  3. [3]

    Gpt-5.5 system card

    OpenAI. Gpt-5.5 system card. Technical report, OpenAI, 4 2026. URLhttps://openai. com/index/gpt-5-5-system-card/

    2026

  4. [4]

    Harvard University Press, 1988

    Hans Moravec.Mind Children: The Future of Robot and Human Intelligence. Harvard University Press, 1988

    1988

  5. [5]

    Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    Pith/arXiv arXiv 2025

  6. [6]

    Qwen2.5-VL technical report.arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenw...

    Pith/arXiv arXiv 2025

  8. [8]

    LLaVA-OneVision-1.5: A family of fully open vision–language models with the llava-onevision-1.5 mid-training and instruct data.arXiv preprint, 2025

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision-1.5: A family of fully open vision–language models with the llava-onevision-1.5 mid-training and instruct data.arXiv preprint, 2025

    2025

  9. [9]

    Kwai keye-vl 1.5 technical report.arXiv:2509.01563, 2025

    Biao Yang, Bin Wen, Boyang Ding, et al. Kwai keye-vl 1.5 technical report.arXiv:2509.01563, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. InNeurIPS, 2024

    2024

  11. [12]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, 33 et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  12. [13]

    MIT press, 2015

    Peter Sterling and Simon Laughlin.Principles of neural design. MIT press, 2015

    2015

  13. [14]

    Energy limitation as a selective pressure on the evolution of sensory systems.Journal of Experimental Biology, 211(11):1792–1804, 2008

    Jeremy E Niven and Simon B Laughlin. Energy limitation as a selective pressure on the evolution of sensory systems.Journal of Experimental Biology, 211(11):1792–1804, 2008

    2008

  14. [15]

    Eye smarter than scientists believed: neural computations in circuits of the retina.Neuron, 65(2):150–164, 2010

    Tim Gollisch and Markus Meister. Eye smarter than scientists believed: neural computations in circuits of the retina.Neuron, 65(2):150–164, 2010

    2010

  15. [16]

    Separate visual pathways for perception and action

    Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20–25, 1992

    1992

  16. [17]

    Macmillan, 2011

    Daniel Kahneman.Thinking, fast and slow. Macmillan, 2011

    2011

  17. [18]

    OneVision-Encoder: Codec-aligned sparsity as a foundational principle for multimodal intelligence.arXiv preprint arXiv:2602.08683, 2026

    Feilong Tang, Xiang An, Yunyao Yan, Yin Xie, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Chunyuan Li, Shikun Feng, Changrui Chen, Huajie Tan, Ming Hu, Manyuan Zhang, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, and Jiankang Deng. OneVision-Encoder: Codec-aligned sparsity as a foundational principle for multimodal intelligence.arXiv preprint arXiv:26...

    arXiv 2026

  18. [19]

    Deep contextual video compression (DCVC).NeurIPS, 2021

    Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression (DCVC).NeurIPS, 2021

    2021

  19. [20]

    LLaVA-OneVision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026

    Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Chunyuan Li, Bo Li, Ziwei Liu, Jiankang Deng, et al. LLaVA-OneVision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026

    Pith/arXiv arXiv 2026

  20. [21]

    Phi-4-Mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin...

    Pith/arXiv arXiv 2025

  21. [22]

    Phi-4-reasoning-vision-15b technical report, 2026

    Jyoti Aneja, Michael Harrison, Neel Joshi, Tyler LaBonte, John Langford, and Eduardo Salinas. Phi-4-reasoning-vision-15b technical report, 2026. URLhttps://arxiv.org/abs/ 2603.03975

    arXiv 2026

  22. [23]

    The plenoptic function and the elements of early vision.Computational models of visual processing, 1(8):3, 1991

    James R Bergen and Edward H Adelson. The plenoptic function and the elements of early vision.Computational models of visual processing, 1(8):3, 1991

    1991

  23. [24]

    Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026

    Yifei Shen, Bo Li, and Xinjie Zhang. Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026. 34

    Pith/arXiv arXiv 2026

  24. [25]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  25. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021

    2021

  26. [27]

    Sigmoid loss for language image pre-training.ICCV, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.ICCV, 2023

    2023

  27. [28]

    DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research (TMLR), 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research (TMLR), 2024

    2024

  28. [29]

    DynamicViT: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efficient vision transformers with dynamic token sparsification. InNeurIPS, 2021

    2021

  29. [30]

    AdaViT: Adaptive vision transformers for efficient image recognition

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. AdaViT: Adaptive vision transformers for efficient image recognition. In CVPR, 2022

    2022

  30. [31]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InICLR, 2023

    2023

  31. [32]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  32. [33]

    LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. InIEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  33. [34]

    Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021

    2021

  34. [35]

    ViViT: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViViT: A video vision transformer. InIEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  35. [36]

    Video Swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video Swin transformer. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  36. [37]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  37. [38]

    LLaMA-VID: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023. 35

    Pith/arXiv arXiv 2023

  38. [39]

    Chat-UniVi: Unified visual representation empowers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-UniVi: Unified visual representation empowers large language models with image and video under- standing. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  39. [40]

    SlowFast-LLaVA: A strong training-free baseline for video large language models.arXiv preprint arXiv:2407.15841, 2024

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. SlowFast-LLaVA: A strong training-free baseline for video large language models.arXiv preprint arXiv:2407.15841, 2024

    Pith/arXiv arXiv 2024

  40. [41]

    MovieChat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. MovieChat: From dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  41. [42]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. LongVU: Spatiotemporal adaptive compression for long video-language understanding.arXiv preprint...

    Pith/arXiv arXiv 2024

  42. [43]

    VideoChat-Flash: Hierarchical compression for long-context video model- ing.arXiv preprint arXiv:2501.00574, 2025

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Chen, Yinan Zhu, Haian Sun, Yali He, Yu Wang, and Limin Wang. VideoChat-Flash: Hierarchical compression for long-context video model- ing.arXiv preprint arXiv:2501.00574, 2025

    Pith/arXiv arXiv 2025

  43. [44]

    Manmatha, Alexander J

    Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Compressed video action recognition. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  44. [45]

    Video-LaVIT: Unified video-language pre-training with decoupled visual-motional tokenization.ICML, 2024

    Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, and Yadong Mu. Video-LaVIT: Unified video-language pre-training with decoupled visual-motional tokenization.ICML, 2024

    2024

  45. [46]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  46. [47]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  47. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023

    2023

  48. [49]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Pith/arXiv arXiv 2023

  49. [50]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Process- ing Systems (NeurIPS), 2023. 36

    2023

  50. [51]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  51. [52]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Pith/arXiv arXiv 2023

  52. [53]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024

    2024

  53. [54]

    DeepSeek-VL: Towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. DeepSeek-VL: Towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525, 2024

    Pith/arXiv arXiv 2024

  54. [55]

    Pixtral 12B.arXiv preprint arXiv:2410.07073, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12B.arXiv preprint arXiv:2410.07073, 2024

    Pith/arXiv arXiv 2024

  55. [56]

    Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    Pith/arXiv arXiv 2024

  56. [57]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  57. [58]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Pith/arXiv arXiv 2024

  58. [59]

    BAGEL: A unified multimodal foundation model for image understand- ing, generation, and editing.arXiv preprint, 2025

    Chaorui Deng et al. BAGEL: A unified multimodal foundation model for image understand- ing, generation, and editing.arXiv preprint, 2025

    2025

  59. [60]

    Emu3.5: Native multimodal models.arXiv preprint, 2025

    Yuying Cui et al. Emu3.5: Native multimodal models.arXiv preprint, 2025

    2025

  60. [61]

    Ovis-u1 technical report

    Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, et al. Ovis-u1 technical report. arXiv preprint arXiv:2506.23044, 2025

    Pith/arXiv arXiv 2025

  61. [62]

    Unified multimodal understanding and generation models: Advances, challenges, and opportunities

    Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Jiakui Hu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567, 2025

    arXiv 2025

  62. [63]

    VideoChat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    Pith/arXiv arXiv 2023

  63. [64]

    Video- ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- ChatGPT: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 37

    Pith/arXiv arXiv 2023

  64. [65]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Pith/arXiv arXiv 2023

  65. [66]

    Video- LLaVA: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video- LLaVA: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Pith/arXiv arXiv 2023

  66. [67]

    LLaVA-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  67. [68]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Pith/arXiv arXiv 2024

  68. [69]

    VideoLLM-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. VideoLLM-online: Online video large language model for streaming video. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  69. [70]

    VideoLLM knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 2024

    Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. VideoLLM knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 2024

    arXiv 2024

  70. [71]

    Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.arXiv preprint arXiv:2501.03218, 2025

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.arXiv preprint arXiv:2501.03218, 2025

    Pith/arXiv arXiv 2025

  71. [72]

    Streammind: Unlocking full frame rate streaming video dialogue through event- gated cognition

    Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, and Ting Cao. Streammind: Unlocking full frame rate streaming video dialogue through event- gated cognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13448–13459, 2025

    2025

  72. [73]

    Joyai-vl-interaction: Real-time vision-language interaction intelligence

    Video Understanding Team of JoyAI-VL @ Joy Future Academy, JD. Joyai-vl-interaction: Real-time vision-language interaction intelligence. Technical report, Joy Future Academy, JD, June 2026

    2026

  73. [74]

    Flash-VStream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-VStream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Pith/arXiv arXiv 2024

  74. [75]

    Streamingvlm: Real-time understanding for infinite video streams, 2026

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams, 2026. URLhttps://arxiv.org/abs/ 2510.09608

    Pith/arXiv arXiv 2026

  75. [76]

    InternLM-XComposer2.5-OmniLive: A comprehensive multimodal system for long-term streaming video and audio interactions.arXiv preprint arXiv:2412.09596, 2024

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. InternLM-XComposer2.5-OmniLive: A comprehensive multimodal system for long-term streaming video and audio interactions.arXiv preprint arXiv:2412.09596, 2024

    Pith/arXiv arXiv 2024

  76. [77]

    Livecc: Learning video llm with streaming speech transcription at scale, 2025

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale, 2025. URLhttps://arxiv.org/ abs/2504.16030. 38

    Pith/arXiv arXiv 2025

  77. [78]

    Matchtime: Towards automatic soccer game commentary generation, 2024

    Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. Matchtime: Towards automatic soccer game commentary generation, 2024. URLhttps://arxiv.org/abs/2406. 18530

    2024

  78. [79]

    StreamingBench: Assessing the gap for mllms to achieve streaming video understanding.arXiv preprint arXiv:2411.03628, 2024

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. StreamingBench: Assessing the gap for mllms to achieve streaming video understanding.arXiv preprint arXiv:2411.03628, 2024

    Pith/arXiv arXiv 2024

  79. [80]

    OVO-Bench: How far is your video-llms from real-world online video understanding? InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. OVO-Bench: How far is your video-llms from real-world online video understanding? InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  80. [81]

    Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand

    Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. InIEEE Transactions on Circuits and Systems for Video Technology, 2012

    2012

Showing first 80 references.