Pith. sign in

REVIEW 3 major objections 6 minor 175 references

Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

T0 review · 3 major / 6 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 4B streaming vision-language model cuts visual tokens by ~75% via codec motion/residual patches and still matches stronger static baselines while speeding video inference up to 3.5×.

desk verdict Solid codec-native streaming stack with real efficiency numbers; the Qwen3-VL head-to-head overclaims causal credit for the front-end because data and task SFT are confounded. read the letter →

arxiv 2607.24904 v1 pith:72FMYQVD submitted 2026-07-27 cs.CV cs.CL

Senqiao Yang , Kaichen Zhang , Zhaoyang Jia , Jinghao Guo , Yifei Shen , Xinjie Zhang , Xiaoyi Zhang , Haoqing Wang
show 15 more authors
Xiao Li Peng Zhang Xiang An Yin Xie Zhening Liu Xun Guo Jiahao Li Shicheng Zheng Jinglu Wang Zongyu Guo Wenxuan Xie Zihan Zheng Yuxuan Luo Bin Li Yan Lu
This is my paper · ORCID
classification cs.CVcs.CL
keywords codec-nativetokenizationstreamingvision-languagemodelsMage-ViTmotionvectorsandresidualenergyproactiveeventgatevisualtokenefficiencyvideospatialreasoningAI4AIdatapipelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard vision-language models are strong at offline image reasoning but wasteful and slow on continuous video because they re-encode mostly static frames. Mage-VL argues the fix is to make the visual front-end codec-native: keep full anchor frames and, on predicted frames, keep only 16×16 patches that motion vectors and residual energy mark as informative, dropping over three-quarters of the tokens while holding spatiotemporal layout via 3D rotary positions. The authors train that encoder, Mage-ViT, from scratch on far less data than flagship image encoders yet match them on probes, then stack a five-stage curriculum and a light System-1 gate that decides when a System-2 language decoder should speak. The resulting 4B model matches a same-scale dense baseline on static tasks, improves video QA, temporal grounding, and 2D/3D spatial scores, and reports up to 3.5× wall-clock speedup. Along the way they package seven empirical recipes: small unlabeled pre-training can suffice, variable resolution scales cleanly, dense captions can replace long VideoQA SFT, motion training helps static spatial skill, closed-loop AI caption pipelines lift quality, and pure-text SFT before multimodal RL can unlock agentic gains.

What carries the argument

Mage-ViT’s codec-driven patchifier: an importance map from HEVC motion-vector magnitude and residual energy (or a neural codec’s bit-allocation map) that keeps all I-frame patches and top-k P-frame patches under a fixed token budget, assembled into canvases with shared 3D rotary positions so a System-1 event gate and System-2 decoder can run on sparse rolling streams.

What would settle it

Train an otherwise identical Mage-VL stack on the same five-stage data but with uniform dense frame sampling at matched token budgets; if video, spatial, and wall-clock advantages over the dense twin disappear, the codec-sparsity claim fails.

Watch

Extended reading notes

Core claim

Codec-aligned sparse tokenization—selecting dynamic, entropy-rich 16×16 patches from motion vectors and residual energy on I/P-style frames—lets a from-scratch visual encoder and a 4B dual-system VLM match dense same-scale baselines on static tasks, beat them on video and spatial reasoning, and cut wall-clock inference by up to 3.5× while supporting proactive streaming with a lightweight speak/silent gate.

Load-bearing premise

Gains against the matched-size dense baseline are treated as evidence for the codec front-end, even though the training data mix, caption pipeline, and five-stage recipe also differ.

Editorial extensions

If this is right

  • Long-video and live perception systems can allocate tokens by temporal unpredictability instead of fixed frame grids and still keep accuracy.
  • Vision encoders for VLMs need not rely on multi-billion image-text pre-training if objectives and image+video joint training are aligned to the downstream interface.
  • Dense timestamped video captions plus short SFT can substitute for heavy long-VideoQA instruction tuning for zero-shot long-video QA.
  • A frozen backbone plus a tiny speak/silent gate is enough to add proactive streaming without a separate online architecture.
  • Cross-codec patch selection (traditional HEVC vs neural codecs) can stay training-free if only the importance map is consumed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If motion/residual importance is truly the right prior, the same selector could front-end non-VLM video stacks (detection, tracking, robotics) without retraining the compressor.
  • Confounding of data recipe and architecture suggests a clean ablate-one-factor leaderboard (codec on/off, captions on/off, gate on/off) would become the decisive community check.
  • Zero-Vision SFT before RL implies post-training compute may be better spent protecting text reasoning than on another round of visual instruction tuning.
  • Edge and always-on cameras are the natural deployment surface: 75% fewer visual tokens plus a silent-most-of-the-time gate maps directly to battery and bandwidth budgets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Mage-VL-4B, a streaming vision-language model whose visual front-end (Mage-ViT) tokenizes video codec-natively: I-frame patches are kept dense while P-frame patches are selected by codec-derived importance (motion-vector magnitude + residual energy in HEVC, or negative log-likelihood in the neural codec DCVC-RT), cutting visual tokens by ~75%. Mage-ViT is trained from scratch with cluster discrimination against MetaCLIP-derived K-means prototypes on ~560M images and 100M video frames. On top, a five-stage curriculum (captions → instruction → long video → codec-stream → streaming gate) builds the full model, with a System-1 event gate deciding when a frozen System-2 decoder should speak. Reported results: Mage-ViT matches SigLIP2-class encoders on linear/attentive probes (Table 1); Mage-VL-4B matches Qwen3-VL-4B on static image tasks and exceeds it on most video QA, temporal grounding, spatial, and tracking benchmarks (Tables 3–5), with up to 3.5× wall-clock speedups; streaming results on SoccerNet-Caption and OVO-Bench (Tables 6–7); plus ablations on caption-pipeline optimization and a Zero-Vision SFT + RL study. Support is broad: frozen-encoder probes, matched-budget scaling curves (Fig. 8), cross-codec transfer without retraining (Table 2), and a standardized lmms-eval protocol. The central weaknesses are causal attribution: the Table 4 claim that Mage-VL and Qwen3-VL-4B "differ only in the visual front-end" is confounded by large training-data/curriculum

Significance. If the results hold, the work makes three useful contributions: (i) evidence that a codec-aligned sparse front-end preserves representation quality at ~75% fewer tokens, supported by encoder-level probes rather than only end-task scores; (ii) a falsifiable efficiency claim (Fig. 8, Table 5) with matched-token-budget curves and wall-clock numbers on stated hardware; and (iii) an open release of encoder, model, and code, plus a set of clearly enumerated, individually testable empirical findings. The cross-codec transfer experiment and the Zero-Vision SFT result are each of independent interest to the community. Significance is tempered by the causal-attribution problems above: the headline 'front-end-only' comparison is actually a full-recipe comparison, and Finding 1 omits that the pretraining targets are distilled from a web-scale supervised teacher. These are framing/attribution defects rather than evidence that the engineering results are wrong.

major comments (3)
  1. [Table 4 caption and §5.3.2] The caption asserts Mage-VL and Qwen3-VL-4B 'differ only in the visual front-end,' but §4.2–4.3 document large training-mixture differences: ~350M AI4AI-recaptioned images, 7.95M video captions, Stage 4 codec-stream supervision, and task-specific SFT data Qwen3-VL did not use. The numbers make the confound concrete: the +18–20 J&F tracking gains (Ref-DAVIS17 25.83 vs 7.48) coincide with explicit Molmo2-VideoTrack/VideoPoint supervision in Stage 4; the +11.0 VSI-Bench gain coincides with 4M spatial samples; the Timelens gains (+7.6/+17.1/+22.5) coincide with TimeLens/VideoChat-Flash data in Stage 3. A front-end change cannot plausibly yield a 3–8× referring-segmentation jump; supervised tracking data can. The authors themselves concede in §5.3.2 (Finding 4) that single-variable ablations are infeasible — the same caveat must be applied to the Table 4 framing. Either retitle the comparison
  2. [§3.2 (Objective) and Finding 1 (§5.2.1)] Mage-ViT is trained by cluster discrimination against K-means prototypes computed over MetaCLIP features (§3.2, 'Objective'). MetaCLIP is itself trained on billions of image–text pairs, so the representation claim 'trained from scratch on ~560M unlabeled images... matches encoders trained on billions of image–text pairs' (Abstract; Finding 1, §5.2.1) describes distillation from a web-scale-supervised teacher, not label-free learning from unlabeled data alone. This materially weakens Finding 1 as stated ('web-scale pre-training is not essential'). The claim is recoverable in a weaker form (codec-aligned objectives plus prototype distillation are data-efficient), but the text should state the MetaCLIP dependence prominently — abstract, Finding 1, and §5.2.1 — and ideally report a control with self-supervised prototypes (e.g., DINOv2 features) to isolate the effect.
  3. [§5.5, Fig. 8, and Table 5] The 'matched budget' in Fig. 8 matches only encoder token workload: tc-N is built from 8N source frames while frame-N sees N frames, so the codec arm consumes 8× more raw visual information plus an HEVC encode/decode pass whose cost is not accounted for. Table 5 says Mage-VL times are 'full measured wall-clock' — please state explicitly whether codec extraction and patchification are included; if not, the '3.5× speedup' (NextQA: tc8/415s vs Qwen/1460s) is not end-to-end. Also note the flagship speedup compares tc8 (64 source frames) against Qwen at 32 frames, i.e., different information budgets in the opposite direction from Fig. 8. A per-stage timing breakdown (codec, ViT, LLM) and at least one accuracy comparison at equal raw-frame count would make Finding 5 sound.
minor comments (6)
  1. [Table 6] Table 6 mixes evaluation conditions: StreamMind numbers are taken from its publication, JoyAI is evaluated at 1 Hz with a relaxed ±1s window while Mage-VL uses zero-tolerance canvas matching. The TriggerAcc/TimVal comparison across these protocols is not apples-to-apples; please flag this in the caption or re-run baselines under a single protocol.
  2. [Table 7 (OVO-Bench)] The caption says Mage-VL runs at '1fps,' but the text says queries are answered under the SimpleStream recent-window protocol (four most recent frames). Clarify whether Mage-VL was evaluated on codec-token streams or on the 4-frame window, and how the gate/EPFE memory was used (or bypassed) in this offline-style protocol. Backward-Tracing accuracy (48.15%) lagging Real-Time (79.84%) also deserves one sentence of analysis given the streaming-memory design.
  3. [Table 3] Several regressions vs Qwen3-VL-4B are unexplained: CC-OCR Doc (32.25 vs 39.69), DUDE (46.44 vs 50.98), WebSRC (92.80 vs 95.40), TextVQA (77.28 vs 80.55), Charades in Table 5 (31.4 vs 45.9). Given the OCR/document emphasis of the caption pipeline, the CC-OCR gap is surprising and worth a remark.
  4. [Abstract / Tables 3–4] 'Comprehensively surpasses the 15B Phi-4-reasoning-vision baseline' (Abstract) should be qualified: Phi-4-R-V is a reasoning-tuned model not optimized for video/streaming, and it beats Mage-VL on some rows (MMOU 51.9 vs 39.3, ERQA, MMSI in part). The comparison is fine to include but the abstract phrasing overstates it.
  5. [Throughout] Typos/formatting: 'a16× 16patch level' (Abstract); 'Duringlatervideostages' (§4.2.1); Fig. 1 label 'VSI-Bench 64.0' vs Table 4's 64.3; double citation [72]/[100] for StreamMind with different venues; several 2026-dated arXiv references lack links. Appendix A prompt rendering has artifacts (e.g., '‘**bold**‘').
  6. [§5.4] PR-AUC of 9.30% (Table 6) is very low in absolute terms; a sentence contextualizing it against the speak-event base rate on SoccerNet-Caption would help readers calibrate the streaming claim.

Circularity Check

1 steps flagged · score 1.0 of 10

Empirical systems paper scored on external public benchmarks; no load-bearing prediction reduces to its inputs by construction.

  1. fitted input called prediction [§4.2.1 Caption-prompt optimization; §6.1 Table 8]
    "a rubric-scoring agent powered by GPT-5 evaluates the generated captions along four dimensions: Completeness; Redundancy; Coherence; OCR fidelity. ... The most notable gains are observed on document- and text-heavy benchmarks, such as InfoVQA (+5.62), OCRBench (+3.80), ChartQA (+1.52), and DocVQA (+1.08). This aligns with the explicit OCR quality metric incorporated in the evaluation sub-agent"

    Mild only: the caption prompt is iteratively fitted using an OCR-fidelity rubric, then downstream OCR/document QA gains are presented as pipeline validation. The reported scores are still external benchmark accuracies after full model training, not the rubric scores themselves, so this is correlated data tuning rather than a by-construction identity. Not load-bearing for the codec-native or Mage-ViT claims.

full rationale

Mage-VL’s central claims are model-performance numbers on independent public suites (ImageNet/CIFAR linear probes, DocVQA, VideoMME, VSI-Bench, Timelens, OVO-Bench, SoccerNet-Caption, wall-clock latency on B200). Mage-ViT’s cluster-discrimination objective uses MetaCLIP prototypes as a training signal, not as the reported metric; linear/attentive probes and downstream VLM scores remain external. Codec token reduction (top-k under budget B) is an explicit design choice whose accuracy retention is measured, not assumed. AI4AI prompt optimization uses rubric dimensions (completeness, OCR fidelity) that correlate with later OCR/document gains (Table 8), which is mild data-tuning toward related capabilities but not a by-construction identity between fit and reported score. Streaming gate labels are derived from the authors’ caption timestamps, yet timing/quality are evaluated on SoccerNet and OVO-Bench under fixed protocols. Overlap with OneVision-Encoder / LLaVA-OneVision-2 citations supplies architectural lineage, not a uniqueness theorem that forces the reported results. The Table 4 “differ only in the visual front-end” framing is a causal-confound issue (custom curriculum and task SFT), not circularity: the metrics are not defined to equal the training objective. No equation or claimed first-principles derivation collapses to its fitted inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

This is an empirical systems paper. Load-bearing premises are modeling choices (codec importance as token allocator; dual-process gate; cluster discrimination via MetaCLIP prototypes) and many training hyperparameters/data filters, not a short formal axiom list. Invented pieces are named modules, not new physical entities.

free parameters (5)
  • P-frame top-k / token budget B = B=4096; tc canvases ≈8/16/32-frame workloads
    Default B=4096 tokens for 64-frame clips and tc8/16/32 operating points set the accuracy-efficiency trade-off; chosen as design knobs, not derived.
  • Codec importance weights (motion magnitude + residual energy)
    HEVC selector combines motion-vector magnitude and residual energy with unspecified relative weighting in the main text; defines which patches survive.
  • Gate threshold τ and speak/silent class weights = τ=0.5
    Inference uses τ=0.5; training uses class-weighted CE to handle speak sparsity—both directly control timing metrics.
  • ViT/pretrain hyperparameters = r=0.1; Stage3 frames=384; RoPE θ=8M
    LR schedule (1e-3 then 5e-5), negative-sample ratio r=0.1, resolution ranges, mix 50/40/10 codec/chunk/collage, RoPE base 8M, Stage-3 384 frames—recipe choices that affect reported quality.
  • K-means prototype count / MetaCLIP feature space for cluster discrimination
    Pretraining targets nearest MetaCLIP-derived clusters; cluster granularity and teacher features are free design choices that shape Mage-ViT.
assumptions (5)
  • domain assumption Codec bit allocation (motion/residual or neural NLL) is a sufficient proxy for spatio-temporal importance for downstream VLM tasks.
    Stated as the codec-aligned sparsity principle (§3); entire token reduction claim rests on this proxy not dropping task-critical static detail.
  • domain assumption A still image is a degenerate one-frame video; shared 3D RoPE preserves geometry after aggressive patch drop.
    §3.1 architecture premise enabling one encoder for image/video/stream modes.
  • domain assumption Proactive streaming can be factored into a frozen System-2 VLM plus a lightweight System-1 speak/silent gate on rolling memory.
    §4.1/Fig. 3 bio-inspired dual-system design; Stage 5 freezes the base model and only trains the gate.
  • standard math Standard transformer/ViT training mathematics (AdamW, attention, CE losses) behave as usual at this scale.
    Background optimization assumptions used throughout §§3–4.
  • ad hoc to paper Dense timestamped captions can be deterministically converted into causal speak/silent supervision without future frames.
    §4.2.3 caption converter following StreamMind-style event gating; defines streaming labels used in Stage 5.
invented entities (3)
  • Mage-ViT codec-driven patchifier + canvas independent evidence
    purpose: Select and pack I-frame-all / P-frame-top-k 16×16 patches before the ViT trunk.
    Core named module; builds on OneVision-style sparsity but is the paper’s central artifact.
  • Cognition gate / System 1 event gate (+ EPFE streaming memory) independent evidence
    purpose: Predict silent vs speak on causal codec windows and trigger the frozen decoder.
    Introduced for proactive streaming; evaluated on SoccerNet-Caption and qualitative live commentary.
  • AI4AI prompt-code co-optimization loop independent evidence
    purpose: Iteratively rewrite caption prompts/code via rubric agents to raise mid-training data quality.
    Methodological entity; Table 8 shows downstream lifts after optimization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model." pith.science (2026). https://pith.science/paper/72FMYQVD

@misc{pith2026260724904,
  author       = {Pith},
  title        = {Pith review of: Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72FMYQVD}},
  note         = {Machine review of arXiv:2607.24904}
}
read the original abstract

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

175 extracted references · 67 linked inside Pith

  1. [1]

    Claude fable 5 & claude mythos 5 system card

    Anthropic. Claude fable 5 & claude mythos 5 system card. Technical report, Anthropic, 6

  2. [2]

    Gemini 3 pro model card

    Google DeepMind. Gemini 3 pro model card. Technical report, Google Deep- Mind, 11 2025. URLhttps://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf

    2025

  3. [3]

    Gpt-5.5 system card

    OpenAI. Gpt-5.5 system card. Technical report, OpenAI, 4 2026. URLhttps://openai. com/index/gpt-5-5-system-card/

    2026

  4. [4]

    Harvard University Press, 1988

    Hans Moravec.Mind Children: The Future of Robot and Human Intelligence. Harvard University Press, 1988

    1988

  5. [5]

    Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  6. [6]

    Qwen2.5-VL technical report.arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv:2502.13923, 2025

    arXiv 2025

  7. [7]

    InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenw...

    arXiv 2025

  8. [8]

    LLaVA-OneVision-1.5: A family of fully open vision–language models with the llava-onevision-1.5 mid-training and instruct data.arXiv preprint, 2025

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision-1.5: A family of fully open vision–language models with the llava-onevision-1.5 mid-training and instruct data.arXiv preprint, 2025

    2025

Show all 175 references
  1. [9]

    Kwai keye-vl 1.5 technical report.arXiv:2509.01563, 2025

    Biao Yang, Bin Wen, Boyang Ding, et al. Kwai keye-vl 1.5 technical report.arXiv:2509.01563, 2025

    2025 arXiv

  2. [10]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. InNeurIPS, 2024

    2024

  3. [12]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, 33 et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, a...

    2025 arXiv

  4. [13]

    MIT press, 2015

    Peter Sterling and Simon Laughlin.Principles of neural design. MIT press, 2015

    2015

  5. [14]

    Energy limitation as a selective pressure on the evolution of sensory systems.Journal of Experimental Biology, 211(11):1792–1804, 2008

    Jeremy E Niven and Simon B Laughlin. Energy limitation as a selective pressure on the evolution of sensory systems.Journal of Experimental Biology, 211(11):1792–1804, 2008

    2008

  6. [15]

    Eye smarter than scientists believed: neural computations in circuits of the retina.Neuron, 65(2):150–164, 2010

    Tim Gollisch and Markus Meister. Eye smarter than scientists believed: neural computations in circuits of the retina.Neuron, 65(2):150–164, 2010

    2010

  7. [16]

    Separate visual pathways for perception and action

    Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20–25, 1992

    1992

  8. [17]

    Macmillan, 2011

    Daniel Kahneman.Thinking, fast and slow. Macmillan, 2011

    2011

  9. [18]

    OneVision-Encoder: Codec-aligned sparsity as a foundational principle for multimodal intelligence.arXiv preprint arXiv:2602.08683, 2026

    Feilong Tang, Xiang An, Yunyao Yan, Yin Xie, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Chunyuan Li, Shikun Feng, Changrui Chen, Huajie Tan, Ming Hu, Manyuan Zhang, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, and Jiankang Deng. OneVision-Encoder: Codec-aligned sparsity...

    2026

  10. [19]

    Deep contextual video compression (DCVC).NeurIPS, 2021

    Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression (DCVC).NeurIPS, 2021

    2021

  11. [20]

    LLaVA-OneVision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026

    Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Chunyuan Li, Bo Li, Ziwei Liu, Jiankang Deng, et al. LLaVA-OneVision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026

    2026 arXiv

  12. [21]

    Phi-4-Mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, ...

    2025 arXiv

  13. [22]

    Phi-4-reasoning-vision-15b technical report, 2026

    Jyoti Aneja, Michael Harrison, Neel Joshi, Tyler LaBonte, John Langford, and Eduardo Salinas. Phi-4-reasoning-vision-15b technical report, 2026. URLhttps://arxiv.org/abs/ 2603.03975

    2026

  14. [23]

    The plenoptic function and the elements of early vision.Computational models of visual processing, 1(8):3, 1991

    James R Bergen and Edward H Adelson. The plenoptic function and the elements of early vision.Computational models of visual processing, 1(8):3, 1991

    1991

  15. [24]

    Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026

    Yifei Shen, Bo Li, and Xinjie Zhang. Skillopt-lite: Better and faster agent self-evolution via one line of vibe.arXiv preprint arXiv:2607.03451, 2026. 34

    2026 arXiv

  16. [25]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  17. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021

    2021

  18. [27]

    Sigmoid loss for language image pre-training.ICCV, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.ICCV, 2023

    2023

  19. [28]

    DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research (TMLR), 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research (TMLR), 2024

    2024

  20. [29]

    DynamicViT: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efficient vision transformers with dynamic token sparsification. InNeurIPS, 2021

    2021

  21. [30]

    AdaViT: Adaptive vision transformers for efficient image recognition

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. AdaViT: Adaptive vision transformers for efficient image recognition. In CVPR, 2022

    2022

  22. [31]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InICLR, 2023

    2023

  23. [32]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  24. [33]

    LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. InIEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  25. [34]

    Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InInternational Conference on Machine Learning (ICML), 2021

    2021

  26. [35]

    ViViT: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViViT: A video vision transformer. InIEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  27. [36]

    Video Swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video Swin transformer. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  28. [37]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  29. [38]

    LLaMA-VID: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023. 35

    2023 arXiv

  30. [39]

    Chat-UniVi: Unified visual representation empowers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-UniVi: Unified visual representation empowers large language models with image and video under- standing. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  31. [40]

    SlowFast-LLaVA: A strong training-free baseline for video large language models.arXiv preprint arXiv:2407.15841, 2024

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. SlowFast-LLaVA: A strong training-free baseline for video large language models.arXiv preprint arXiv:2407.15841, 2024

    2024 arXiv

  32. [41]

    MovieChat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. MovieChat: From dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer...

    2024

  33. [42]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. LongVU: Spatiote...

    2024 arXiv

  34. [43]

    VideoChat-Flash: Hierarchical compression for long-context video model- ing.arXiv preprint arXiv:2501.00574, 2025

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Chen, Yinan Zhu, Haian Sun, Yali He, Yu Wang, and Limin Wang. VideoChat-Flash: Hierarchical compression for long-context video model- ing.arXiv preprint arXiv:2501.00574, 2025

    2025 arXiv

  35. [44]

    Manmatha, Alexander J

    Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Compressed video action recognition. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  36. [45]

    Video-LaVIT: Unified video-language pre-training with decoupled visual-motional tokenization.ICML, 2024

    Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, and Yadong Mu. Video-LaVIT: Unified video-language pre-training with decoupled visual-motional tokenization.ICML, 2024

    2024

  37. [46]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  38. [47]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  39. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023

    2023

  40. [49]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  41. [50]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. InAdvances in Neural Information Process- ing Systems (NeurIPS...

    2023

  42. [51]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  43. [52]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  44. [53]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024

    2024

  45. [54]

    DeepSeek-VL: Towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. DeepSeek-VL: Towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  46. [55]

    Pixtral 12B.arXiv preprint arXiv:2410.07073, 2024

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12B.arXiv preprint arXiv:2410.07073, 2024

    2024 arXiv

  47. [56]

    Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  48. [57]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  49. [58]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  50. [59]

    BAGEL: A unified multimodal foundation model for image understand- ing, generation, and editing.arXiv preprint, 2025

    Chaorui Deng et al. BAGEL: A unified multimodal foundation model for image understand- ing, generation, and editing.arXiv preprint, 2025

    2025

  51. [60]

    Emu3.5: Native multimodal models.arXiv preprint, 2025

    Yuying Cui et al. Emu3.5: Native multimodal models.arXiv preprint, 2025

    2025

  52. [61]

    Ovis-u1 technical report

    Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, et al. Ovis-u1 technical report. arXiv preprint arXiv:2506.23044, 2025

    2025 arXiv

  53. [62]

    Unified multimodal understanding and generation models: Advances, challenges, and opportunities

    Xinjie Zhang, Jintao Guo, Shanshan Zhao, Minghao Fu, Lunhao Duan, Jiakui Hu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. arXiv preprint arXiv...

    2025

  54. [63]

    VideoChat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  55. [64]

    Video- ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- ChatGPT: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 37

    2023 arXiv

  56. [65]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  57. [66]

    Video- LLaVA: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video- LLaVA: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  58. [67]

    LLaVA-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  59. [68]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  60. [69]

    VideoLLM-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. VideoLLM-online: Online video large language model for streaming video. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (...

    2024

  61. [70]

    VideoLLM knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 2024

    Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. VideoLLM knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 2024

    2024

  62. [71]

    Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.arXiv preprint arXiv:2501.03218, 2025

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.arXiv preprint arXiv:2501.03218, 2025

    2025 arXiv

  63. [72]

    Streammind: Unlocking full frame rate streaming video dialogue through event- gated cognition

    Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, and Ting Cao. Streammind: Unlocking full frame rate streaming video dialogue through event- gated cognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13448...

    2025

  64. [73]

    Joyai-vl-interaction: Real-time vision-language interaction intelligence

    Video Understanding Team of JoyAI-VL @ Joy Future Academy, JD. Joyai-vl-interaction: Real-time vision-language interaction intelligence. Technical report, Joy Future Academy, JD, June 2026

    2026

  65. [74]

    Flash-VStream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-VStream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    2024 arXiv

  66. [75]

    Streamingvlm: Real-time understanding for infinite video streams, 2026

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams, 2026. URLhttps://arxiv.org/abs/ 2510.09608

    2026 arXiv

  67. [76]

    InternLM-XComposer2.5-OmniLive: A comprehensive multimodal system for long-term streaming video and audio interactions.arXiv preprint arXiv:2412.09596, 2024

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. InternLM-XComposer2.5-OmniLive: A comprehensive multimodal system for long-term streaming video and audio interactions.arXiv preprint arXiv:2412.09596, 2024

    2024 arXiv

  68. [77]

    Livecc: Learning video llm with streaming speech transcription at scale, 2025

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale, 2025. URLhttps://arxiv.org/ abs/2504.16030. 38

    2025 arXiv

  69. [78]

    Matchtime: Towards automatic soccer game commentary generation, 2024

    Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. Matchtime: Towards automatic soccer game commentary generation, 2024. URLhttps://arxiv.org/abs/2406. 18530

    2024

  70. [79]

    StreamingBench: Assessing the gap for mllms to achieve streaming video understanding.arXiv preprint arXiv:2411.03628, 2024

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. StreamingBench: Assessing the gap for mllms to achieve streaming video understanding.arXiv preprint arXiv:2411.03628, 2024

    2024 arXiv

  71. [80]

    OVO-Bench: How far is your video-llms from real-world online video understanding? InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. OVO-Bench: How far is your video-llms from real-world online video understanding? InIEE...

    2025

  72. [81]

    Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand

    Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (HEVC) standard. InIEEE Transactions on Circuits and Systems for Video Technology, 2012

    2012

  73. [82]

    Towards practical real-time neural video compression

    Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, and Yan Lu. Towards practical real-time neural video compression. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12543–12552, 2025

    2025

  74. [83]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  75. [84]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 2024

    2024

  76. [85]

    LAION-400m: Open dataset of CLIP-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400m: Open dataset of CLIP-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  77. [86]

    COYO-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. COYO-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022

    2022

  78. [87]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image- text documents. InAd...

    2023

  79. [88]

    CCMB: A large-scale chinese cross-modal benchmark

    Chunyu Xie, Heng Cai, Jincheng Li, Fanjing Kong, Xiaoyu Wu, Jianfei Song, Henrique Morimitsu, Lin Yao, Dexin Wang, Xiangzheng Zhang, Dawei Leng, Baochang Zhang, Xiangyang Ji, and Yafeng Deng. CCMB: A large-scale chinese cross-modal benchmark. arXiv preprint arXiv:2205.03860, 2023

    2023 arXiv

  80. [89]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009

    2009

  81. [90]

    HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. InIEEE/CVF International Conference on Computer Vision (ICCV), 2019. 39

    2019

  82. [91]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InIEEE/CVF Conference on...

    2024

  83. [92]

    Qwen3 technical report.arXiv preprint, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint, 2025

    2025

  84. [93]

    Mage-flow: An efficient native-resolution foundation model for image generation and editing.arXiv preprint arXiv:2607.19064, 2026

    Xinjie Zhang, Peng Zhang, Shicheng Zheng, Jinghao Guo, Zhaoyang Jia, Yifei Shen, Xun Guo, Yuxuan Luo, Jiahao Li, Wenxuan Xie, Fanyi Pu, Xiaoyi Zhang, Kaichen Zhang, Zongyu Guo, Tianci Bi, Dongnan Gui, Zhening Liu, Zimo Wen, Zihan Zheng, Senqiao Yang, Xiao Li, Jinglu Wang, Bin ...

    2026 arXiv

  85. [94]

    Ai4ai at scale: A full-pipeline system for enhancing llm agentic capabilities

    Microsoft Mage Team. Ai4ai at scale: A full-pipeline system for enhancing llm agentic capabilities. Technical report, Microsoft, 2026

    2026

  86. [95]

    FineVision: Open data is all you need.arXiv preprint arXiv:2510.17269, 2025

    Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, Rui Li, Thibaud Frere, Leandro von Werra, Aritra Roy Gosthipaty, and Andrés Marafioti. FineVision: Open data is all you need.arXiv preprint arXiv:2510.17269, 2025

    2025 arXiv

  87. [96]

    Bee: A high-quality corpus and full-stack suite to unlock advanced fully open MLLMs, 2026

    YiZhang, BolinNi, Xin-ShengChen, Heng-RuiZhang, YongmingRao, HouwenPeng, Qinglin Lu, Han Hu, Meng-Hao Guo, and Shi-Min Hu. Bee: A high-quality corpus and full-stack suite to unlock advanced fully open MLLMs, 2026. URLhttps://arxiv.org/abs/2510.13795

    2026

  88. [99]

    Molmo2: Open weights and data for vision-language models with video understanding and grounding

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Rohun Tripathi, Sangho Lee, Mohammadreza Salehi, Jason Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. In Proceedings of the IE...

    2026

  89. [100]

    Streammind: Unlocking full frame rate streaming video dialogue through event-gated cognition, 2025

    Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai, Zhibo Chen, and Ting Cao. Streammind: Unlocking full frame rate streaming video dialogue through event-gated cognition, 2025. URLhttps://arxiv.org/abs/2503.06220

    2025 arXiv

  90. [101]

    Describing textures in the wild.arXiv preprint arXiv:1311.3618, 2013

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild.arXiv preprint arXiv:1311.3618, 2013

    2013 arXiv

  91. [102]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  92. [103]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. InCVPR, 2010

    2010

  93. [104]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InECCV, 2014. 40

    2014

  94. [105]

    Berg, and Li Fei- Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge.International Journal of Computer Vision, 2015

    2015

  95. [106]

    Meta clip 2: A worldwide scaling recipe

    Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen-tau Yih, Shang-Wen Li, and Hu Xu. Meta clip 2: A worldwide scaling recipe. arXiv preprint arX...

    2025 arXiv

  96. [107]

    Multimodal autoregressive pre-training of large vision encoders.arXiv preprint arXiv:2411.14402, 2024

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, et al. Multimodal autoregressive pre-training of large vision encoders.arXiv preprint arXiv:2411.14402, 2024

    2024 arXiv

  97. [108]

    Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, et al

    Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, et al. DINOv3.arXiv preprint arXiv:2508.10104, 2025

    2025 arXiv

  98. [109]

    Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    Kimi Team. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    2025 arXiv

  99. [110]

    RESOUND: Towards action recognition without representation bias

    Yingwei Li, Yi Li, and Nuno Vasconcelos. RESOUND: Towards action recognition without representation bias. InECCV, 2018

    2018

  100. [111]

    HMDB: A large video database for human motion recognition

    Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. InICCV, 2011

    2011

  101. [112]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens, et al. Perception test: A diagnostic benchmark for multimodal video models. InNeurIPS Datasets and Benchmarks Track, 2023

    2023

  102. [113]

    Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari

    Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. InCVPR, 2018

    2018

  103. [114]

    The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, et al. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  104. [115]

    Lmms-eval: Reality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. InFindings of the Association for Computational Linguistics: NAA...

    2025

  105. [116]

    MVBench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, et al. MVBench: A comprehensive multi-modal video understanding benchmark. InCVPR, 2024

    2024

  106. [117]

    NExT-QA: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NExT-QA: Next phase of question-answering to explaining temporal actions. InCVPR, 2021

    2021

  107. [118]

    TempCompass: Do video LLMs really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, et al. TempCompass: Do video LLMs really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  108. [119]

    Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  109. [120]

    LongVideoBench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024. 41

    2024 arXiv

  110. [121]

    LVBench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, et al. LVBench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  111. [122]

    MLVU: Benchmarking multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, et al. MLVU: Benchmarking multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  112. [123]

    TALL: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: Temporal activity localization via language query. InICCV, 2017

    2017

  113. [124]

    Timelens: Rethinking video temporal grounding with multimodal llms.arXiv preprint arXiv:2512.14698, 2025

    Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms.arXiv preprint arXiv:2512.14698, 2025

    2025

  114. [125]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, 2025

    2025

  115. [126]

    Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. DocVQA: A dataset for VQA on document images. InWACV, 2021

    2021

  116. [127]

    Minesh Mathew, Viraj Bagal, Rubèn Pérez-Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. InfographicVQA.arXiv preprint arXiv:2104.12756, 2021

    2021 arXiv

  117. [128]

    A diagram is worth a dozen images.arXiv preprint arXiv:1603.07396, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images.arXiv preprint arXiv:1603.07396, 2016

    2016 arXiv

  118. [129]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, 2022

    2022

  119. [130]

    OCRBench: On the hidden mystery of OCR in large multimodal models.Science China Information Sciences, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, et al. OCRBench: On the hidden mystery of OCR in large multimodal models.Science China Information Sciences, 2024

    2024

  120. [131]

    CC-OCR: A comprehensive and challenging OCR benchmark for evaluating large multimodal models in literacy.arXiv preprint arXiv:2412.02210, 2024

    Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, et al. CC-OCR: A comprehensive and challenging OCR benchmark for evaluating large multimodal models in literacy.arXiv preprint arXiv:2412.02210, 2024

    2024 arXiv

  121. [132]

    Hierarchical multimodal transform- ers for multi-page DocVQA.arXiv preprint arXiv:2212.05935, 2022

    Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transform- ers for multi-page DocVQA.arXiv preprint arXiv:2212.05935, 2022

    2022 arXiv

  122. [133]

    Document understanding dataset and evaluation (DUDE)

    Jordy Van Landeghem, Rubén Tito, Łukasz Borchmann, Michał Pietruszka, et al. Document understanding dataset and evaluation (DUDE). InICCV, 2023

    2023

  123. [134]

    WebSRC: A dataset for web-based structural reading comprehension

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension. InEMNLP, 2021

    2021

  124. [135]

    ChartQAPro: A more diverse and challenging benchmark for chart question answering.arXiv preprint arXiv:2504.05506, 2025

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, et al. ChartQAPro: A more diverse and challenging benchmark for chart question answering.arXiv preprint arXiv:2504.05506, 2025

    2025 arXiv

  125. [136]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. InCVPR, 2019. 42

    2019

  126. [137]

    CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, et al. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs. InNeurIPS Datasets and Benchmarks Track, 2024

    2024

  127. [138]

    MMBench: Is your multi-modal model an all-around player? InECCV, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, et al. MMBench: Is your multi-modal model an all-around player? InECCV, 2024

    2024

  128. [139]

    RealWorldQA: A benchmark for real-world spatial understanding.https://x.ai/ news/grok-1.5v, 2024

    xAI. RealWorldQA: A benchmark for real-world spatial understanding.https://x.ai/ news/grok-1.5v, 2024

    2024

  129. [140]

    Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, et al. Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024

    2024

  130. [141]

    MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, et al. MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  131. [142]

    SEED-Bench: Benchmarking multimodal LLMs with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking multimodal LLMs with generative comprehension. InCVPR, 2024

    2024

  132. [143]

    SEED-Bench-2- Plus: Benchmarking multimodal large language models with text-rich visual comprehension

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. SEED-Bench-2- Plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790, 2024

    2024 arXiv

  133. [144]

    MMT-Bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, et al. MMT-Bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. InICML, 2024

    2024

  134. [145]

    MME- RealWorld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? InICLR, 2025

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, et al. MME- RealWorld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? InICLR, 2025

    2025

  135. [146]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–...

    2024

  136. [147]

    BLINK: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, et al. BLINK: Multimodal large language models can see but not perceive. InECCV, 2024

    2024

  137. [148]

    EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models

    Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. InACL, 2024

    2024

  138. [149]

    The all-seeing project V2: Towards general relation comprehension of the open world

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, et al. The all-seeing project V2: Towards general relation comprehension of the open world. InECCV, 2024

    2024

  139. [150]

    Towards cross-view point correspondence in vision-language models.arXiv preprint arXiv:2512.04686, 2025

    Yipu Wang, Yuheng Ji, Yuyang Liu, Enshen Zhou, Ziqiang Yang, Yuxuan Tian, Ziheng Qin, Yue Liu, Huajie Tan, Cheng Chi, Zhiyuan Ma, Daniel Dajun Zeng, and Xiaolong Zheng. Towards cross-view point correspondence in vision-language models.arXiv preprint arXiv:2512.04686, 2025

    2025

  140. [151]

    Gemini robotics: Bringing AI into the physical world

    Gemini Robotics Team. Gemini robotics: Bringing AI into the physical world. Technical report, Google DeepMind, 2025. arXiv:2503.20020. 43

    2025 arXiv

  141. [152]

    MMSI-Bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, et al. MMSI-Bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    2025 arXiv

  142. [153]

    SAT: Dynamic spatial aptitude training for multimodal language models

    Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, et al. SAT: Dynamic spatial aptitude training for multimodal language models. InCOLM, 2025

    2025

  143. [154]

    Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding.arXiv preprint arXiv:2604.05015, 2026

    Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, et al. Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding.arXiv preprint arXiv:2604.05015, 2026

    2026 arXiv

  144. [155]

    VideoEval-Pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025

    Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. VideoEval-Pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025

    2025 arXiv

  145. [156]

    Mmou: A massive multi-task omni understanding and reasoning benchmark for long and complex real-world videos.arXiv preprint arXiv:2603.14145, 2026

    Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, et al. Mmou: A massive multi-task omni understanding and reasoning benchmark for long and complex real-world videos.arXiv preprint ar...

    2026 arXiv

  146. [157]

    Video object segmentation with language referring expressions

    Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. InACCV, 2018

    2018

  147. [158]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023

    2023

  148. [159]

    VISA: Reasoning video object segmentation via large language models.arXiv preprint arXiv:2407.11325, 2024

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, et al. VISA: Reasoning video object segmentation via large language models.arXiv preprint arXiv:2407.11325, 2024

    2024 arXiv

  149. [160]

    URVOS: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. URVOS: Unified referring video object segmentation network with a large-scale benchmark. InECCV, 2020

    2020

  150. [161]

    SoccerNet-Caption: Dense video captioning for soccer broadcasts commentaries

    Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. SoccerNet-Caption: Dense video captioning for soccer broadcasts commentaries. InIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023

    2023

  151. [162]

    A simple baseline for streaming video understanding.arXiv preprint arXiv:2604.02317, 2026

    Yujiao Shen, Shulin Tian, Jingkang Yang, and Ziwei Liu. A simple baseline for streaming video understanding.arXiv preprint arXiv:2604.02317, 2026

    2026

  152. [163]

    Video instruction tuning with synthetic data.Transactions on Machine Learning Research (TMLR), 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.Transactions on Machine Learning Research (TMLR), 2024. arXiv:2410.02713

    2024 arXiv

  153. [164]

    TimeChat-Online: 80% visual tokens are naturally redundant in streaming videos.arXiv preprint arXiv:2504.17343, 2025

    Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, et al. TimeChat-Online: 80% visual tokens are naturally redundant in streaming videos.arXiv preprint arXiv:2504.17343, 2025

    2025 arXiv

  154. [165]

    StreamFor- est: Efficient online video understanding with persistent event memory.arXiv preprint arXiv:2509.24871, 2025

    Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, et al. StreamFor- est: Efficient online video understanding with persistent event memory.arXiv preprint arXiv:2509.24871, 2025

    2025

  155. [166]

    Streaming video instruction tuning.arXiv preprint arXiv:2512.21334, 2026

    Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, and Kaiyang Zhou. Streaming video instruction tuning.arXiv preprint arXiv:2512.21334, 2026. 44

    2026 arXiv

  156. [167]

    Gueter Josmy Faure, Jia-Fong Yeh, Min-Hung Chen, Hung-Ting Su, Shang-Hong Lai, and Winston H. Hsu. HERMES: temporal-coherent long-form understanding with episodes and semantics.arXiv preprint arXiv:2408.17443, 2025

    2025 arXiv

  157. [168]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021

    2021

  158. [169]

    MathVista: Evaluating mathe- matical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathe- matical reasoning of foundation models in visual contexts. InICLR, 2024

    2024

  159. [170]

    We-Math: Does your large multimodal model achieve human-like mathematical reasoning?arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, et al. We-Math: Does your large multimodal model achieve human-like mathematical reasoning?arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  160. [171]

    Mea- suring multimodal mathematical reasoning with MATH-Vision dataset.arXiv preprint arXiv:2402.14804, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with MATH-Vision dataset.arXiv preprint arXiv:2402.14804, 2024

    2024 arXiv

  161. [172]

    MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? InECCV, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, et al. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? InECCV, 2024

    2024

  162. [173]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024

    2024

  163. [174]

    DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. InICLR, 2025

    2025

  164. [175]

    MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. InACL, 2025

    2025

  165. [176]

    Openmmreasoner: Pushing the frontiers in multimodal reasoning with an open and general recipe

    Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, Kairui Hu, Bin Wang, Xingxuan Li, and Lidong Bing. Openmmreasoner: Pushing the frontiers in multimodal reasoning with an open and general recipe. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2026

  166. [177]

    Nemotron-Pretraining-SFT-v1 Dataset

    NVIDIA. Nemotron-Pretraining-SFT-v1 Dataset. https://huggingface.co/datasets/ nvidia/Nemotron-Pretraining-SFT-v1, 2025. 45

    2025

  167. [2026]

    URLhttps://www.anthropic.com/system-cards

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.