Pith. sign in

REVIEW 5 major objections 5 minor 18 cited by

Apollo: An Exploration of Video Understanding in Large Multimodal Models

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that design choices for video large multimodal models can be made on small models and datasets and transferred reliably to larger ones, and that the resulting Apollo model family sets strong results on video benchmarks.

desk verdict A wide and useful design-space study whose headline Scaling Consistency claim is overreached; the SOTA and extrapolation claims need major revision, but the fps/encoder/data findings are worth taking seriously. read the letter →

arxiv 2412.10360 v1 pith:EZPHIONT submitted 2024-12-13 cs.CV cs.AI

Orr Zohar , Xiaohan Wang , Yann Dubois , Nikhil Mehta , Tong Xiao , Philippe Hansen-Estruch , Licheng Yu , Xiaofang Wang
show 4 more authors
Felix Juefei-Xu Ning Zhang Serena Yeung-Levy Xide Xia
This is my paper · ORCID
classification cs.CVcs.AI
keywords videolargemultimodalmodelsScalingConsistencyApolloBenchframespersecondsamplingvisionencoderselectiontokenresamplingtrainingschedulesbenchmarkdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to find what actually drives video understanding in large multimodal models (LMMs) instead of inheriting image-model conventions. Its central claim is Scaling Consistency: design and training decisions made with models of roughly 2–4 billion parameters and datasets of about 500K samples transfer reliably to larger models, so the design space can be explored cheaply. Using that shortcut, the authors compare video sampling, vision encoders, token resamplers, training schedules, and data mixtures, and extract concrete rules such as frames-per-second sampling beating uniform frame sampling. They also introduce ApolloBench, a 400-question multiple-choice benchmark filtered so that text-only or single-frame inputs cannot answer it, which they report is $41\times$ faster to evaluate and highly correlated with existing video benchmarks. Guided by these findings, Apollo models are claimed to outperform most existing 7B models at 3B scale and to match or beat many 30B models at 7B scale, with Apollo-7B scoring 70.9 on MLVU and 63.3 on Video-MME.

What carries the argument

The mechanism that carries the argument is Scaling Consistency, defined as the transfer of design-decision rankings from small to large LMMs once a critical size is passed; the paper measures this with rank-correlation ($R^2$) over 21 hand-selected model variations and reports the fit $R^2 = 0.12\log(x) + 0.78$ against the 7B model. The second load-bearing piece is ApolloBench, a 400-question multiple-choice suite filtered so that questions answerable by text-only or single-frame inputs are removed, which makes the small-scale comparisons cheap and more video-specific. These two pieces work together: ApolloBench supplies the fast, discriminative signal, and Scaling Consistency says the signal seen at 2-4B scale will still order larger models correctly, letting the paper's other findings (fps sampling, encoder choice, Perceiver resampler, three-stage training, 10-14% text data) be read as design rules rather than small-model accidents.

What would settle it

Train the same 21 design variations within one LLM family at 0.5B, 1.5B, 3B, 7B and, if resources allow, 30B+, then compare their rankings on ApolloBench; if the 3B-to-30B ranking correlation drops well below the reported 4B-to-7B value of $R^2 = 0.938$, Scaling Consistency is falsified. A first check is whether replacing the Qwen1.5-4B point with a same-family Qwen2-4B preserves the reported fit.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is Scaling Consistency: for video large multimodal models there is a critical scale—roughly 2–4 billion parameters and around 500K training samples—above which design decisions transfer reliably to larger models. The evidence is a set of 84 models built by crossing 21 design variations with four LLM sizes (Qwen2-0.5B, Qwen2-1.5B, Qwen1.5-4B, Qwen2-7B), where the agreement between the 4B and 7B configurations reaches $R^2 = 0.938$ and the correlation grows roughly log-linearly with model size. The paper also claims that on this small-scale testbed, frames-per-second sampling beats uniform sampling, SigLIP-SO400M is the best single vision encoder and combines best with InternVideo2, the Perceiver Resampler beats average pooling, and a three-stage schedule with about 10-14% text data in the fine-tuning mix works best. On the evaluation side, it claims many existing video benchmarks can be partially answered from text or a single frame, and it curates ApolloBench, 400 multiple-choice questions that resist such shortcuts and are $41\times$ faster to score while staying highly correlated with the larger suites. Guided by these choices, the paper presents Apollo-3B and Apollo-7B, which it claims outperform most existing 7B models and rival 30B models, with Apollo-7B scoring 70.9 on MLVU and 63.3 on Video-MME.

Load-bearing premise

The load-bearing premise is that the ranking of design choices obtained on 2-to-4-billion-parameter models trained on roughly 500K samples is the same ranking that would be obtained on much larger models, including sizes beyond the 7B models actually tested.

Editorial extensions

If this is right

  • Video-LMM design studies can be run on 2–4B models and about 500K samples, reserving large-scale training for final verification, which cuts the compute of design exploration by a large factor.
  • Frames-per-second sampling should replace uniform frame sampling for long-video training, with 8–32 tokens per frame as the recommended operating range.
  • A recipe combining SigLIP-SO400M and InternVideo2 encoders, a Perceiver Resampler, and clip timestamps between video tokens is a strong default architecture.
  • ApolloBench provides a cheaper development-time evaluation that ranks models similarly to the full video benchmark suites it was drawn from.
  • Careful design choices can let a 3B or 7B model compete with open models two to four times its size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper extrapolates the log-linear correlation beyond 7B to 32B/72B, but no models above 7B were measured; a direct same-family scaling study at 30B+ would test whether that extrapolation holds.
  • Editorial extension: because the 4B point uses a different LLM family than the smaller sizes, a same-family replication (for example, Qwen2-0.5/1.5/3/7B) would cleanly separate model size from model family effects.
  • Editorial extension: the benchmark-decontamination principle behind ApolloBench (filter out questions answerable from text or a single frame) could be applied when building future video benchmarks, not just to the questions selected here.
  • Editorial extension: if Scaling Consistency holds for unseen choices, small-model-first experimentation could become the default workflow for video-LMM research, with large models used mainly for final confirmation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a systematic study of video understanding in large multimodal models (LMMs), covering video sampling, vision encoders, token resampling, token integration, training schedules, and data composition. It introduces ApolloBench, a 400-question benchmark subset intended to be 41x faster and more video-centric than existing suites, and proposes 'Scaling Consistency,' the claim that design decisions validated on smaller models (roughly 2-4B parameters) transfer reliably to larger models. Guided by these findings, the authors train Apollo-1.5B, Apollo-3B, and Apollo-7B and report strong performance on MLVU, Video-MME, LongVideoBench, and other benchmarks, including claims of state-of-the-art results among 7B models.

Significance. If Scaling Consistency holds, the paper would provide a practically important methodology: researchers could validate video-LMM design choices on 2-4B models and ~500K samples instead of training large models, substantially reducing compute. The systematic exploration also yields concrete, actionable findings (e.g., fps sampling over uniform sampling, SigLIP-SO400M as a strong single encoder, Perceiver Resampler over average pooling) and makes raw experimental results available in the appendix. ApolloBench, despite its potential for evaluation-gaming concerns, is a useful contribution toward cheaper and more video-discriminative evaluation. However, the central scientific claim is currently supported by a very small and confounded dataset, and the headline benchmark claims are overstated relative to the paper's own Table 4.

major comments (5)
  1. [Sec. 3, Fig. 3, App. D] The Scaling Consistency extrapolation is not valid as stated. The fitted curve y = 0.12 log(x) + 0.78 describes R2 between models of size x and a fixed 7B reference. Extending the x-axis beyond 7B cannot produce the correlation between a 7B model and a 32B or 72B model; that would require new data with the larger model as the reference. Moreover, the curve violates the boundary condition R2(7B,7B)=1, as it reaches about 1.01 at x=7 and exceeds 1.2 at x=32, which is impossible for a correlation coefficient. The Appendix D claim that models above 3-4B can be expected to have R2 > 0.86 with 32B or R2 > 0.84 with 72B models is therefore unsupported and should be removed or replaced with actual measurements.
  2. [Sec. 3, Fig. 3, App. D] The in-range evidence for Scaling Consistency is confounded by model family. The 0.5B and 1.5B points use Qwen2 models, while the 4B point uses Qwen1.5-4B. The jump from R2=0.831 (1.5B) to R2=0.938 (4B) may reflect family differences rather than a pure size effect. The paper even states that Qwen1.5-4B has similar or lower performance than Qwen2-1.5B, so the correlation difference cannot be attributed to size alone. A clean test would include a Qwen2-4B model or a Qwen1.5-1.5B model.
  3. [Sec. 4, first paragraph; Sec. 3] The design exploration is carried out with Qwen2.5-3B, but Qwen2.5-3B is absent from the Scaling Consistency study, which uses Qwen2-0.5B, Qwen2-1.5B, Qwen1.5-4B, and Qwen2-7B. The claim that 'these findings exhibit a strong correlation (R2 > 0.9) with results on larger models and across different model families' does not follow from the presented data, since the specific model family used for all subsequent ablations was never part of the correlation analysis. This weakens every Finding 2-10 that relies on the Qwen2.5-3B proxy.
  4. [Table 4; Abstract; Sec. 6] The claim that Apollo-7B is 'state-of-the-art compared to 7B LMMs' is contradicted by the paper's own Table 4. Qwen2-VL-7B achieves 68.5 on TempCompass vs Apollo-7B's 64.9, and 63.3/69.0 on Video-MME with/without subtitles vs Apollo-7B's 61.3/63.3. Apollo-7B does lead on MLVU and several other columns, but the unqualified SOTA claim overstates the results. The abstract and Section 6 should either name the specific benchmarks where Apollo-7B is SOTA or qualify the claim relative to per-benchmark comparisons.
  5. [Sec. 5.1, Table 3] The training-schedule comparison reports only the best-performing model among three hyperparameter settings per schedule ('For each training schedule, three hyperparameters were tested, and we report the best-performing model'). Reporting best-of-N inflates scores and makes it difficult to assess whether the three-stage advantage is robust or an artifact of hyperparameter selection. Reporting all runs, or at least the median and variance, would substantially strengthen Finding 8.
minor comments (5)
  1. [Sec. 6, architecture description] Typo: 'Perciver Resampler' should read 'Perceiver Resampler'.
  2. [Intro, Sec. 6, Table 4] The Introduction reports Apollo-7B's Video-MME (w/o subtitles) as 61.2, while Table 4 reports 61.3. Please reconcile the numbers.
  3. [Sec. 2.3] ApolloBench is used both as the main ablation metric and as a final evaluation benchmark. Since ApolloBench was curated with a separate set of ten open models and Apollo is also evaluated on external benchmarks, this is not circular reasoning, but the paper should state more explicitly that ApolloBench rankings may not perfectly reflect rankings on external benchmarks.
  4. [Fig. 3 caption] The left plot's fitted curve is labeled with '7B 0.5B' in a way that is ambiguous about which points are used for the fit. Please clarify whether the 7B self-correlation point (1.0) is included in the fit.
  5. [App. D] The numbers '> R2 ≃ 0.86' for 32B and '> R2 ≃ 0.84' for 72B appear inconsistent with the fitted equation y = 0.12 log(x) + 0.78, which gives values above 1 for x > 7. Please either remove these extrapolated values or derive them from a correctly specified model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical findings are measured on an independently curated benchmark and validated against external benchmarks.

full rationale

The paper's central claims are empirical design findings rather than results derived from their own inputs. Scaling Consistency (Sec. 3 and App. D) is an observed R2 correlation across 21 design variations trained at four sizes; the R2 values are computed from ApolloBench accuracy and are not fitted parameters later renamed as predictions. ApolloBench itself was curated using a separate set of ten open-source models and entropy-based filtering, not by fitting the 84-model scaling study, so the correlation does not reduce by construction to the curation process. The design findings in Secs. 4 and 5 are ablations measured on ApolloBench, and although ApolloBench is introduced in the same paper, the final Apollo models are also evaluated on external benchmarks (MLVU, Video-MME, LongVideoBench, TempCompass, PerceptionTest), providing independent evidence. The self-citations (Zohar et al. 2024; Wang et al. 2024c) appear only as related work and do not carry the derivation. The extrapolation in App. D to 32B/72B is a statistical over-extension of the fitted log-linear curve and is a correctness risk rather than a circularity, because the claimed R2 values are not equal to any input by construction. No step in the derivation chain is equivalent to its own inputs, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several unstated assumptions: benchmark proxy validity, representativeness of model family, and statistical extrapolation. The numeric hyperparameters chosen from ablations are treated as settled design choices rather than derived quantities.

free parameters (5)
  • Critical model size threshold = ~2-4B parameters
    Inferred from R2 vs model size in Fig. 3 left; used to justify 3B experiments.
  • Dataset size plateau = ~500K samples
    Inferred from Fig. 3 right; used to choose 750K training set for ablations.
  • Training fps = 2 fps
    Chosen from sampling analysis in Sec. 4.1 as default; not derived.
  • Tokens per frame = 16 (32 tokens per clip)
    Chosen from tps/tpf analysis in Finding 3; range 8-32 considered optimal.
  • Text data proportion = 10-14%
    Chosen from data mixture sweeps in Finding 10; outside this range hurts performance.
assumptions (4)
  • domain assumption R2 correlation across 21 model variations measures transferability of design decisions
    Used in Sec. 3 to define Scaling Consistency; no theoretical guarantee that rank correlation on this variation set predicts arbitrary new design choices.
  • domain assumption ApolloBench is a valid proxy for the full benchmark suite
    Sec. 2.3 selects 400 questions based on discrimination among 10 open models; assumes this subset stays representative for new models like Apollo.
  • domain assumption Qwen LLMs with SigLIP and InternVideo2 represent the video-LMM design space
    Sec. 4 and 5 draw general conclusions from one LLM family and two encoders.
  • ad hoc to paper Log-linear extrapolation of R2 to larger model sizes
    App. D predicts R2 greater than 0.84 for 72B from a four-point fit in Fig. 3 left, with no theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Apollo: An Exploration of Video Understanding in Large Multimodal Models." pith.science (2026). https://pith.science/paper/EZPHIONT

@misc{pith2026241210360,
  author       = {Pith},
  title        = {Pith review of: Apollo: An Exploration of Video Understanding in Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZPHIONT}},
  note         = {Machine review of arXiv:2412.10360}
}
abstract

Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ARGUS: Hallucination and Omission Evaluation in Video-LLMs

    cs.CV 2025-06 conditional novelty 7.0 of 10

    ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.

  2. Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    MS-SFNN encodes multi-scale Fourier features in a separable product of fixed-weight cosine subnetworks and solves for linear coefficients by least squares, claiming better accuracy than PINN and SV-SNN on high-frequency PDEs.

  3. VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.

  4. "Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.

  5. ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.

  6. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  7. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  8. Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Nar-KFC improves long-video question answering by interleaving query-relevant diverse keyframes with captions of non-keyframes, raising accuracy on Video-MME, LongVideoBench, and MLVU across 7B MLLMs.

  9. Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.

  10. UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.

  11. Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Video LLM benchmark scores are inflated by language-prior and static-frame questions; VBenchComp separates those from true temporal questions and shows a trimmed core subset preserves rankings.

  12. Temporal Preference Optimization for Long-Form Video Understanding

    cs.CV 2025-01 conditional novelty 6.0 of 10

    TPO trains video-LMMs to prefer answers generated from complete, relevant frames over answers from incomplete or irrelevant frames, improving temporal grounding on LongVideoBench, MLVU, and Video-MME.

  13. Learnings from Scaling Visual Tokenizers for Reconstruction and Generation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A ViT-based visual tokenizer study shows latent code size drives reconstruction quality, encoder scaling gives little benefit for generation, and ViTok reaches competitive or state-of-the-art results with fewer FLOPs.

  14. PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Video MLLMs mostly ignore motion in pixel-level visual grounding; a new motion-centric benchmark shows large performance drops.

  15. Beyond Text Compression: Evaluating Tokenizers Across Scales

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.

  16. FlexSelect: Flexible Token Selection for Efficient Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.

  17. Redundancy Principles for MLLMs Benchmarks

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Correlations of model rankings across benchmark dimensions, instances, and math benchmarks show substantial redundancy, with rank correlations saturating at around 50% of instances in most benchmarks.

  18. Do Language Models Understand Time?

    cs.CV 2024-12 conditional novelty 3.0 of 10

    A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.

Reference graph

Works this paper leans on

83 extracted references · 16 canonical work pages · cited by 18 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Scaling laws for generative mixed-modal language models

    Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pages 265--279. PMLR, 2023

    2023

  4. [4]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv

  5. [5]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  6. [6]

    Claude-3.5

    Anthropic. Claude-3.5. https://www.anthropic.com/news/claude-3-5-sonnet, 2024

    work page 2024

  7. [7]

    Goldfish: Vision-language understanding of arbitrarily long videos

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, J \"u rgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-language understanding of arbitrarily long videos. arXiv preprint arXiv:2407.12679, 2024

    arXiv 2024

  8. [8]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

Show all 83 references
  1. [9]

    V-jepa: Latent video prediction for visual representation learning

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. arXiv preprint arXiv:2404.08471, 2023

    2023 arXiv

  2. [10]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  3. [11]

    Revisiting the ``Video'' in Video-Language Understanding

    Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the ``Video'' in Video-Language Understanding . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  4. [12]

    Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024

    arXiv

  5. [13]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    2024 arXiv

  6. [14]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 b

    2024 arXiv

  7. [15]

    Exploring the design space of visual context representation in video mllms

    Yifan Du, Yuqi Huo, Kun Zhou, Zijia Zhao, Haoyu Lu, Han Huang, Wayne Xin Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. Exploring the design space of visual context representation in video mllms. arXiv preprint arXiv:2410.13694, 2024

    arXiv

  8. [16]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023, 2024

    2024 arXiv

  9. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  10. [18]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017

    2017

  11. [19]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  12. [20]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651--4664. PMLR, 2021

    2021

  13. [21]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700--...

    2024

  14. [22]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML), 2024

    2024

  15. [23]

    An image grid can be worth a video: Zero-shot video question answering using a vlm

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth a video: Zero-shot video question answering using a vlm. arXiv preprint arXiv:2403.18406, 2024

    2024 arXiv

  16. [24]

    Vidcompress: Memory-enhanced temporal compression for video understanding in large language models

    Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Vidcompress: Memory-enhanced temporal compression for video understanding in large language models. arXiv preprint arXiv:2410.11417, 2024

    2024 arXiv

  17. [25]

    Building and better understanding vision-language models: insights and future directions

    Hugo Lauren c on, Andr \'e s Marafioti, Victor Sanh, and L \'e o Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024 a

    2024 arXiv

  18. [26]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024 b

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024 b

    2024 arXiv

  19. [27]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  20. [28]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024 b

    2024 arXiv

  21. [29]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024 c

    2024 arXiv

  22. [30]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023 a

    2023 arXiv

  23. [31]

    Mvbench: A comprehensive multi-modal video understanding benchmark, 2023 b

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark, 2023 b

    2023

  24. [32]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323--340. Springer, 2025

    2025

  25. [33]

    Videovista: A versatile benchmark for video understanding and reasoning

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303, 2024 d

    2024 arXiv

  26. [34]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  27. [35]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689--26699, 2024

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 a

    2024

  29. [37]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542, 2024 b

    2024 arXiv

  30. [38]

    Tempcompass: Do video llms really understand videos? arXiv preprint arXiv: 2403.00476, 2024 c

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv: 2403.00476, 2024 c

    2024 arXiv

  31. [39]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024 d

    2024 arXiv

  32. [40]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  33. [41]

    OpenAI. Gpt-4v. https://openai.com/index/gpt-4v-system-card/, 2023

    2023

  34. [42]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024

    2024

  35. [43]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  36. [44]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, joseph heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fr\' e chette, Hanna Klimczak, Raphael Koster,...

    2023

  37. [45]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024

    arXiv

  38. [46]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024

    2024 arXiv

  39. [47]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485, 2024

    2024 arXiv

  40. [48]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  41. [49]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLM s

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHYA JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal...

    2024

  42. [50]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 0 10078--10093, 2022

    2022

  43. [51]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  44. [52]

    What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    2023 arXiv

  45. [53]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  46. [54]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024 b

    2024 arXiv

  47. [55]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517, 2024 c

    2024 arXiv

  48. [56]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024 d

    2024 arXiv

  49. [57]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024 e

    2024 arXiv

  50. [58]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. https://arxiv.org/abs/2407.15754

    2024 arXiv

  51. [59]

    Freeva: Offline mllm as training-free video assistant

    Wenhao Wu. Freeva: Offline mllm as training-free video assistant. arXiv preprint arXiv:2405.07798, 2024

    2024 arXiv

  52. [60]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777--9786, June 2021

    2021

  53. [61]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017

    2017

  54. [62]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288--5296, 2016

    2016

  55. [63]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv e-prints, pages arXiv--2404, 2024 a

    2024

  56. [64]

    Slowfast-llava: A strong training-free baseline for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15841, 2024 b

    2024 arXiv

  57. [65]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  58. [66]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS, 2022

    2022

  59. [67]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2 0 (3), 2023

    2023 arXiv

  60. [68]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127--9134, 2019

    2019

  61. [69]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975--11986, 2023

    2023

  62. [70]

    Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring

    Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring. arXiv preprint arXiv:2403.09333, 2024

    2024 arXiv

  63. [71]

    Video- LL a MA : An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video- LL a MA : An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2023

    2023

  64. [72]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024 a

    2024 arXiv

  65. [73]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 b

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024 b . https://arxiv.org/abs/2407.12772

    2024 arXiv

  66. [75]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024 d

    2024 arXiv

  67. [76]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024 e

    2024 arXiv

  68. [77]

    Llava-next: A strong zero-shot video understanding model, April 2024 f

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024 f . https://llava-vl.github.io/blog/2024-04-30-llava-next-video/

    2024

  69. [79]

    Video instruction tuning with synthetic data, 2024 h

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024 h . https://arxiv.org/abs/2410.02713

    2024 arXiv

  70. [80]

    Omchat: A recipe to train multimodal language models with strong long context and video understanding

    Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang, Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma, and Ruochen Xu. Omchat: A recipe to train multimodal language models with strong long context and video understanding. arXiv preprint arXiv:2407.04923, 2024 a

    2024 arXiv

  71. [81]

    Distilling vision-language models on millions of videos

    Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, et al. Distilling vision-language models on millions of videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  72. [82]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  73. [83]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023 a

    arXiv

  74. [84]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023 b

    2023 arXiv

  75. [85]

    Video-star: Self-training enables video instruction tuning with any supervision

    Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, and Serena Yeung-Levy. Video-star: Self-training enables video instruction tuning with any supervision. arXiv preprint arXiv:2407.06189, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.