Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SALOVA claims that a video language model that retrieves only the segments relevant to a user's question and routes them into the model can understand hour-long untrimmed videos better than models that sample or compress the entire video…

desk verdict A serious long-video QA paper with a useful new dataset and plausible architecture, but its own ablations fail to isolate the router's contribution, so the central claim is not actually demonstrated. read the letter →

arxiv 2411.16173 v2 pith:PTZLWU7Q submitted 2024-11-25 cs.CV cs.AI

Junho Kim , Hyunjun Kim , Hosu Lee , Yong Man Ro This is my paper · ORCID
classification cs.CVcs.AI
keywords longvideounderstandinglargelanguagemodelretrieval-augmentedgenerationsegmentretrievalquestionansweringSceneWalkdatasetinstructiontuningVideo-MME
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SALOVA claims that long-form video understanding improves when a video-LLM retrieves only the segments of an untrimmed video that are relevant to the user's question and routes those segments into the language model, instead of sampling or compressing the whole video into a fixed context. The paper contributes the SceneWalk dataset—87.8K long web videos split into 1.29M segments with dense descriptions (137.5 words each on average)—and a training pipeline that teaches the model which segments matter for a query, then fuses the retrieved details with a compact global representation. On Video-MME without subtitles, SALOVA-7B reaches 53.1 overall, ahead of LongVA's 52.6, with the largest gains on medium and long videos; it also scores 44.6 on LongVideoBench validation. If the claim holds, retrieval-and-routing offers a practical way to handle hour-long videos within today's context-length limits while keeping memory overhead low.

What carries the argument

The mechanism has three cooperating pieces. First, the SceneWalk dataset provides dense per-segment captions (137.5 words on average) from 87.8K long videos, giving the model training signal about scene continuity across segment boundaries. Second, the Segment Retrieval Router—a two-layer transformer that attends to query text embeddings—produces video-to-text similarity scores, and a margin-based similarity loss trains it to match the precomputed correspondence matrices; the object is to decide which segments to route. Third, the FocusFast pathway splits the LLM input into a focus stream (the top-5 retrieved segment features, processed densely) and a fast stream (routing tokens from all segments, providing global context), so the model can answer detailed questions without losing the surrounding narrative. A dynamic token-drop mechanism in the spatio-temporal connector keeps the computational cost of processing long inputs manageable.

What would settle it

Build a subset of Video-MME where the segment containing the correct answer has low video-text and text-text similarity under the paper's thresholds, then run SALOVA-7B on it; if accuracy on this subset is no better than the 16-frame no-router baseline, the router's retrieval supervision, not the architecture, is responsible for the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that a retrieval-driven architecture can outperform whole-video processing for long, untrimmed videos. SALOVA segments the video, embeds each segment with a spatio-temporal connector, and uses a segment retrieval router to score how well each segment matches the user's query; it then feeds the top-scoring segments to the LLM while also passing compact routing tokens from all segments to preserve global context (the FocusFast pathway). The router is trained end-to-end with a similarity loss whose supervision comes from precomputed video-text and text-text correspondence scores, binarized with thresholds $\tau_{\mathrm{V2T}}=0.18$ and $\tau_{\mathrm{T2T}}=0.8$ and combined as a union. The paper reports that this design lets SALOVA-7B surpass prior open video-LLMs on Video-MME (53.1 overall, without subtitles) and set strong results on LongVideoBench (44.6 validation), with ablations showing the SceneWalk knowledge-injection stage and the FocusFast pathway are each responsible for part of the gain.

Load-bearing premise

The training signal for which video segments matter comes from precomputed similarity scores that are turned into binary labels with fixed cutoffs; if those scores misjudge relevance, the router learns to retrieve the wrong segments and the reported benefit of the pipeline rests on that proxy.

Editorial extensions

If this is right

  • Long-video QA can be scaled to hour-long inputs by routing only the relevant segments into the LLM, reducing the token and memory footprint compared with dense frame sampling.
  • The SceneWalk knowledge-injection stage is a load-bearing part of training: skipping it drops Video-MME overall from 45.3 to 43.6 in the 3B model, and the gains are largest on medium and long videos.
  • The FocusFast fusion of retrieved details with global routing tokens contributes substantially to accuracy; removing it lowers the 3B model's overall score from 45.3 to 36.9.
  • The top-$K$ retrieval count matters: increasing from 1 to 9 segments improves Video-MME overall from 43.9 to 45.3, after which performance saturates, indicating the pipeline's sweet spot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the fixed thresholds on precomputed similarity scores are the weakest link in the supervision; a version that learns the relevance threshold jointly, or filters segments with human-reviewed labels, would test how much of the gain is attributable to the retrieval supervision itself.
  • Because the reported benefit concentrates on videos longer than four minutes, one direct extension is a hybrid model that decides per video whether to use dense sampling or segment routing, which the paper itself floats as a future direction.
  • The same segment-and-route idea could transfer to other long sequential modalities such as audio or instructions, where a query-specific router could select the relevant portion of a long stream before expensive reasoning.
  • A testable extension would be to run SALOVA on streaming or incremental video inputs, where segments arrive over time and the router would need to decide when to update the retrieved set.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SALOVA, a video-LLM framework for long-form video understanding based on segment-level retrieval and routing. The authors introduce the SceneWalk dataset, comprising 87.8K long untrimmed YouTube videos partitioned into 1.29M segments with dense captions generated by VILA-1.5-13B. The architecture consists of a CLIP/SigLIP vision encoder, a Perceiver-style spatio-temporal connector, a two-layer Segment Retrieval Router trained with a similarity loss, and a FocusFast mechanism that combines top-K retrieved segment features with global routing tokens. The model is trained in three stages: cross-modal alignment, SceneWalk-based knowledge injection, and video instruction tuning. Experiments on Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, and MVBench report competitive numbers, including 53.1 overall on Video-MME without subtitles for SALOVA-7B versus 52.6 for LongVA-7B. The paper's central claim is that targeted segment retrieval and routing, rather than only denser sampling, improves long-video understanding.

Significance. If the central claim were established, the work would be a meaningful step: the SceneWalk dataset is large and segment-level dense captions of long untrimmed videos are genuinely useful for long-video research, and the retrieval-routing design is a clean way to bypass context-length limits. The paper also gives an explicit retrieval supervision objective and evaluates on multiple established benchmarks, which is a strength. However, the headline result is not currently supported by the internal ablations: the only frame-sampling ablation in Table 3 appears to show that dense 1 FPS sampling without the SR-Router matches the full pipeline exactly, so the paper does not isolate the contribution of retrieval and routing. No code or trained models are released in the manuscript, and all benchmark numbers are single runs without error bars or significance estimates. The contribution is therefore promising but conditional on a controlled ablation of the router and on reconciling the table interpretation.

major comments (4)
  1. [§5.3, Table 3 (first block)] The first block of Table 3 is labeled 'Frame sampling rate (w/o SR-Router)', but its '1 FPS' row reports exactly the same Video-MME numbers (48.3/46.3/41.1/45.3) as the full SALOVA-3B configuration in the same table. As printed, this means that at fixed 1 FPS sampling, removing the Segment Retrieval Router changes nothing on Video-MME, so the observed gains in this block could come entirely from denser frame sampling. This is the missing controlled ablation for the paper's central claim. Please either add a true controlled row (full pipeline versus the same pipeline with the SR-Router removed, at fixed 1 FPS and matched token budget) or correct the table labeling if the '1 FPS' row is intended to represent the full model. The Top-K ablation in Appendix D does not fill this gap because it varies the number of retrieved segments without comparing against a no-router baseline.
  2. [§5.3, Fig. 3] The V-NIAH comparison in Fig. 3 compares 'SALOVA-3B (16 frm sample)' against 'SALOVA-3B (1 FPS sample)', so the frame count and the presence of the retrieval/routing mechanism are varied simultaneously. This does not demonstrate that retrieval improves needle localization; the improvement could be due to the much larger number of input frames. A controlled comparison at equal frame count or equal token budget is needed before the figure can support the routing claim.
  3. [§5.2, Table 1] The headline comparison of SALOVA-7B (53.1) against LongVA-7B (52.6) on Video-MME is not controlled for backbone model, training data, vision encoder, or compute, and all benchmark numbers are reported as single runs with no error bars or statistical significance. This is not by itself disqualifying for a systems paper, but the abstract and Section 5.2 present the improvement as evidence for the retrieval mechanism. Please soften the claim or add controlled comparisons that at least match backbone and training data, and report variances or multiple seeds where feasible.
  4. [§A.2 and Eq. (1)] The retrieval supervision signal y_i used to train the SR-Router is derived by binarizing LanguageBind and SBERT similarity matrices with fixed thresholds tau_V2T=0.18 and tau_T2T=0.8 and taking the union. No sensitivity analysis, ablation, or qualitative validation of these thresholds is provided. Since the router learns entirely from this proxy supervision, the central retrieval claim depends on the thresholds being reasonable. Please include a threshold sensitivity study or at least qualitative retrieval examples that verify the positive segments are genuinely relevant to the queries.
minor comments (5)
  1. [§5.3, Table 3] The rows labeled '! ✗ !' and '! ! !' for the training-stage ablation are not defined in the table caption or in the main text; please spell out which stages are included and excluded.
  2. [Throughout] The paper uses inconsistent spelling of the model name, alternating between 'SALOVA' (abstract) and 'SALOV A' (body and figures); please choose one and use it consistently.
  3. [§5.1 vs. Table 6] Section 5.1 says the 7B 'frontier model' uses SigLIP, but Appendix D reports LongVideoBench results for 'SALOVA-7B†' using CLIP at smaller resolution; please clarify explicitly which configuration produced the Table 1 headline numbers.
  4. [Eq. (1)] The notation 'L_bce(y_i, s_i)_{N_v}_{i=1}' is malformed; it should be written as a sum over i, e.g., (1/N_v) sum_i BCE(y_i, s_i), to be unambiguous.
  5. [§3.1] The sentence 'Among the total32 coarse and diverse video categories YouTube API provided' has a missing space and should read 'total 32'; please also fix the duplicated '(ii)' in the list of video source criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; external benchmarks and end-to-end training keep the empirical claim independent of its training supervision.

full rationale

The paper's central claim is that segment retrieval and routing improve long-form video understanding, and this claim is validated on external benchmarks (Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, MVBench) with fixed public evaluation protocols. No benchmark result is produced by fitting constants to those benchmarks, and no reported number is derived by construction from the retrieval supervision. The LanguageBind and SBERT correspondence scores and thresholds in Appendix A.2 define a training signal for the Segment Retrieval Router, but they are not used to compute the external evaluation outcomes, so the evaluation remains independent of the self-constructed supervision. The skeptical concern about Table 3 is an experimental-control issue rather than circularity: the 'w/o SR-Router' 1 FPS row matches the full pipeline (45.3 overall), which suggests the router's contribution is not isolated, but this is a confound in the ablation design, not a reduction of a prediction to an input by equation or definition. There is no load-bearing self-citation, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. Accordingly, no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The framework relies on hand-set hyperparameters and domain assumptions about embedding-based relevance scoring and automatic caption quality. No new physical entities are introduced; the architectural components are software designs.

free parameters (4)
  • Retrieval supervision thresholds = tau_V2T=0.18, tau_T2T=0.8
    Used to binarize LanguageBind and SBERT similarity matrices into training labels for the segment router (Appendix A.2). The paper does not report sensitivity to these values.
  • Top-K retrieved segments = 5 in Stage 2; 9 found best in ablation
    Number of video segments concatenated in the Focus pathway. Section 5.3 shows performance saturates near 9, so the deployed top-K is a performance-tuned choice.
  • Margin parameter delta = 0.2
    The margin in the score margin loss of Eq. (1). Fixed without sensitivity analysis.
  • Max token drop rates = 0.7 (Stage 1.5), 0.4 (Stage 2)
    Dynamic token drop limits chosen per training stage to trade off computation and information retention; no systematic study.
assumptions (3)
  • domain assumption Benchmark scores on Video-MME and LongVideoBench are valid measures of long-video understanding.
    The paper's central evaluation rests on these benchmarks without justifying their reliability or using a broader set of tasks.
  • domain assumption LanguageBind and SBERT similarity scores reflect semantic relevance between video segments and text queries.
    These embedding scores are used as ground-truth supervision for the router (Section 4.1.2, Appendix A.2), assuming they align with true query-relevance.
  • domain assumption Captions generated by VILA-1.5-13B are accurate and detailed enough for training a long-video model.
    The SceneWalk dataset's captions are produced by an off-the-shelf model without large-scale human verification, yet they serve as the primary training signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis." pith.science (2026). https://pith.science/paper/PTZLWU7Q

@misc{pith2026241116173,
  author       = {Pith},
  title        = {Pith review of: SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PTZLWU7Q}},
  note         = {Machine review of arXiv:2411.16173}
}
read the original abstract

Despite advances in Large Multi-modal Models, applying them to long and untrimmed video content remains challenging due to limitations in context length and substantial memory overhead. These constraints often lead to significant information loss and reduced relevance in the model responses. With the exponential growth of video data across web platforms, understanding long-form video is crucial for advancing generalized intelligence. In this paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content through targeted retrieval process. We address two main challenges to achieve it: (i) We present the SceneWalk dataset, a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich descriptive context. (ii) We develop robust architectural designs integrating dynamic routing mechanism and spatio-temporal projector to efficiently retrieve and process relevant video segments based on user queries. Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries, thereby improving the contextual relevance of the generated responses. Through extensive experiments, SALOVA demonstrates enhanced capability in processing complex long-form videos, showing significant capability to maintain contextual integrity across extended sequences.

Figures

Figures reproduced from arXiv: 2411.16173 by the authors.

Figure 1
Figure 1. The overview of the SceneWalk dataset includes (a) dataset comparison, (b) detailed statistics, and (c) the annotation [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The network overview of SALOVA. Our framework consists of four structural components: vision encoder, ST [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison results of V-NIAH. The x/y-axis in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Detailed video duration range statistics for each [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: WordCloud analysis of the SceneWalk dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Examples of the SceneWalk dataset (i). 4 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Examples of the SceneWalk dataset (ii). 5 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Qualitative examples on Video-MME [20] with SALOVA-7B. Note that the red dashed lines indicates the top-1 relevant video segment estimation for the question. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

  2. Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Video-Ma2mba shows that a Mamba-2 backbone with multi-axis gradient checkpointing can train on very long video token sequences with near-linear memory while scoring competitively against much larger video-LLMs.

Reference graph

Works this paper leans on

73 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 4, 6

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736,

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 7

    arXiv 2023

  4. [4]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  5. [5]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 2

    1901

  6. [6]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 2

    work page 2015

  7. [7]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. arXiv preprint arXiv:2312.06742, 2023. 2

    arXiv 2023

  8. [8]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 2, 4, 7, 3

    arXiv 2024

Show all 73 references
  1. [9]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. 6

    2016 arXiv

  2. [10]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Confere...

    2024

  3. [11]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 1

    2023 arXiv

  4. [12]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 3, 7

    2024 arXiv

  5. [13]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2

    2023

  6. [14]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023. 1, 2

    2023

  7. [15]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 6

    2023 arXiv

  8. [16]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdul- mohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neu...

    2024

  9. [17]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  10. [18]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  11. [19]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 1, 6

    2019

  12. [20]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 7, 6

    2024 arXiv

  13. [21]

    Short film dataset (sfd): A benchmark for story- level video understanding.arXiv preprint arXiv:2406.10221,

    Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, and Ivan Laptev. Short film dataset (sfd): A benchmark for story- level video understanding.arXiv preprint arXiv:2406.10221,

    arXiv

  14. [22]

    Gemini, 2023

    Google. Gemini, 2023. 1

    2023

  15. [23]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam 9 Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 7

    2021 arXiv

  17. [25]

    Language is not all you need: Aligning perception with language mod- els

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language mod- els. Advances in Neural Information Processing Systems , 36, 2024. 2

    2024

  18. [26]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  19. [27]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xin- tao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358 ,

    arXiv

  20. [28]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Syst...

    2020

  21. [29]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 3, 6

    2024 arXiv

  22. [30]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 1, 2, 6

    2024 arXiv

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational Conference on Machine Learning. PMLR, 2023. 2

    2023

  24. [32]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  25. [33]

    Mo- mentdiff: Generative video moment retrieval from random to real

    Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Mo- mentdiff: Generative video moment retrieval from random to real. Advances in neural information processing systems, 36, 2024. 5

    2024

  26. [34]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 1, 3, 7

    2025

  27. [35]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 3, 7

    2023 arXiv

  28. [36]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 4, 1

    2024

  29. [37]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 2, 6, 3

    2023 arXiv

  30. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, 2023. 1, 2, 6

    2023

  31. [39]

    St-llm: Large language models are effective tem- poral learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective tem- poral learners. In European Conference on Computer Vision, pages 1–18. Springer, 2025. 1, 7

    2025

  32. [40]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 5

    2024 arXiv

  33. [41]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 1, 3, 7

    2023 arXiv

  34. [42]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  35. [43]

    OpenAI. ChatGPT. https://openai.com/blog/ chatgpt/, 2023. 1

    2023

  36. [44]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 3

    2023

  37. [45]

    GPT-4V(ision) System Card, 2023

    OpenAI. GPT-4V(ision) System Card, 2023. 7

    2023

  38. [46]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 3, 7

    2024

  39. [47]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  41. [49]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 6 10

    2020

  42. [50]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  43. [51]

    Sharegemini: Scaling up video caption data for mul- timodal large language models, 2024

    Share. Sharegemini: Scaling up video caption data for mul- timodal large language models, 2024. 7

    2024

  44. [52]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  45. [53]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  46. [54]

    Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks

    Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tat...

    2021

  47. [55]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  48. [56]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023. 2

    2023 arXiv

  49. [57]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 7, 3

    2024 arXiv

  50. [58]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 6

    2021

  51. [59]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1, 3, 7

    2024 arXiv

  52. [60]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 1

    2024 arXiv

  53. [61]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 3

    2024 arXiv

  54. [62]

    Ad- vancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Ad- vancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2022

  55. [63]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 4, 6

    2024 arXiv

  56. [64]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 3

    2024 arXiv

  57. [65]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 1, 2

    2023 arXiv

  58. [66]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 6, 7

    2019

  59. [67]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 3, 4, 6

    2023

  60. [68]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 7

    2023 arXiv

  61. [69]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 1

    2024 arXiv

  62. [70]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 1, 3, 7, 8

    2024 arXiv

  63. [71]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 6

    2024 arXiv

  64. [72]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 1, 7

    2024 arXiv

  65. [73]

    Provide a detailed description of both the visual content and the storyline depicted in the video

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, W ANG HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zong- wei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic align- ment. In The Twelfth International Conference on Lea...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.