Pith. sign in

REVIEW 2 major objections 5 minor 6 cited by

Vidi: Large Multimodal Models for Video Understanding and Editing

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vidi-1.5, a 9B multimodal model, retrieves timestamp ranges matching natural-language queries in videos from 20 seconds to over an hour, outperforming GPT-4o and Gemini on the VUE-TR benchmark.

desk verdict VUE-TR is a genuinely useful long-video temporal retrieval benchmark, but the paper's central claim that Vidi beats GPT-4o/Gemini is not established because the test set is generated by the same pipeline as the training data. read the letter →

arxiv 2504.15681 v3 pith:PJXFSY5Z submitted 2025-04-22 cs.CV

Vidi Team , Celong Liu , Chia-Wen Kuo , Dawei Du , Fan Chen , Guang Chen , Jiamin Yuan , Lingxi Zhang
show 15 more authors
Lu Guo Lusha Li Longyin Wen Qingyu Chen Rachel Deng Sijie Zhu Stuart Siew Tong Jin Wei Lu Wen Zhong Xiaohui Shen Xin Gu Xing Mei Xueqiong Qu Zhenfang Chen
This is my paper · ORCID
classification cs.CV
keywords temporalretrievallongvideounderstandingmultimodallargelanguagemodelseditingaudio-visualgroundingbenchmarkdecomposedattentionmoment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Vidi, a family of large multimodal models that retrieve the time ranges in videos matching a natural-language query, with emphasis on long, real-world footage from 20 seconds to over an hour. The authors also build the VUE-TR benchmark, 428 videos and 1,598 manually refined queries in keyword, phrase, and sentence form, split across vision-only, audio-only, and vision-plus-audio. On this benchmark, Vidi-7B and Vidi-1.5-9B reach an overall AUC of IoU of 35.4% and 39.6%, while Gemini-2.0-Flash reaches 21.2%, Gemini-2.5-Pro 15.2%, and GPT-4o 13.6%. The paper argues that dense one-frame-per-second sampling plus an efficient attention design can make second-precision temporal grounding practical for hour-long video, a core step in automatic editing.

What carries the argument

The load-bearing mechanism is Decomposed Attention (D-Attn), which rewrites the causal self-attention of a pretrained LLM into visual-to-visual self-attention, text-to-text self-attention, and text-to-visual cross-attention. A diagonal variant of the visual-to-visual attention cuts the computational cost from $O(N^2)$ to $O(N)$ in the number of visual tokens, and debiased positional encodings in the cross-attention remove position bias between text and vision. Vidi also fixes the mixture weights to $\alpha_V = \alpha_A = \alpha_T = 1$ so that very long videos, which produce far more visual tokens than text tokens, do not let the multimodal stream drown out the query. Together these components allow the model to ingest one visual frame per second plus a 16 kHz audio track across hour-long inputs while keeping second-level localization.

What would settle it

Evaluate Vidi on temporal-retrieval datasets whose annotations were created independently of this paper's pipeline (QVHighlights, ActivityNet Captions, Charades-STA) using their standard evaluation protocols, or build a new benchmark with fresh human-written queries from annotators who have never seen Vidi's training data; if Vidi's margin over Gemini and GPT-4o shrinks sharply or reverses, the VUE-TR results would reflect training-test distribution overlap rather than general long-video grounding.

Watch

Extended reading notes

Core claim

Vidi's central claim is that a single multimodal model can localize natural-language queries—keywords, phrases, or sentences—to exact timestamp ranges in videos spanning 20 seconds to more than one hour, including queries that require audio alone or vision plus audio. On the VUE-TR benchmark, both Vidi and Vidi-1.5 beat the best proprietary baselines on every duration bucket for the primary AUC of IoU metric, and the gap widens on long and ultra-long videos: on videos over 60 minutes, Vidi-1.5 scores 32.3% versus 2.9% for Gemini-2.0-Flash and 2.4% for Gemini-2.5-Pro. The authors attribute this to a Decomposed Attention LLM that reduces complexity in the number of visual tokens from $O(N^2)$ to $O(N)$, fixed modality weightings that keep text and multimodal inputs balanced for any video length, and a training pipeline that progresses from synthetic timestamped data to dense-captioned real videos.

Load-bearing premise

The VUE-TR benchmark's queries and timestamps come from the same LLM-based generation pipeline that produced Vidi's post-training data, so the benchmark may not be an independent test of general long-video grounding.

Editorial extensions

If this is right

  • Hour-long raw footage becomes searchable by natural language at second-level precision, making the 'find the good take' step of video editing automatable.
  • Audio-based queries can be retrieved, not just visual events; the benchmark shows 18% of queries are audio-only and 47% require both vision and audio.
  • Temporal retrieval accuracy degrades far more slowly as video length grows than it does for the proprietary baselines, whose AUC of IoU collapses to single digits on videos over 30 minutes.
  • The model runs on a single 80 GB GPU without quantization, so it is deployable in editing products rather than requiring specialized hardware.
  • The refined IoU metric, which handles multiple predicted and ground-truth time ranges, gives the field a standard for multi-span temporal retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to evaluate Vidi on independently annotated short-video benchmarks such as QVHighlights and ActivityNet; if the margins over Gemini and GPT-4o hold there, the gains generalize beyond VUE-TR.
  • Because 47% of VUE-TR queries mix vision and audio, ablating the audio encoder (removing audio tokens from vision+audio queries) would quantify how much of Vidi's edge comes from the audio channel.
  • The fixed $\alpha = 1$ weighting is a simple remedy for the video-length imbalance; an alternative that normalizes attention contributions by token count could retain the same benefit while remaining a principled dynamic weighting scheme.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Vidi and Vidi-1.5, large multimodal models for temporal retrieval in long videos (up to and beyond one hour), together with a new benchmark, VUE-TR. The model architecture uses the decomposed-attention mechanism of Kuo et al. [13] with a modified fixed-alpha weighting, dense 1 fps visual sampling, and audio input via Whisper. Training proceeds through adapter alignment, synthetic video/audio data with timestamp supervision, real-video dense-caption/subtitle supervision, and application post-training on temporal retrieval queries. VUE-TR contains 428 videos and 1,598 queries spanning 20 seconds to over one hour, with keyword/phrase/sentence formats and vision/audio/vision+audio modality tags, and uses an AUC-of-IoU metric for multispan evaluation. The paper reports large margins over GPT-4o, Gemini-2.0-Flash, and Gemini-2.5-Pro on VUE-TR, especially for long videos.

Significance. If the results hold, VUE-TR is a valuable evaluation resource: it is substantially longer than existing temporal retrieval benchmarks, includes audio-based and vision+audio queries, covers multiple query formats, and proposes a multispan IoU metric. The model's ability to process densely sampled hour-long videos at 1 fps and 16 kHz audio on a single 80 GB GPU is practically meaningful for video editing workflows. The benchmark and model are accompanied by substantial engineering effort, including a multi-stage training pipeline with 30 million timestamped samples. However, the significance of the head-to-head comparison depends on VUE-TR being an independent test; as discussed below, the current design leaves that condition unmet, and the API constraints on baselines further weaken the comparison. The paper's architecture contribution beyond the authors' prior D-Attn work is also modest.

major comments (2)
  1. [§5, §6, §2] The VUE-TR benchmark is not independent of Vidi's post-training data in the way required to support the abstract's claim. Section 5 describes an annotation pipeline in which LLM-generated queries are derived from dense captions and subtitles and then verified by human annotators; Section 6 states that VUE-TR annotations are "manually curated using a robust annotation pipeline described in Section 5"; and Section 2 states that the temporal retrieval training data "mirror the structure of the VUE-TR benchmark." Since Section 4.3 trains Vidi directly on the same dense captions and sentence-level subtitles, Vidi is trained on the exact mapping that VUE-TR evaluates, while the API baselines are not. Please report (a) whether any of the 428 benchmark videos appear in the roughly 1M-video training corpus, (b) the distribution of n-gram or embedding similarity between benchmark queries and post-training queries, and (c) results on a held-out set of queries written by annotators directly from raw video, without LLM generation. Without (a)–(c), the reported margins may reflect distributional overlap rather than general temporal grounding ability.
  2. [Table 3, §7.2] The baseline comparison is asymmetric in ways that can only lower the proprietary models' scores. GPT-4o receives at most 120 uniformly sampled frames and no audio, Gemini long videos are downscaled to 256 pixels to satisfy the 100 MB upload limit, and Gemini-2.5-Pro "often" produces empty outputs due to content filtering. The paper does not state how empty outputs were parsed or scored, nor does it provide an equalized control (e.g., Vidi with 120 frames, Vidi without audio, Vidi at 256p) to quantify the effect of these constraints. Please report such controls and the empty-output rate and score handling; otherwise the claim that Vidi "significantly outperforms" these systems is not established.
minor comments (5)
  1. [Abstract, §7.2, §8] There are several typos and spacing errors: the abstract says "video understand editing," §7.2 says "There models" instead of "These models," and the Section 8 heading appears as "T emporal Retrieval." Please proofread the manuscript.
  2. [§6.2, Eq. (5)] The metric definitions should specify how I(·,·) and U(·,·) are computed for multi-interval predictions and ground truths; the current prose ("interaction and union function") is not a formal definition. The summation notation also needs a clear statement of whether it runs over predicted intervals, ground-truth intervals, or all pairwise overlaps.
  3. [Table 3] The paper reports a single run per model with no variance or significance measures. Given 1,598 queries, please provide confidence intervals or at least a per-category breakdown of empty and unparseable outputs for the API baselines, especially for Gemini-2.5-Pro.
  4. [§4.2] The statement that the model "achieves over 80% accuracy on a synthetic evaluation set" lacks details on how accuracy is computed and on the size and difficulty of that set; please specify the metric and the evaluation protocol.
  5. [§2, §7.1] The paper claims efficient processing of videos exceeding two hours on a single 80 GB GPU, but no runtime, peak memory, or token-count measurements are reported; please add concrete numbers to support this practical claim.

Circularity Check

1 steps flagged · score 6.0 of 10

VUE-TR benchmark is built from the same Section 5 LLM caption/subtitle pipeline used to generate Vidi post-training data, so the headline retrieval comparison is partially circular.

  1. fitted input called prediction [Section 5 (Application Post-training), reused in Section 6; cross-ref Section 2, Section 4.3]
    "To support the temporal retrieval task, we build an annotation pipeline to generate user-like search queries and ground-truth timestamps to guide the training phase. As shown in Figure 7, we take advantage of the video clip split in Section 4.3 with dense captions to generate user-style queries and timestamp ranges."

    Section 6 states VUE-TR annotations are 'manually curated using a robust annotation pipeline described in Section 5', and Section 2 says the temporal-retrieval training data 'mirror the structure of the VUE-TR benchmark'. The benchmark queries are therefore produced by the same LLM-over-dense-captions/subtitles pipeline that generates Vidi's post-training queries, while Section 4.3 trains Vidi directly on 'caption-based localization' and 'subtitle-based localization' over those same dense captions and sentence-level subtitles. Thus the task Vidi is evaluated on is the same input-to-timestamp mapping it was trained on, while the API baselines (GPT-4o at 120 frames, Gemini at 256p) see that distribution for the first time.

full rationale

The paper is an empirical systems report rather than a mathematical derivation, and most of its content—architecture, training stages, metric definition—is self-contained engineering. The only circularity I can substantiate from the text is in the evaluation loop: VUE-TR test annotations are generated by the same annotation pipeline (Section 5) that creates Vidi's post-training queries, and Section 4.3 explicitly trains Vidi to localize the very dense captions and subtitles from which those queries are paraphrased. Because the post-training data 'mirror the structure' of the benchmark and the benchmark uses 'a robust annotation pipeline described in Section 5', the benchmark does not provide an independent test of temporal retrieval; it largely measures ability to handle the authors' own query-generation distribution. This is a partial, construction-level circularity in the central evaluation claim. The D-Attn self-citation (Kuo et al. [13]) is load-bearing for the architecture but is a legitimate prior-work citation with an explicit modification (fixed alpha weights), so I do not count it as circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The model uses no new physical or mathematical entities. The central claim rests on the D-Attn efficiency result, the hand-set sampling and attention weights, and the assumption that the benchmark annotations and query generation are both accurate and appropriately independent from training data.

free parameters (4)
  • Visual frame sampling rate = 1 fps
    Set by hand as a design choice; the model's second-level localization precision depends on this rate.
  • Audio sampling rate = 16000 Hz
    Chosen by hand to match Whisper's input format; affects audio-based query localization.
  • Alpha weighting coefficients in decomposed attention = alpha_V = alpha_A = alpha_T = 1
    Set by hand instead of the original adaptive sigmoid weights; described as empirically better but not derived (Section 3).
  • Query filtering thresholds in annotation pipeline = confidence >= 0.9; merge gap <= 0.5 s; max 10 timestamps
    Hand-set rules in the data generation pipeline (Section 5), which define both training and benchmark queries.
assumptions (5)
  • domain assumption Decomposed Attention from Kuo et al. (2025) reduces multimodal attention complexity from O(N^2) to O(N) without sacrificing performance
    The model's ability to process hour-long videos rests on this cited result from the authors' own prior work; it is not re-verified here.
  • ad hoc to paper Setting alpha weights to 1 preserves a useful approximation to the original attention
    Section 3 replaces the sigmoid-derived weights with fixed 1 without proof that the weighted sum of log-sum-exp terms remains close to true attention output.
  • domain assumption Dense sampling at 1 fps yields sufficient temporal resolution for second-level localization
    No ablation across frame rates is reported; the claim of second-level precision depends on this.
  • domain assumption SigLIP, Whisper, and Mistral/Gemma provide sufficient representations for vision-audio-text grounding
    These off-the-shelf encoders and backbones are taken as adequate without task-specific analysis.
  • domain assumption VUE-TR annotations are accurate and complete
    Two-round human annotation is described, but no inter-annotator agreement or quality metrics are reported (Section 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vidi: Large Multimodal Models for Video Understanding and Editing." pith.science (2026). https://pith.science/paper/PJXFSY5Z

@misc{pith2026250415681,
  author       = {Pith},
  title        = {Pith review of: Vidi: Large Multimodal Models for Video Understanding and Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJXFSY5Z}},
  note         = {Machine review of arXiv:2504.15681}
}
read the original abstract

Humans naturally share information with those they are connected to, and video has become one of the dominant mediums for communication and expression on the Internet. To support the creation of high-quality large-scale video content, a modern pipeline requires a comprehensive understanding of both the raw input materials (e.g., the unedited footage captured by cameras) and the editing components (e.g., visual effects). In video editing scenarios, models must process multiple modalities (e.g., vision, audio, text) with strong background knowledge and handle flexible input lengths (e.g., hour-long raw videos), which poses significant challenges for traditional models. In this report, we introduce Vidi, a family of Large Multimodal Models (LMMs) for a wide range of video understand editing scenarios. The first release focuses on temporal retrieval, i.e., identifying the time ranges within the input videos corresponding to a given text query, which plays a critical role in intelligent editing. The model is capable of processing hour-long videos with strong temporal understanding capability, e.g., retrieve time ranges for certain queries. To support a comprehensive evaluation in real-world scenarios, we also present the VUE-TR benchmark, which introduces five key advancements. 1) Video duration: significantly longer than videos of existing temporal retrival datasets, 2) Audio support: includes audio-based queries, 3) Query format: diverse query lengths/formats, 4) Annotation quality: ground-truth time ranges are manually annotated. 5) Evaluation metric: a refined IoU metric to support evaluation over multiple time ranges. Remarkably, Vidi significantly outperforms leading proprietary models, e.g., GPT-4o and Gemini, on the temporal retrieval task, indicating its superiority in video editing scenarios.

Figures

Figures reproduced from arXiv: 2504.15681 by the authors.

Figure 1
Figure 1. Temporal retrieval accuracy of different models on the proposed VUE-TR benchmark. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of temporal retrieval queries and their corresponding time ranges from the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of the Vidi architecture. Raw visual and audio inputs are first process by [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Decomposed Attention [13] equiva￾lently decomposes causal self-attention in an LLM into three components: visual-to-visual (V2V), textual-to-textual (T2T), and textual-to-visual (T2V) attentions. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: An overview of the synthetic training data generation pipeline. Visual and audio segments [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: An overview of the proposed real video training data generation pipeline. Long videos [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: An illustration of post-training data generation pipeline for temporal retrieval. Queries [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The distribution of query modality and format in the VUE-TR benchmark. This diverse [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Definition of intersection and union for temporal retrieval. Both prediction and ground [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Overall performance curves for temporal retrieval on the VUE-TR benchmark. We [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    TimeLens2 shows that a compact video MLLM can localize multiple evidence intervals in long videos by training on verified interval labels and a Wasserstein-based time-distance reward.

  2. VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A multi-tool video agent trained with dual-branch RL (BiSPO) and a new VideoSearch-QA benchmark substantially beats open-source agentic baselines on video deep research and multimodal search tasks.

  3. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  4. EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    The paper introduces an event-level hierarchical control benchmark and a training-free agentic pipeline for video-to-audio generation, claiming 40.7% better controllability and 12.5% better perceptual quality than pri...

  5. Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

    cs.AR 2025-11 conditional novelty 6.0 of 10

    Timeripple cuts vDiT self-attention compute by up to 85% by reusing partial attention scores of spatially and temporally correlated tokens across channels, with VBench quality essentially unchanged.

  6. Empowering Nanoscale Connectivity through Molecular Communication: A Case Study of Virus Infection

    cs.NI 2025-08 unverdicted novelty 4.0 of 10

    A position paper proposing molecular communication as the link layer for epidemic-control bio-nano networks, with an ORF3a-based mutation identification simulation; the provided manuscript body does not match this abstract.

Reference graph

Works this paper leans on

37 extracted references · 15 canonical work pages · cited by 6 Pith papers

  1. [13]

    Rethinking homogeneity of vision and text tokens in large vision-and-language models

    Chia-Wen Kuo, Sijie Zhu, Fan Chen, Xiaohui Shen, and Longyin Wen. Rethinking homogeneity of vision and text tokens in large vision-and-language models. CoRR, abs/2502.01906, 2025

    arXiv 2025

  2. [1]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin John- son, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin John- son, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael ...

    arXiv 2023

  3. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966, 2023

    arXiv 2023

  4. [3]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report...

    arXiv 2025

  5. [4]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Ji- aqi Wang. Internlm-xcomposer2: Mastering free-form text-image composition and compre...

    arXiv 2024

  6. [5]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. CoRR, abs/240...

    arXiv 2024

  7. [6]

    TALL: temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via language query. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 5277–5285. IEEE Computer Society, 2017

    work page 2017

  8. [7]

    Long- vale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Long- vale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. CoRR, abs/2411.19772, 2024. 16

    arXiv 2024

Show all 37 references
  1. [8]

    Fullstop: Multi- lingual deep models for punctuation prediction

    Oliver Guhr, Anne-Kathrin Schumann, Frank Bahrmann, and Hans-Joachim B¨ ohme. Fullstop: Multi- lingual deep models for punctuation prediction. In Proceedings of the Swiss Text Analytics Conference 2021, Winterthur, Switzerland, June 14-16, 2021 (held online due to COVID19 pand...

    2021

  2. [9]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. Localizing moments in video with natural language. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5804–5813. IEEE Comp...

    2017

  3. [10]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, and et al

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, and et al. Gpt-4o system card. CoRR, abs/2410.21276, 2024

    arXiv

  4. [11]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L´ elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    arXiv

  5. [12]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 706–715. IEEE Computer Society, 2017

    2017

  6. [14]

    Berg, and Mohit Bansal

    Jie Lei, Tamara L. Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virt...

    2021

  7. [15]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326, 2024

    2024 arXiv

  8. [16]

    Structured context transformer for generic event boundary detection

    Congcong Li, Xinyao Wang, Dexiang Hong, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. Structured context transformer for generic event boundary detection. CoRR, abs/2206.02985, 2022

    2022 arXiv

  9. [17]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models.CoRR, abs/2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models.CoRR, abs/2407.07895, 2024

    2024 arXiv

  10. [18]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 ...

    2023

  11. [19]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023

    2023

  12. [20]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019

    2019

  13. [21]

    Timesearch: Hierarchical video search with spotlight and reflection for human-like long video understanding

    Junwen Pan, Rui Zhang, Xin Wan, Yuan Zhang, Ming Lu, and Qi She. Timesearch: Hierarchical video search with spotlight and reflection for human-like long video understanding. CoRR, abs/2504.01407, 2025. 17

    2025 arXiv

  14. [22]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceeding...

    2023

  15. [23]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. CoRR, abs/2405.08813, 2024

    2024 arXiv

  16. [24]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 14313–1432...

    2024

  17. [25]

    Gemma 2: Improving open language models at a practical size

    Morgane Rivi` ere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´ e, and et al. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024

    2024 arXiv

  18. [26]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In IEEE/CVF Conference on Computer...

    2024

  19. [27]

    Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J

    Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. H´ enaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-la...

    2025 arXiv

  20. [28]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, De...

    2017

  21. [29]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  22. [30]

    Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content

    Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. CoRR, abs...

    arXiv

  23. [31]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. CoRR, abs/2406.08035, 2024

    2024 arXiv

  24. [32]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Info...

    2022

  25. [33]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancou...

    2024

  26. [34]

    Re-thinking temporal search for long-form video understanding

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. CoRR, abs/2504.02259, 2025. 18

    2025 arXiv

  27. [35]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 11941–11952. IEEE, 2023

    2023

  28. [36]

    Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jing- wen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm- x...

    2023 arXiv

  29. [37]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.