Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Movie2Story: A framework for understanding videos and telling stories in the form of novel text

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read M2S-LLM beats VideoChat2 by 15% on a new movie-story benchmark.

desk verdict The benchmark idea is reasonable, but the headline 15% gain is unsupported because M2S gets the answer key that the metrics then score against. read the letter →

arxiv 2412.14965 v2 pith:L5GBIOCP submitted 2024-12-19 cs.CV cs.AIcs.CL

Kangning Li , Zheyang Jia , Anyu Ying This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multi-modalstorygenerationvideounderstandingaudiobenchmarknarrativetextMLLMevaluationLoRAfine-tuningmovie-to-storypipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MSBench, a benchmark for evaluating whether multi-modal models can turn long videos plus auxiliary audio information (speech transcripts, speaker labels, emotions, timestamps) into coherent novel-style stories. It also presents M2S, a pipeline that extracts video captions and audio features with separate specialist models and then uses an LLM to weave them into narrative text. On MSBench, current multi-modal large language models score poorly, and M2S-LLM outperforms VideoChat2 by nearly 15% on the key metrics. If these results hold, MSBench gives the field a way to test long-form, information-rich story generation without expensive manual annotation.

What carries the argument

The load-bearing mechanism is a two-stage pipeline the paper calls M2S. Stage one uses a pool of foundation models to convert video and audio into timestamped text features: VideoChat2 for video captions, Whisper for ASR, Emotion2Vec for emotion, pyannote.audio for speaker diarization, FaceNet plus YOLOv8 for character recognition, and simple word-rate calculation for speech speed. Stage two is a large language model (GPT-4o or a LoRA-tuned open model) that receives these features with a system prompt demanding chronological ordering, full use of the audio text, and avoidance of repetition. The same pipeline generates the benchmark's ground-truth stories, and the evaluation metrics reward exactly what the pipeline is designed to produce: low repetition, high coverage of the extracted knowledge, and narrative coherence.

What would settle it

Give a model only the ASR transcripts (no video frames or visual captions) and run it on MSBench; if it scores within a few points of M2S-LLM, then the benchmark is not testing visual understanding, and the claimed 15% gain over VideoChat2 would not be evidence of video comprehension.

Watch

Extended reading notes

Core claim

The central claim is that story generation from long videos is best treated as a text-to-text integration problem rather than an end-to-end video-to-text problem. The paper builds MSBench by automatically extracting video captions, ASR transcripts, speaker embeddings, emotion labels, and timestamps from existing video datasets, then prompting an LLM to fuse these into a novel. It then evaluates models with reference-free metrics — intra-story repetition (ISR) for fluency, and InfoSim/InfoDiverse for how well generated stories cover the knowledge in the video-audio captions — plus GPT-based scores for environment, character, emotion, language, and storyline coherence. The experiments show that standard MLLMs like VideoChat2 miss much of the audio and long-range temporal information, while the M2S pipeline, which combines VideoChat2 with GPT-4o and LoRA-tuned LLMs, achieves about 15% higher scores on the central metrics.

Load-bearing premise

The benchmark's ground truth is built from automatically extracted captions, speech transcripts, and emotion labels; if those machine labels are inaccurate or hallucinated, MSBench would be measuring whether models can reproduce AI captions rather than whether they understand the movie.

Editorial extensions

If this is right

  • If MSBench is valid, long-video story generation can be benchmarked automatically from timestamped video and audio captions, without hiring human annotators to write full reference stories.
  • If the M2S pipeline is the reason for the score jump, then the strongest recipe for narrative video understanding is modular: specialized extractors plus an LLM integrator, rather than a single end-to-end video-language model.
  • The reference-free metrics (ISR, InfoSim, InfoDiverse) could be reused beyond this benchmark to compare any system that produces long text from structured multi-modal annotations.
  • M2S-LLM's output is long enough and coherent enough that the authors argue it can serve as an independent way to understand a movie, not just as a subtitle supplement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's ground truth inherits any errors in the automatically extracted captions and ASR; if those are wrong or hallucinated, MSBench would measure a model's ability to reproduce AI-generated captions rather than its understanding of the actual video and audio.
  • The 15% improvement is measured on MSBench's own metrics, which favor the two-stage design; a human preference study on a diverse set of movies would test whether the gain translates to perceived story quality.
  • The same two-stage recipe could be applied to accessibility tasks, such as generating audio descriptions for people with visual impairments, where timestamped visual and audio features are already available.
  • Because the virtual data used for LoRA fine-tuning is generated from captions, the pipeline's gains may come mostly from learning to format and stitch text rather than from deeper video understanding; an ablation that feeds the LLM only shuffled captions would separate these effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MSBench, an automatically constructed benchmark for multi-modal story generation from long videos with auxiliary audio/text information, together with M2S, a pipeline that extracts video captions, ASR transcripts, emotion labels, and speaker tags with off-the-shelf models and then uses an LLM (optionally LoRA-finetuned) to produce novel-style stories. The authors claim that current MLLMs perform poorly under their proposed metrics and that M2S-LLM surpasses VideoChat2 by nearly 15% across key metrics. The evaluation uses reference-free-style metrics InfoSim, InfoDiverse, and ISR, plus GPT-scored qualitative dimensions such as environment, character, emotion, language, and coherence.

Significance. If the benchmark and the comparison were valid, the paper would address a real gap: long-video narrative generation with audio is under-evaluated, and automatic dataset construction could reduce annotation cost. The authors also release code, models, and evaluation frameworks, which is a constructive contribution. However, the central claims are not supported as stated. The reference stories are LLM-generated from automatically extracted captions without factual verification, and Section 3.4 explicitly disclaims grounding in the original video/audio. Moreover, the M2S pipeline is fed exactly the auxiliary captions that define the scoring knowledge base in Eq. (1) and Eq. (2), while baseline MLLMs receive only raw video/audio. The reported improvements therefore measure caption-to-story reproduction rather than movie understanding. Because both premises of the central claim are load-bearing and cannot be repaired by local edits, the paper is not publishable in its current form.

major comments (4)
  1. [Section 4 and Eq. (1)-(2)] The reported improvement is confounded by input leakage. M2S's pipeline (Sections 4.1-4.3) feeds text captions generated by VideoChat2, Whisper, Emotion2Vec, and pyannote into an LLM, while InfoSim and InfoDiverse in Eq. (1) and Eq. (2) measure the match between the generated story and a knowledge base K constructed from the very same auxiliary captions. A baseline MLLM that only sees raw video/audio cannot access this information. Tables 2 and 3 therefore cannot support the abstract's claim of a ~15% advantage in movie understanding; they support at most the ability to reproduce given captions in novel form.
  2. [Section 3.2.3 and Section 3.4] MSBench's reference stories are generated by an LLM from automatically extracted video captions, ASR transcripts, emotion labels, and speaker tags, with no human verification. Section 3.4 explicitly states that the authors care whether the generated story contains the caption information 'rather than whether the information in the caption is in the original video and audio.' Since the same unverified pipeline creates the benchmark's ground truth, errors or hallucinations in the upstream extractors are propagated into the reference stories and will be rewarded if reproduced. The benchmark therefore does not measure understanding of the source movie.
  3. [Section 5.1, Eq. (1)-(2)] The metrics are not reference-free as claimed. InfoSim and InfoDiverse require a knowledge base K, and that K is constructed from the automatic captions; ISR is a vocabulary-overlap measure, not a fluency measure independent of the task. Additionally, Eq. (2) uses a fixed cosine threshold of 0.9 with no ablation or human-correlation study, so the threshold is an unexamined free parameter that can affect rankings. The paper provides no evidence that these metrics correlate with human judgment of story quality.
  4. [Tables 2 and 3 and Section 4.1] The comparison is also unfair because M2S uses VideoChat2 itself as its visual caption extractor (Section 4.1). The rows labeled 'VideoChat2 + A-GPT4o' and the LoRA variants in Table 3 therefore receive privileged text captions that the VideoChat2 baseline does not receive. A valid comparison would either give all systems the same auxiliary text, or give none, or include an ablation of M2S without its caption inputs. As presented, the numbers conflate the model architecture with the input modality.
minor comments (6)
  1. [Section 3.4 heading] The heading reads 'Additional caption-stroy pair for fine-tuning LLMs'; 'stroy' should be 'story'.
  2. [Section 4] The sentence 'The basic process is the same as the way MSBench generates data' is repeated verbatim twice in Section 4; one occurrence should be removed.
  3. [Appendix tables] Tables 1-4 lack captions and are not referenced consistently in the text: Section 4 refers to 'the first three rows in Table 2' while Section 5.2.1 and Section 5.2.2 both refer to 'Table 4' for different sets of results.
  4. [References] Several references are malformed, with entries such as 'et al. 2019' and 'et al. 2023c' missing the first author, and one reference is duplicated under different labels ('Qwen technical report' and 'Qwen-VL').
  5. [Section 5.1] The text says 'we propose a set of reference-free metrics: Language Fluency, Key-knowledge Relevance,' but InfoSim and InfoDiverse require a knowledge base and are not reference-free; the terminology is misleading.
  6. [General] The abbreviation 'A-GPT4o' is used in tables but never defined, and no dataset statistics (number of videos, segments, story lengths, train/test split) or details of the human evaluation protocol are reported.

Circularity Check

3 steps flagged · score 7.0 of 10

MSBench's reference stories and InfoSim/InfoDiverse knowledge base are constructed from the same unvalidated captions that M2S is explicitly prompted to 'use all of' in its input; the reported 15% gain is a caption-reproduction effect, not measured movie understanding.

  1. self definitional [Section 5.1 (Eqs. 1-2) with Fig. 1 query prompt; Section 3.2.3]
    "For evaluating the relevance of key knowledge in the generated story, we define a knowledge base that includes both visual and audio information ... Information Similarity (InfoSim): Measures the alignment between the knowledge points in the story and the knowledge repository. ... Information Diversity (InfoDiverse): Evaluates the breadth of the knowledge used in the story. ... This is input video captions: … This is input audio captions: … You must use all of the audio text in the final output."

    The knowledge repository scored by InfoSim/InfoDiverse is the caption/ASR/emotion text that MSBench was built from, and M2S's LLM receives exactly that text as its input, with the instruction to use all of the audio text. Eq. (1) measures similarity between the story and knowledge points; Eq. (2) counts coverage of those knowledge points. A model that reformulates the supplied captions therefore maximizes both metrics without any grounding in the video or audio. The 'multi-modal story generation' score reduces, by construction, to text-to-text fidelity to the authors' own caption layer.

  2. fitted input called prediction [Section 4 (M2S pipeline) and Tables 1-2; compare Section 3.2]
    "To fill the gap, we develop a robust MLLM-LLM pipeline, which is dubbed as M2S(movie to story). The basic process is the same as the way MSBench generates data."

    M2S is the benchmark's own data-generation pipeline re-run and then evaluated on the benchmark it generates. Its video captions come from VideoChat2 and its audio features from Whisper/Emotion2Vec/pyannote, the same toolset used to create the MSBench reference stories, and its final story is produced by an LLM from those auxiliary texts. Baseline MLLMs receive only raw video/audio. The ~15% advantage is thus the expected consequence of giving M2S the gold auxiliary text while withholding it from the baselines; the comparison does not measure better movie understanding, only access to the reference-generating inputs.

1 more flagged steps
  1. other [Section 3.4 (Additional caption-story pair for fine-tuning LLMs)]
    "Because at this stage, we only need to adjust the long text generated in the form of a novel, focusing mainly on whether the generated story contains the information in the caption and temporal alignment, rather than whether the information in the caption is in the original video and audio. Therefore, the data generated in this way is reasonable."

    The authors explicitly define the task as reproducing caption information and temporarily de-prioritize whether captions are true of the source media. Since the same caption-generation chain creates both the MSBench reference stories and the knowledge base used by InfoSim/InfoDiverse, the benchmark's ground truth is never anchored to the video/audio. Any hallucination or error in the caption layer becomes part of the gold standard and is rewarded when reproduced. This makes the benchmark self-referential: it tests consistency with an unvalidated caption layer rather than understanding of the movie.

full rationale

No self-citation chain is load-bearing here; the circularity is structural. The paper's central claims have two mutually reinforcing loops: (1) MSBench ground truth is LLM-generated from automatically extracted captions/ASR/emotion/speaker tags, and the Section 3.4 text explicitly waives verification that captions are grounded in the original media; (2) the InfoSim/InfoDiverse metrics compare outputs to a knowledge repository built from those same captions, while M2S's system prompt feeds those captions into the model and orders it to 'use all of the audio text'. The benchmark therefore measures how faithfully M2S can turn the reference knowledge into novel prose, not how well it understands the movie. The comparison against VideoChat2 and other MLLMs is also asymmetric: M2S gets the auxiliary text as input and the baselines do not, so the reported ~15% improvement is the expected outcome of the setup, not an empirical discovery about model capabilities. The score is 7 rather than higher because the paper is explicit about its text-to-text objective and the pipeline could in principle be revalidated with human-verified captions and input-matched baselines; as written, however, the central evaluation reduces by construction to caption reproduction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three shaky pillars: automatically generated ground truth, a scoring metric built from the same captions used as model input, and an unfair comparison in which M2S receives auxiliary captions while baselines do not. The paper adds no free physical constants or invented entities; the free parameters are benchmark design choices such as the 20-second segment length and the 0.9 threshold in InfoDiverse.

free parameters (2)
  • InfoDiverse matching threshold = 0.9
    Equation (2) counts a knowledge point as covered when cosine similarity exceeds 0.9; this hand-picked threshold determines InfoDiverse scores, which are reported as key quantitative results.
  • Video segment length = 20 seconds
    Section 3.2.1 divides videos into 20 second segments; this choice affects how many captions and timestamps are available and therefore the difficulty of the story task.
assumptions (5)
  • domain assumption Automatically extracted video captions, ASR transcripts, emotions, and speaker labels are accurate enough to serve as ground-truth auxiliary information.
    The entire MSBench dataset is built from these automated tools without human verification; Section 3.2 describes the pipeline.
  • ad hoc to paper GPT-generated virtual training and evaluation data is valid even if its content is not grounded in the original video or audio.
    Section 3.4 explicitly states this is reasonable because the focus is on whether the generated story contains the caption information, not whether the caption is true of the media.
  • ad hoc to paper InfoSim, InfoDiverse, and ISR, adapted from Yang et al. (2024), are valid reference-free measures of story quality.
    The formulas are modified and no validation, human correlation, or error analysis is provided in Section 5.1.
  • standard math Embedding-based cosine similarity and set operations used in the metrics are standard and correctly implemented.
    Equations (1) and (2) rely on normalized embeddings and set unions without derivation; no proof is given, but they are routine operations.
  • ad hoc to paper Baselines and M2S can be compared fairly even though M2S receives auxiliary captions and baselines receive only raw video.
    Tables 1 and 2 compare VideoChat2, Internvideo2, and VideoLlava with and without GPT-4o, but the M2S pipeline consumes the same caption features that define the scoring knowledge base.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Movie2Story: A framework for understanding videos and telling stories in the form of novel text." pith.science (2026). https://pith.science/paper/L5GBIOCP

@misc{pith2026241214965,
  author       = {Pith},
  title        = {Pith review of: Movie2Story: A framework for understanding videos and telling stories in the form of novel text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L5GBIOCP}},
  note         = {Machine review of arXiv:2412.14965}
}
read the original abstract

In recent years, large-scale models have achieved significant advancements, accompanied by the emergence of numerous high-quality benchmarks for evaluating various aspects of their comprehension abilities. However, most existing benchmarks primarily focus on spatial understanding in static image tasks. While some benchmarks extend evaluations to temporal tasks, they fall short in assessing text generation under complex contexts involving long videos and rich auxiliary information. To address this limitation, we propose a novel benchmark: the Multi-modal Story Generation Benchmark (MSBench), designed to evaluate text generation capabilities in scenarios enriched with auxiliary information. Our work introduces an innovative automatic dataset generation method to ensure the availability of accurate auxiliary information. On one hand, we leverage existing datasets and apply automated processes to generate new evaluation datasets, significantly reducing manual efforts. On the other hand, we refine auxiliary data through systematic filtering and utilize state-of-the-art models to ensure the fairness and accuracy of the ground-truth datasets. Our experiments reveal that current Multi-modal Large Language Models (MLLMs) perform suboptimally under the proposed evaluation metrics, highlighting significant gaps in their capabilities. To address these challenges, we propose a novel model architecture and methodology to better handle the overall process, demonstrating improvements on our benchmark.

Figures

Figures reproduced from arXiv: 2412.14965 by the authors.

Figure 1
Figure 1. Workflow template laying the foundation for in-depth analysis of the video content. Image Character Recognition Our goal is to match real-person names with video characters by recognizing and matching individuals across frames in long videos, despite challenges like vary￾ing appearances and multiple viewpoints. We use Facenet(Schroff et al., 2015) for robust facial recognition and a text feature-based method with VI… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 6 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    arXiv 2022

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. https://arxiv.org/abs/2308.12966 Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond . Preprint, arXiv:2308.12966

    arXiv 2023

  3. [3]

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022 a . https://arxiv.org/abs/2202.00874 Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection . Preprint, arXiv:2202.00874

    arXiv 2022

  4. [4]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022 b . https://arxiv.org/abs/2212.09058 Beats: Audio pre-training with acoustic tokenizers . Preprint, arXiv:2212.09058

    arXiv 2022

  5. [5]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024. https://arxiv.org/abs/2312.14238 Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks . Preprint, arXiv:2312.14238

    arXiv 2024

  6. [6]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. https://arxiv.org/abs/2406.07476 Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms . Preprint, arXiv:2406.07476

    arXiv 2024

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    arXiv 2019

  8. [8]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://arxiv.org/abs/2010.11929 An image is worth 16x16 words: Transformers for image recognition at scale . Preprint, arXiv:2010.11929

    arXiv 2021

Show all 58 references
  1. [9]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023 arXiv

  2. [10]

    Aaron Grattafiori et al. 2024 a . https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  3. [11]

    Jiang et al

    Albert Q. Jiang et al. 2023 a . https://arxiv.org/abs/2310.06825 Mistral 7b . Preprint, arXiv:2310.06825

    2023 arXiv

  4. [12]

    Hu et al

    Edward J. Hu et al. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  5. [13]

    Hervé Bredin et al. 2019. https://arxiv.org/abs/1911.01255 pyannote.audio: neural building blocks for speaker diarization . Preprint, arXiv:1911.01255

    2019 arXiv

  6. [14]

    Jinze Bai et al. 2023 b . https://arxiv.org/abs/2309.16609 Qwen technical report . Preprint, arXiv:2309.16609

    2023 arXiv

  7. [15]

    OpenAI et al. 2024 b . https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  8. [16]

    Ziyang Ma et al. 2023 c . https://arxiv.org/abs/2312.15185 emotion2vec: Self-supervised pre-training for speech emotion representation . Preprint, arXiv:2312.15185

    2023 arXiv

  9. [17]

    FFmpeg Developers . 2023. FFmpeg tool (Version 4.4.1) . Software available from http://ffmpeg.org/ . Accessed: 2023-10-05

    2023

  10. [18]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024 a . https://arxiv.org/abs/2...

    2024 arXiv

  11. [19]

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. 2024 b . https://arxiv.org/abs/2408.05211 Vita: Towards open-source interactive omni m...

    2024 arXiv

  12. [20]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. https://arxiv.org/abs/2305.05665 Imagebind: One embedding space to bind them all . Preprint, arXiv:2305.05665

    2023 arXiv

  13. [21]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. https://arxiv.org/abs/1706.0426...

    2017 arXiv

  14. [22]

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023. https://arxiv.org/abs/2302.140...

    2023 arXiv

  15. [23]

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. https://arxiv.org/abs/1705.06950 The kinetics human action video dataset . Preprint...

    2017 arXiv

  16. [24]

    Hamza Kheddar, Mustapha Hemis, and Yassine Himeur. 2024. https://doi.org/10.1016/j.inffus.2024.102422 Automatic speech recognition using advanced deep learning approaches: A survey . Information Fusion, 109:102422

    2024

  17. [25]

    Seon-Ho Lee, Jue Wang, David Fan, Zhikang Zhang, Linda Liu, Xiang Hao, Vimal Bhat, and Xinyu Li. 2024. https://arxiv.org/abs/2412.10002 Nowyousee me: Context-aware automatic audio description . Preprint, arXiv:2412.10002

    2024 arXiv

  18. [26]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. https://arxiv.org/abs/2307.16125 Seed-bench: Benchmarking multimodal llms with generative comprehension . Preprint, arXiv:2307.16125

    2023 arXiv

  19. [27]

    Kankanhalli

    Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2020. https://doi.org/10.1109/tmm.2019.2930041 Video storytelling: Textual summaries for events . IEEE Transactions on Multimedia, 22(2):554–565

    2020

  20. [28]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024 a . https://arxiv.org/abs/2305.06355 Videochat: Chat-centric video understanding . Preprint, arXiv:2305.06355

    2024 arXiv

  21. [29]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024 b . https://arxiv.org/abs/2311.17005 Mvbench: A comprehensive multi-modal video understanding benchmark . Preprint, arXiv:2311.17005

    2024 arXiv

  22. [30]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. https://arxiv.org/abs/2311.10122 Video-llava: Learning united visual representation by alignment before projection . Preprint, arXiv:2311.10122

    2024 arXiv

  23. [31]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ Rouge: A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  24. [32]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  25. [33]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  26. [34]

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. https://arxiv.org/abs/2306.09093 Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration . Preprint, arXiv:2306.09093

    2023 arXiv

  27. [35]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. https://arxiv.org/abs/2306.05424 Video-chatgpt: Towards detailed video understanding via large vision and language models . Preprint, arXiv:2306.05424

    2024 arXiv

  28. [36]

    OpenAI. 2022. Whisper: A general-purpose speech recognition model. https://cdn.openai.com/papers/whisper.pdf. Accessed: 2023-10-06

    2022

  29. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--31...

    2002

  30. [38]

    Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koste...

    2023 arXiv

  31. [39]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  32. [40]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. https://doi.org/10.1109/cvpr.2015.7298682 Facenet: A unified embedding for face recognition and clustering . In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 815–823. IEEE

    2015

  33. [41]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  34. [42]

    Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Po-Yao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Shinji Watanabe, Abdelrahman Mohamed, Chi-Luen Feng, and Hung yi Lee. 2024. https:...

    2024 arXiv

  35. [43]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. 2024. https://arxiv.org/abs/2403.15...

    2024 arXiv

  36. [44]

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022. https://arxiv.org/abs/2212.03191 Internvideo: General video foundation models...

    2022 arXiv

  37. [45]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. https://arxiv.org/abs/2105.08276 Next-qa:next phase of question-answering to explaining temporal actions . Preprint, arXiv:2105.08276

    2021 arXiv

  38. [46]

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. 2024. https://arxiv.org/abs/2306.14899 Funqa: Towards surprising video comprehension . Preprint, arXiv:2306.14899

    2024 arXiv

  39. [47]

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia

    2017

  40. [48]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. https://doi.org/10.1109/CVPR.2016.571 Msr-vtt: A large video description dataset for bridging video and language . In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288--5296

    2016 doi

  41. [49]

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. https://arxiv.org/abs/2306.09265 Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models . Preprint, arXiv:2306.09265

    2023 arXiv

  42. [50]

    Dingyi Yang, Chunru Zhan, Ziheng Wang, Biao Wang, Tiezheng Ge, Bo Zheng, and Qin Jin. 2024. https://arxiv.org/abs/2405.14040 Synchronized video storytelling: Generating video narrations with structured storyline . Preprint, arXiv:2405.14040

    2024 arXiv

  43. [51]

    Muhammad Yaseen. 2024. https://arxiv.org/abs/2408.15857 What is yolov8: An in-depth exploration of the internal features of the next-generation object detector . Preprint, arXiv:2408.15857

    2024 arXiv

  44. [52]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024 a . https://arxiv.org/abs/2304.14178 mplug-owl: Modul...

    2024 arXiv

  45. [53]

    Xiaojun Ye, Junhao Chen, Xiang Li, Haidong Xin, Chao Li, Sheng Zhou, and Jiajun Bu. 2024 b . https://aclanthology.org/2024.lrec-main.998/ MMAD :multi-modal movie audio description . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Languag...

    2024

  46. [54]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. https://arxiv.org/abs/2308.02490 Mm-vet: Evaluating large multimodal models for integrated capabilities . Preprint, arXiv:2308.02490

    2024 arXiv

  47. [55]

    Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, Philipp Krähenbühl, and Liangzhe Yuan. 2024. https://arxiv.org/abs/2401.06129 Distilling vision-language models on millions of videos . Preprint, arXiv:2401.06129

    2024 arXiv

  48. [56]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://arxiv.org/abs/2304.10592 Minigpt-4: Enhancing vision-language understanding with advanced large language models . Preprint, arXiv:2304.10592

    2023 arXiv

  49. [57]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.