{"id":"7913beca-ef35-4814-b685-2063c8220071","arxiv_id":"2412.14965","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"MSBench evaluates video-plus-audio to novel-style story generation; the M2S pipeline combines existing video, speech, emotion, and speaker tools with an LLM and reportedly beats video-only baselines.","lead":"This paper introduces MSBench, a new benchmark that asks AI models to turn short movie clips with audio into novel-style stories, and proposes a pipeline, M2S, that feeds video captions, speech transcripts, emotions, and speaker identities into a language model. The authors report that existing video-language models score poorly on their metrics and that M2S improves scores, but the ground truth and scoring are generated by AI from the same captions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's ground truth is built from unverified AI captions, and Section 3.4 explicitly disclaims factual grounding; the M2S comparison feeds those captions to M2S but not baselines, so the reported gains measure caption-to-story reproduction, not movie understanding.","rationale":"The paper aims to provide a benchmark and a model that improve story generation from long video with auxiliary audio information. For that claim to hold, two conditions must be met: reference stories must be accurate accounts of the source movies, and the comparison must isolate genuine multimodal understanding. The weakest point is the first condition. The references are generated automatically from captions and ASR without human verification, and Section 3.4 states the authors deliberately do not check caption-to-media faithfulness. That is a correctness risk, not a normative disagreement. A benchmark can be useful even with synthetic references if it measures something well-defined, but here the defined target is 'reproduce the auxiliary annotations as a novel', not 'understand the movie'. The comparison then compounds the problem because M2S receives those annotations as input while baselines do not, and the metrics reward alignment with those same annotations. The concrete test of a text-only baseline would distinguish 'good at integrating supplied captions' from 'understands video/audio'; if a text-only model matches M2S, the central claim fails. I agree with the reader's weakest-assumption identification; the same Section 3.4 passage is the hinge. The reader's REJECT remains appropriate, so no verdict adjustment is needed.","tokens_in":13852,"tokens_out":5055,"duration_ms":30538,"concrete_test":"Release the MSBench auxiliary JSON (video captions, ASR, emotion, speaker tags) and the LLM prompt, select 50 test videos, and have a text-only GPT-4o model generate stories from that JSON without any video or audio access. Score these stories with the paper's InfoSim, InfoDiverse, SACOR, and GPT-human metrics. If the text-only model's scores are within statistical noise of M2S-LLM's reported scores, the benchmark does not require video/audio understanding and the 15% improvement is an artifact of privileged auxiliary captions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two premises: MSBench is a valid benchmark for story generation from video/audio, and M2S-LLM's ~15% improvement over VideoChat2 is real. The first premise requires that the reference stories are faithful to the source media. They are not validated: Section 3.2 constructs them by feeding automatically extracted video captions, Whisper ASR, Emotion2Vec labels, and pyannote speaker tags to an LLM, with no human verification. Section 3.4 explicitly says virtual data is reasonable because the authors focus on 'whether the generated story contains the information in the caption... rather than whether the information in the caption is in the original video and audio.' That de-prioritization is not limited to fine-tuning data; it is the same pipeline that creates the benchmark's ground truth. If the upstream captions or emotion labels are wrong, hallucinated, or misattributed, the reference story encodes those errors and the benchmark rewards reproducing them. The second premise is undermined by an input asymmetry: M2S's pipeline (Sections 4.1-4.3) feeds those same extracted auxiliary texts into an LLM, while baseline MLLMs receive only raw video/audio. Eq. (1) InfoSim and Eq. (2) InfoDiverse compare generated text to knowledge points extracted from those same captions, so M2S is tested on a text transformation it was designed to perform, not on movie understanding. Consequently, the reported numbers do not establish that M2S understands the movie better; they may only establish that it reproduces the pipeline's own auxiliary annotations more fluently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MSBench, an automatically constructed benchmark for multi-modal story generation from long videos with auxiliary audio/text information, together with M2S, a pipeline that extracts video captions, ASR transcripts, emotion labels, and speaker tags with off-the-shelf models and then uses an LLM (optionally LoRA-finetuned) to produce novel-style stories. The authors claim that current MLLMs perform poorly under their proposed metrics and that M2S-LLM surpasses VideoChat2 by nearly 15% across key metrics. The evaluation uses reference-free-style metrics InfoSim, InfoDiverse, and ISR, plus GPT-scored qualitative dimensions such as environment, character, emotion, language, and coherence.","tokens_in":14174,"tokens_out":4017,"duration_ms":32241,"significance":"If the benchmark and the comparison were valid, the paper would address a real gap: long-video narrative generation with audio is under-evaluated, and automatic dataset construction could reduce annotation cost. The authors also release code, models, and evaluation frameworks, which is a constructive contribution. However, the central claims are not supported as stated. The reference stories are LLM-generated from automatically extracted captions without factual verification, and Section 3.4 explicitly disclaims grounding in the original video/audio. Moreover, the M2S pipeline is fed exactly the auxiliary captions that define the scoring knowledge base in Eq. (1) and Eq. (2), while baseline MLLMs receive only raw video/audio. The reported improvements therefore measure caption-to-story reproduction rather than movie understanding. Because both premises of the central claim are load-bearing and cannot be repaired by local edits, the paper is not publishable in its current form.","major_comments":[{"comment":"The reported improvement is confounded by input leakage. M2S's pipeline (Sections 4.1-4.3) feeds text captions generated by VideoChat2, Whisper, Emotion2Vec, and pyannote into an LLM, while InfoSim and InfoDiverse in Eq. (1) and Eq. (2) measure the match between the generated story and a knowledge base K constructed from the very same auxiliary captions. A baseline MLLM that only sees raw video/audio cannot access this information. Tables 2 and 3 therefore cannot support the abstract's claim of a ~15% advantage in movie understanding; they support at most the ability to reproduce given captions in novel form.","section":"Section 4 and Eq. (1)-(2)"},{"comment":"MSBench's reference stories are generated by an LLM from automatically extracted video captions, ASR transcripts, emotion labels, and speaker tags, with no human verification. Section 3.4 explicitly states that the authors care whether the generated story contains the caption information 'rather than whether the information in the caption is in the original video and audio.' Since the same unverified pipeline creates the benchmark's ground truth, errors or hallucinations in the upstream extractors are propagated into the reference stories and will be rewarded if reproduced. The benchmark therefore does not measure understanding of the source movie.","section":"Section 3.2.3 and Section 3.4"},{"comment":"The metrics are not reference-free as claimed. InfoSim and InfoDiverse require a knowledge base K, and that K is constructed from the automatic captions; ISR is a vocabulary-overlap measure, not a fluency measure independent of the task. Additionally, Eq. (2) uses a fixed cosine threshold of 0.9 with no ablation or human-correlation study, so the threshold is an unexamined free parameter that can affect rankings. The paper provides no evidence that these metrics correlate with human judgment of story quality.","section":"Section 5.1, Eq. (1)-(2)"},{"comment":"The comparison is also unfair because M2S uses VideoChat2 itself as its visual caption extractor (Section 4.1). The rows labeled 'VideoChat2 + A-GPT4o' and the LoRA variants in Table 3 therefore receive privileged text captions that the VideoChat2 baseline does not receive. A valid comparison would either give all systems the same auxiliary text, or give none, or include an ablation of M2S without its caption inputs. As presented, the numbers conflate the model architecture with the input modality.","section":"Tables 2 and 3 and Section 4.1"}],"minor_comments":[{"comment":"The heading reads 'Additional caption-stroy pair for fine-tuning LLMs'; 'stroy' should be 'story'.","section":"Section 3.4 heading"},{"comment":"The sentence 'The basic process is the same as the way MSBench generates data' is repeated verbatim twice in Section 4; one occurrence should be removed.","section":"Section 4"},{"comment":"Tables 1-4 lack captions and are not referenced consistently in the text: Section 4 refers to 'the first three rows in Table 2' while Section 5.2.1 and Section 5.2.2 both refer to 'Table 4' for different sets of results.","section":"Appendix tables"},{"comment":"Several references are malformed, with entries such as 'et al. 2019' and 'et al. 2023c' missing the first author, and one reference is duplicated under different labels ('Qwen technical report' and 'Qwen-VL').","section":"References"},{"comment":"The text says 'we propose a set of reference-free metrics: Language Fluency, Key-knowledge Relevance,' but InfoSim and InfoDiverse require a knowledge base and are not reference-free; the terminology is misleading.","section":"Section 5.1"},{"comment":"The abbreviation 'A-GPT4o' is used in tables but never defined, and no dataset statistics (number of videos, segments, story lengths, train/test split) or details of the human evaluation protocol are reported.","section":"General"}],"recommendation":"reject","confidential_remarks":"The reader's concern is well founded and central: the evaluation is circular because the M2S pipeline is fed the same captions that define the benchmark's scoring knowledge base, and the reference stories are unverified LLM outputs whose factual grounding the authors explicitly disclaim. These are load-bearing issues that cannot be fixed with local edits; the benchmark and the claimed improvement would need to be revalidated with human-verified references and symmetric model inputs. I would encourage the authors to resubmit after such a revalidation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me save you some time on this one. The paper proposes MSBench, a benchmark for long-video narrative generation with audio context, and M2S-LLM, a pipeline that stitches together off-the-shelf captioners, ASR, emotion, and speaker models into a story. The task space is real and under-served, and the 5-way decomposition of video captions, ASR, audio features, alignment, and timeline is a sensible way to lay it out. Credit where due: the authors correctly identify that current MLLMs are weak at long-form, audio-informed storytelling, and the automatic dataset generation pipeline is a pragmatic way to avoid manual annotation.\n\nThe soft spot is not in the assembly; it's in the evaluation, and it is load-bearing. M2S-LLM is fed the same auxiliary captions that later constitute the scoring knowledge base for InfoSim and InfoDiverse. Baselines see only raw video/audio. So the metrics are measuring how well M2S reproduces its own input text, not how well it understands the movie. That asymmetry alone invalidates the reported ~15% improvement over VideoChat2. The problem is compounded by the reference stories: they are LLM-generated from the same unverified captions, with no human check. Section 3.4 is honest about this, saying the authors only care whether the story contains the caption information, not whether the caption information is in the video. That disclaimer is about fine-tuning data, but the benchmark ground truth is built with the identical pipeline, so the benchmark rewards reproducing hallucinations and misattributed audio labels.\n\nOther issues are secondary but real: no code, data, or evaluation artifacts are released despite the abstract's claim; the appendix tables have missing entries and appear truncated; and there are no error bars or statistical tests. These are not minor presentation slips; they make the empirical claims impossible to verify.\n\nWho is this for? A reviewer or researcher working on multimodal storytelling benchmarks might take the task definition as a starting point, but the resource itself is not usable in current form. The fixes are clear: validate ground truth with human judges, feed baselines the same auxiliary text (or make the comparison genuinely multimodal), and release the artifacts. Until then, the central claims do not hold. My recommendation is reject, but tell the authors the task formulation is salvageable and the evaluation design is fixable.","headline":"The benchmark idea is reasonable, but the headline 15% gain is unsupported because M2S gets the answer key that the metrics then score against.","tokens_in":736,"tokens_out":912,"would_cite":false,"duration_ms":26204,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M2S-LLM beats VideoChat2 by 15% on a new movie-story benchmark.","keywords":["multi-modal story generation","video understanding","audio understanding","benchmark","narrative text generation","MLLM evaluation","LoRA fine-tuning","movie-to-story pipeline"],"falsifier":"Give a model only the ASR transcripts (no video frames or visual captions) and run it on MSBench; if it scores within a few points of M2S-LLM, then the benchmark is not testing visual understanding, and the claimed 15% gain over VideoChat2 would not be evidence of video comprehension.","tokens_in":13582,"feed_emoji":"🎬","tokens_out":7068,"duration_ms":47783,"temperature":0.7,"pith_summary":"The paper introduces MSBench, a benchmark for evaluating whether multi-modal models can turn long videos plus auxiliary audio information (speech transcripts, speaker labels, emotions, timestamps) into coherent novel-style stories. It also presents M2S, a pipeline that extracts video captions and audio features with separate specialist models and then uses an LLM to weave them into narrative text. On MSBench, current multi-modal large language models score poorly, and M2S-LLM outperforms VideoChat2 by nearly 15% on the key metrics. If these results hold, MSBench gives the field a way to test long-form, information-rich story generation without expensive manual annotation.","feed_headline":"M2S-LLM beats VideoChat2 by 15% on a new movie-story benchmark","feed_subtitle":"MSBench scores story quality from video and audio; M2S-LLM shows current MLLMs lag on long-form narrative.","key_machinery":"The load-bearing mechanism is a two-stage pipeline the paper calls M2S. Stage one uses a pool of foundation models to convert video and audio into timestamped text features: VideoChat2 for video captions, Whisper for ASR, Emotion2Vec for emotion, pyannote.audio for speaker diarization, FaceNet plus YOLOv8 for character recognition, and simple word-rate calculation for speech speed. Stage two is a large language model (GPT-4o or a LoRA-tuned open model) that receives these features with a system prompt demanding chronological ordering, full use of the audio text, and avoidance of repetition. The same pipeline generates the benchmark's ground-truth stories, and the evaluation metrics reward exactly what the pipeline is designed to produce: low repetition, high coverage of the extracted knowledge, and narrative coherence.","core_discovery":"The central claim is that story generation from long videos is best treated as a text-to-text integration problem rather than an end-to-end video-to-text problem. The paper builds MSBench by automatically extracting video captions, ASR transcripts, speaker embeddings, emotion labels, and timestamps from existing video datasets, then prompting an LLM to fuse these into a novel. It then evaluates models with reference-free metrics — intra-story repetition (ISR) for fluency, and InfoSim/InfoDiverse for how well generated stories cover the knowledge in the video-audio captions — plus GPT-based scores for environment, character, emotion, language, and storyline coherence. The experiments show that standard MLLMs like VideoChat2 miss much of the audio and long-range temporal information, while the M2S pipeline, which combines VideoChat2 with GPT-4o and LoRA-tuned LLMs, achieves about 15% higher scores on the central metrics.","pith_inferences":["The benchmark's ground truth inherits any errors in the automatically extracted captions and ASR; if those are wrong or hallucinated, MSBench would measure a model's ability to reproduce AI-generated captions rather than its understanding of the actual video and audio.","The 15% improvement is measured on MSBench's own metrics, which favor the two-stage design; a human preference study on a diverse set of movies would test whether the gain translates to perceived story quality.","The same two-stage recipe could be applied to accessibility tasks, such as generating audio descriptions for people with visual impairments, where timestamped visual and audio features are already available.","Because the virtual data used for LoRA fine-tuning is generated from captions, the pipeline's gains may come mostly from learning to format and stitch text rather than from deeper video understanding; an ablation that feeds the LLM only shuffled captions would separate these effects."],"forward_implications":["If MSBench is valid, long-video story generation can be benchmarked automatically from timestamped video and audio captions, without hiring human annotators to write full reference stories.","If the M2S pipeline is the reason for the score jump, then the strongest recipe for narrative video understanding is modular: specialized extractors plus an LLM integrator, rather than a single end-to-end video-language model.","The reference-free metrics (ISR, InfoSim, InfoDiverse) could be reused beyond this benchmark to compare any system that produces long text from structured multi-modal annotations.","M2S-LLM's output is long enough and coherent enough that the authors argue it can serve as an independent way to understand a movie, not just as a subtitle supplement."],"supporting_citations":[{"why":"Supplies the VideoChat2 model, used both as the video caption extractor inside M2S and as the main baseline that M2S-LLM outperforms.","marker":"(Li et al., 2024b)"},{"why":"Whisper provides the ASR transcripts that form the audio half of the story inputs.","marker":"(OpenAI, 2022)"},{"why":"Emotion2Vec provides per-segment emotion labels used in the audio feature captions.","marker":"(et al., 2023c)"},{"why":"pyannote.audio provides speaker diarization and speaker embeddings for matching characters to dialogue.","marker":"(et al., 2019)"},{"why":"FaceNet supplies facial identity embeddings that let the pipeline match the same character across video frames.","marker":"(Schroff et al., 2015)"},{"why":"YOLOv8 supplies person bounding boxes for the character-recognition and matching module.","marker":"(Yaseen, 2024)"},{"why":"GPT-4o acts as the LLM that fuses the timestamped video and audio features into novel-style story text.","marker":"(et al., 2024b)"},{"why":"Provides the InfoSim and InfoDiverse metric formulas that MSBench adapts into its key-knowledge relevance scores.","marker":"(Yang et al., 2024)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to adapt the open LLMs for the story-generation stage.","marker":"(et al., 2021)"}],"fun_headline_variants":["New MSBench benchmark exposes MLLM gaps in long video storytelling","M2S-LLM fuses video and audio into novels, beating current MLLMs","Story generation from video: M2S-LLM scores 15% higher on MSBench","MLLMs lag on long-video story tasks: MSBench and M2S-LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is built from automatically extracted captions, speech transcripts, and emotion labels; if those machine labels are inaccurate or hallucinated, MSBench would be measuring whether models can reproduce AI captions rather than whether they understand the movie.","fun_headline_variants_meta":{"raw":{"variants":["New MSBench benchmark exposes MLLM gaps in long video storytelling","M2S-LLM fuses video and audio into novels, beating current MLLMs","Story generation from video: M2S-LLM scores 15% higher on MSBench","MLLMs lag on long-video story tasks: MSBench and M2S-LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2399,"prompt_tokens":958,"completion_tokens":1441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":574,"tokens_out":1441,"duration_ms":9069,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:44:30.703214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a model only the ASR transcripts (no video frames or visual captions) and run it on MSBench; if it scores within a few points of M2S-LLM, then the benchmark is not testing visual understanding, and the claimed 15% gain over VideoChat2 would not be evidence of video comprehension.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Whisper provides the ASR transcripts that form the audio half of the story inputs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FaceNet supplies facial identity embeddings that let the pipeline match the same character across video frames."}],"review_version":1}