{"id":"9d9eb4d6-9b1a-4314-b1df-c5801697261f","arxiv_id":"2504.16030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":15,"one_line_summary":"A video LLM trained on millions of YouTube videos with timestamp-aligned automatic speech recognition transcripts achieves state-of-the-art 7B-scale video QA and wins against 72B models on a custom real-time commentary benchmark.","lead":"This paper trains a 7B video language model to produce real-time commentary by interleaving speech-recognition words with video frames at their exact timestamps. If the method holds up, it offers a low-cost, scalable route to streaming video understanding that reportedly outperforms much larger models on commentary quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LiveSports-3K-CC's judge rewards alignment with the ASR reference, which is exactly LiveCC's training objective; without a human-preference check, the claim of surpassing 72B models in commentary quality is not established.","rationale":"The reader's weakest assumption is exactly the one I would defend. The success of the paper's unique contribution hinges on LiveSports-3K-CC measuring commentary quality rather than transcript mimicry. The design is internally coherent: the authors filter visually grounded ASR events, use GPT-4o as judge with the ASR transcript as reference, and train LiveCC to generate ASR-like streams. This makes the metric aligned with the training objective and therefore partly circular for the 'quality' claim. This is not an internal inconsistency or evidence of bad faith; it is an evaluation-validity gap, and the proposed human study can settle it. The QA results rest on standard benchmarks and are not undermined, so no rejection is warranted. The paper should remain conditional: the headline commentary claim needs external validation or softened wording.","tokens_in":27532,"tokens_out":4455,"duration_ms":43645,"concrete_test":"Sample 100 LiveSports-3K events held out from all training. Have at least five fluent English annotators pairwise compare LiveCC-7B-Instruct versus LLaVA-Video-72B (and versus GPT-4o-08-06) without seeing the ground-truth ASR, rating informativeness, factual accuracy, naturalness, and watchability. Then correlate human choices with the GPT-4o win rates on the same 100 events. If human preference does not track the LLM judge, or does not favor LiveCC, the commentary claim should be downgraded. A secondary check: re-run the GPT-4o judge with the ground-truth reference removed or with a reference-free rubric to see how much of the win rate depends on ASR-style matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central new capability claim is that LiveCC-7B-Instruct surpasses 72B models in commentary quality (abstract; Table 4). That claim rests entirely on the LiveSports-3K-CC win rate. In Sections 4.2 and 5.1, GPT-4o serves both as the fixed opponent and as the judge, and it is instructed to prefer the response that best matches the ground-truth ASR transcript in semantic and stylistic alignment (supplementary Section 8.2). LiveCC is trained, per Section 3.2, to predict dense, timestamp-aligned ASR word sequences and to emit an ellipsis for silent frames, so it is explicitly optimized to produce ASR-like fragmentary commentary. The benchmark therefore measures closeness to the ASR transcript, not an independent notion of commentary quality. The paper's own note that Qwen models 'tend to simply caption the video rather than follow the preceding ASR context to continue the video commentary' (Table 4 caption) confirms that the task is transcript continuation. A model that mimics the reference style will win even if humans would find a fluent descriptive caption more informative. The QA results on VideoMME and OVOBench are separate and are not damaged by this concern, but the abstract's commentary-superiority claim is not supported until the metric is validated against human preference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LiveCC, a video large language model trained by densely interleaving timestamp-aligned ASR word transcripts with video frames in a streaming, autoregressive fashion. The authors construct a 5M-clip pretraining dataset (Live-CC-5M) from YouTube closed captions and a 526K-clip SFT dataset (Live-WhisperX-526K) using WhisperX, with filtering for visual grounding and talking-head removal. They introduce LiveSports-3K, a sports benchmark with a commentary win-rate track (LiveSports-3K-CC) and a QA track (LiveSports-3K-QA), and they report that LiveCC-7B-Instruct surpasses 72B models (Qwen2.5-VL-72B, LLaVA-Video-72B) in commentary quality and achieves state-of-the-art results at 7B/8B scale on VideoMME and OVOBench. The paper also releases all resources.","tokens_in":27831,"tokens_out":5190,"duration_ms":47569,"significance":"If the central claims hold, the contribution is substantial: a low-cost, scalable training paradigm that uses freely available ASR transcripts to imbue video LLMs with real-time commentary ability and improves general video QA. The paper is commendably thorough in its data pipeline design, ablations, and resource release; the ablations in Tables 1 and 2 are informative, and the OVOBench gain (59.8 vs. 50.4 for Qwen2-VL-7B) is a concrete, falsifiable improvement. However, the headline commentary-superiority claim rests entirely on a new benchmark whose metric is aligned with the training objective, which undermines the claim unless the metric is independently validated.","major_comments":[{"comment":"The LiveSports-3K-CC win-rate metric is not a validated measure of commentary quality. GPT-4o serves both as the fixed baseline opponent and as the judge, and the judge is instructed to prefer the response that best matches the ground-truth ASR transcript in semantic and stylistic alignment. LiveCC is explicitly trained to predict dense timestamp-aligned ASR word sequences and to emit an ellipsis for silent frames (Section 3.2), so the metric largely measures fidelity to the ASR reference rather than an independent notion of commentary quality. The abstract's claim that LiveCC-7B-Instruct surpasses 72B models in commentary quality therefore rests on a metric that is aligned with the training objective. Without a human-preference study or a correlation analysis showing that the judge's choices track human judgments of commentary quality, this central claim is not established. I recommend adding such validation (e.g., a 100-example human pairwise study with agreement statistics) or reframing the claim as 'better ASR-transcript alignment' rather than 'better commentary quality.'","section":"Sections 4.2, 5.1, Supplementary 8.2"},{"comment":"Because GPT-4o is both the judge and the fixed opponent, the win rates may also be affected by self-preference bias. The paper states that each prompt is evaluated twice with positions swapped, but it does not report the judge's agreement rate or how ties are resolved. I ask the authors to report these statistics, and to include at least one independent judge (e.g., a different LLM or human annotators) on a subset to confirm that the ranking is not an artifact of judge self-preference. This is particularly important because the win rates are the only evidence for the headline commentary claim.","section":"Table 4 and Supplementary Section 8.2"}],"minor_comments":[{"comment":"Several typos and inconsistencies should be fixed: 'escepically' in Section 1, 'Implemetation' in Sections 5.1 and 8, 'removel' in Figure 4b, and the inconsistent spelling 'Live-Whisper-526K' versus 'Live-WhisperX-526K'.","section":"Throughout"},{"comment":"The data pipeline diagram is dense and the flow is hard to follow; consider splitting it into two sub-figures (pretraining and SFT) or annotating the filtering thresholds directly on the edges.","section":"Figure 2"},{"comment":"The statement that 'for all models, we calculate the logits of multiple choices to select answers' is ambiguous for API-based models like GPT-4o, for which logits may not be accessible; please clarify whether generation-based evaluation is used for those models and how the logit method is applied consistently.","section":"Section 5.1"},{"comment":"The OVOBench column abbreviations (RTVP, BT, FAR) are not expanded in the caption or in the text; a footnote or definition would improve readability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's QA results on standard benchmarks appear solid and the data pipeline is a genuine contribution. However, the headline commentary claim is currently supported only by a metric that may reward ASR mimicry; if the authors can add a human-preference validation or substantially temper the claim, the paper would be suitable for publication. The self-judging setup (GPT-4o as both baseline and judge) also warrants careful reporting of agreement statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a genuine systems contribution. The idea—densely interleaving timestamp-aligned ASR words with video frames during pretraining—is simple but effective, and the authors build a convincing case that this beats treating ASR as a global caption. The Live-CC-5M and Live-WhisperX-526K datasets are real artifacts, the pipeline is described in enough detail to reproduce, and the ablations are thoughtful. The QA results on VideoMME and OVOBench are competitive for 7B/8B models and are not tainted by the commentary-evaluation concerns. Real-time low-latency commentary at 2 FPS is demonstrated with actual demos.\n\nThe soft spot is exactly where the abstract makes its boldest claim. LiveSports-3K-CC's win rate is computed with GPT-4o as both the fixed opponent and the judge, and the judge is explicitly told to prefer the response that best matches the ground-truth ASR transcript semantically and stylistically (supplementary 8.2). LiveCC is trained to produce short, fragmentary ASR-like word sequences per frame. So the benchmark largely rewards stylistic mimicry of speech transcription. It is not an independent measure of commentary quality. The paper even notes that Qwen models caption rather than continue the ASR style—which confirms the task is transcript continuation. A human-preference study could fix this; without it, the claim that 7B LiveCC surpasses 72B models in commentary quality is not established.\n\nTwo smaller issues. First, there is no reported deduplication against VideoMME, OVOBench, or MVBench. Since the training data comes from public YouTube and these benchmarks are built from YouTube videos, overlap is plausible and should be checked. Second, the ablation win rates are themselves computed with the same circular metric, so the streaming-vs-caption comparison, while suggestive, is not decisive without human evaluation.\n\nNone of this sinks the paper. The QA evidence and the data artifacts stand on their own, and the core pretraining paradigm is a useful contribution. The commentary claim needs a metric fix before it can be taken at face value.\n\nFor peer review: yes, send it out. A serious referee should focus on the LiveSports-3K-CC protocol and the dedup question. The paper is honest and reproducible enough to deserve that engagement.","headline":"Solid systems paper: timestamp-dense ASR interleaving works for QA and enables real-time commentary, but the headline commentary win-rate is measured against the model's own training target, so read that claim with caution.","tokens_in":28435,"tokens_out":2146,"would_cite":true,"duration_ms":19793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video LLM trained on timestamp-aligned speech transcripts can deliver real-time commentary and match much larger models on video QA.","keywords":["streaming video LLM","ASR pre-training","real-time commentary","timestamp interleaving","closed captions","video QA","LiveSports-3K","low-latency inference"],"falsifier":"A decisive test would be a human rating study on a sample of LiveSports-3K clips where raters, seeing only the video and the candidate commentaries, rank LiveCC-7B-Instruct against Qwen2.5-VL-72B-Instruct and LLaVA-Video-72B; if human preferences diverge from the GPT-4o judge's win rates, the commentary-quality claim loses its support. A second test—shuffling word-to-frame timestamps during pre-training—would isolate whether the dense temporal interleaving itself is the cause of the gains.","tokens_in":27296,"feed_emoji":"🎙️","tokens_out":8577,"duration_ms":74842,"temperature":0.7,"pith_summary":"This paper sets out to show that timestamp-aligned automatic speech recognition (ASR) transcripts—the closed captions that already exist for millions of YouTube videos—are enough to train a video large language model, without the expensive human- or GPT-crafted annotations usually required. The training scheme interleaves each short block of ASR words with the video frames that were on screen while the words were spoken, so the model learns to generate the few words of commentary current for each frame. On five million YouTube clips this yields a 7B model that, even before instruction tuning, can produce real-time play-by-play commentary, and after tuning outperforms 72B models on the commentary benchmark while reaching state-of-the-art 7B/8B results on standard video QA. The paper also contributes the LiveSports-3K benchmark for measuring free-form streaming commentary and temporal Who/When/What question answering.","feed_headline":"Closed captions teach a 7B video model live commentary","feed_subtitle":"Streaming pre-training interleaves ASR words with frames, beating 72B rivals on commentary at sub-second latency.","key_machinery":"The load-bearing mechanism is a training sequence that interleaves visual and textual tokens at frame-level granularity:\n$$[\\text{Con}]\\langle F_{t:t+k}\\rangle \\langle W_{t:t+k}\\rangle \\langle F_{t+k:t+2k}\\rangle \\langle W_{t+k:t+2k}\\rangle \\ldots$$\nwhere $\\langle F\\rangle$ holds the visual tokens of one or more frames, $\\langle W\\rangle$ holds the ASR words assigned to the same time interval, and $[\\text{Con}]$ is the video title, previous ASR text, or user prompt. At 2 FPS with $k=1$ second, the model predicts only the two seconds' worth of words per frame block, with an ellipsis token marking pauses, which is what makes low-latency streaming inference possible. This interleaving is the object that lets the model bind spoken language to specific visual moments rather than to the whole clip.","core_discovery":"The central discovery is that a streaming task—predicting the few spoken words that align with each pair of video frames—is a powerful and scalable pre-training objective for video LLMs. Dense timestamp-level interleaving of ASR words and frames, rather than treating the transcript as one global caption, is what teaches temporally fine-grained vision-language alignment, produces a genuinely new real-time commentary ability, and transfers to general video question answering. The resulting LiveCC-7B-Instruct, initialized from a 7B vision-language base and trained on Live-CC-5M plus instruction data, is reported to surpass 72B models in commentary quality while running at sub-half-second latency per frame, and to lead the 7B/8B class on VideoMME and OVOBench.","pith_inferences":["The paper does not test whether the LiveSports-3K-CC win-rate metric tracks human judgment; if GPT-4o's preference is driven by stylistic mimicry of ASR output, the reported margin over 72B models may not transfer to human listeners. A human rating study would settle this.","The paper observes QA performance degrading beyond 5M pre-training clips, which suggests single-source streaming ASR eventually saturates general understanding; mixing streaming ASR with captioning or QA objectives during pre-training is a natural extension it defers to future work.","The same interleaving machinery could be pointed at any visually grounded narration stream—tutorials, weather reports, live events—and the title-plus-previous-ASR context design hints at a text-memory architecture in which visual tokens are dropped periodically while text context persists, supporting arbitrarily long streams."],"forward_implications":["Live sports commentary and other frame-by-frame narration can be produced by a 7B model with sub-0.5-second latency per frame, making real-time deployment at 2 FPS plausible.","ASR and closed captions can substitute for expensive proprietary annotations during video pre-training, reducing data cost and allowing the training set to scale to millions of clips.","Streaming ASR pre-training transfers to general video QA: the ASR-only base model is already competitive, and the SFT model reaches state-of-the-art results at the 7B/8B scale on VideoMME and OVOBench.","The per-frame ellipsis/pause token lets the model distinguish temporary pauses from the true end of a response, a necessary piece for streaming generation.","The LiveSports-3K benchmark gives the community a reusable way to measure free-form streaming commentary alongside Who/When/What question answering."],"supporting_citations":[{"why":"Supplies the 7B vision-language backbone and prompt format that LiveCC initializes from and extends.","marker":"[80]"},{"why":"The closest prior video-ASR pretraining baseline, which the paper contrasts with its streaming interleaving paradigm.","marker":"[90]"},{"why":"One of the four large YouTube video sources used to build the Live-CC-5M pre-training set.","marker":"[60]"},{"why":"One of the four large YouTube video sources used to build the Live-CC-5M pre-training set.","marker":"[88]"},{"why":"One of the four large YouTube video sources used to build the Live-CC-5M pre-training set.","marker":"[96]"},{"why":"WhisperX produces the word-level timestamped transcripts used in the high-quality SFT dataset.","marker":"[7]"},{"why":"Provides the general video instruction-tuning data and the LLaVA-Video baseline used throughout the comparisons.","marker":"[105]"},{"why":"GPT-4o generates SFT prompts, serves as the fixed opponent in commentary comparisons, and acts as the judge for live commentary win rates.","marker":"[29]"},{"why":"The standard video QA benchmark where LiveCC reports state-of-the-art performance at the 7B/8B scale.","marker":"[23]"}],"fun_headline_variants":["Closed captions teach real-time commentary to a 7B video LLM","Streaming ASR words with frames yields 7B model that beats 72B on commentary","Cheap ASR data trains 7B video LLM for real-time commentary","7B video LLM from streaming captions outperforms 72B in commentary","ASR pre-training yields 7B video LLM with state-of-the-art QA and live commentary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result rests on trusting GPT-4o's judgment that a commentary is good because it resembles the ground-truth speech transcript in meaning and style, a metric that may favor a model trained to imitate exactly that style.","fun_headline_variants_meta":{"raw":{"variants":["Closed captions teach real-time commentary to a 7B video LLM","Streaming ASR words with frames yields 7B model that beats 72B on commentary","Cheap ASR data trains 7B video LLM for real-time commentary","7B video LLM from streaming captions outperforms 72B in commentary","ASR pre-training yields 7B video LLM with state-of-the-art QA and live commentary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001833,"raw_usage":{"total_tokens":7263,"prompt_tokens":1055,"completion_tokens":6208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":6096}},"tokens_in":671,"tokens_out":6208,"duration_ms":41860,"temperature":1.0,"reasoning_tokens":6096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:59.552441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be a human rating study on a sample of LiveSports-3K clips where raters, seeing only the video and the candidate commentaries, rank LiveCC-7B-Instruct against Qwen2.5-VL-72B-Instruct and LLaVA-Video-72B; if human preferences diverge from the GPT-4o judge's win rates, the commentary-quality claim loses its support. A second test—shuffling word-to-frame timestamps during pre-training—would isolate whether the dense temporal interleaving itself is the cause of the gains.","supporting_citations":[{"cited_title":"Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning","cited_arxiv_id":null,"evidence_quote":"The closest prior video-ASR pretraining baseline, which the paper contrasts with its streaming interleaving paradigm."},{"cited_title":"Ad- vancing high-resolution video-language representation with large-scale video transcriptions","cited_arxiv_id":null,"evidence_quote":"One of the four large YouTube video sources used to build the Live-CC-5M pre-training set."},{"cited_title":"MERLOT RESERVE: neural script knowledge through vision and language and sound","cited_arxiv_id":null,"evidence_quote":"One of the four large YouTube video sources used to build the Live-CC-5M pre-training set."}],"review_version":1}