{"id":"08501447-2bcc-4c55-a6f2-60c1ca662f08","arxiv_id":"2504.14693","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.","lead":"Video-MMLU is a new benchmark that tests AI models on 1,065 short math, physics, and chemistry lecture videos, with both detailed captioning and reasoning questions. More than 90 models were evaluated, and most score between 10% and 50%, with text-only models answering many questions without seeing the video.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quiz track does not establish a video-perception gap: text-only Qwen2.5-72B already scores 40.54%, and QA is generated from subtitles plus general knowledge, so the headline conclusion is overstated.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption identified is the same one that I consider load-bearing: the Quiz track's video requirement is not validated and is contradicted by the paper's own 40.54% vision-blind baseline. The proposed transcript-only check is decisive because it isolates the contribution of visual frames from the transcript and world knowledge. If transcript-only accuracy is high, the paper's core finding ('existing models generally perform poorly, with accuracy ranging from only 10% to 50%') would need to be reframed as a finding about LLM reasoning/knowledge plus limited visual reading of in-frame text, not about lecture video understanding per se. I do not see a reason to reject the paper outright: the resource is large, the captioning track likely does require perception, and the multi-model evaluation is a useful contribution. However, the headline claim should only be accepted after a transcript-only baseline and a human-validation sample are added.","tokens_in":37805,"tokens_out":6067,"duration_ms":60221,"concrete_test":"Run all 15,746 Quiz questions through a strong text-only LLM (e.g., Claude-3.5-sonnet text or GPT-4o text) with the corresponding YouTube transcript/subtitles provided but no frames, and score with the same Qwen2.5-72B judge used in the paper. If transcript-only accuracy is within about 10 points of the best video LMM's Quiz score (71.24%), or if it exceeds 60%, the Quiz track does not require visual perception and the paper's central claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Video-MMLU's central claim is that LMMs perform poorly on multi-discipline lecture understanding, with the best proprietary model at 69.3% overall. The quiz track is the evidence for reasoning, but its construct validity is the weakest link. Section 3.2 states quiz QA pairs are generated by Claude-3.5-sonnet from detailed captions AND transcribed subtitles; the supplementary prompts show many items are generic domain knowledge (e.g., 'What is the derivative of e^{2t}?' -> '2 e^{2t}'), and Table 2's vision-blind Qwen2.5-72B baseline already answers 40.54% of quiz questions with no video, no captions, and no subtitles. This is not a small floor: it means roughly two-fifths of the quiz is language/world-knowledge, and the real ceiling after adding the actual transcript is likely much higher. If a text-only model given the transcript reaches or exceeds the best video LMMs' quiz scores, the benchmark's 'video understanding' conclusion collapses into an LLM-knowledge test with occasional OCR. The paper also provides no human validation of quiz answers or of the Qwen2.5-72B judge, so the 10-50% accuracy claim may partly reflect judge/annotation artifacts. The captioning track is more clearly visual, but the headline claim is based on overall/quiz numbers and is therefore not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Video-MMLU is a new benchmark for multi-discipline lecture video understanding, comprising 1,065 short videos (10-240 seconds) from mathematics, physics, and chemistry, with two tasks: a detailed-captioning 'Notebook' track with 15 surface QA pairs per video and a reasoning 'Quiz' track with 15 open-ended QA pairs per video (15,746 pairs total). The authors describe a multi-stage annotation pipeline that combines Aria and GPT-4o captions with Claude-3.5-sonnet refinement against YouTube subtitles and manual review. They evaluate 87 models, including vision-blind LLM baselines, proprietary LMMs, and open-source image/video LMMs from 0.5B to 40B parameters, and report that proprietary models reach 43-69% overall while open-source models mostly fall in the 10-50% range. They also analyze the effect of model size, LLM backbone, and visual token compression (including AuroraCap ablations). The paper's central claim is that current LMMs perform poorly on lecture understanding, especially on tasks requiring both perception and reasoning.","tokens_in":38115,"tokens_out":4578,"duration_ms":39685,"significance":"If the benchmark is valid, it fills a real gap: existing video benchmarks often focus on short open-world clips or simple factual questions, while lecture videos require dense OCR, formula recognition, and domain reasoning; a large public benchmark with 87 evaluated models and open code/dataset would be a valuable community resource. Strengths include the scale of annotation and evaluation, the inclusion of vision-blind baselines, and the systematic study of visual-token compression. However, the construct validity of the Quiz track is not established: the vision-blind Qwen2.5-72B baseline already scores 40.54% (Table 2), and the QA generation uses subtitles as input, so a substantial fraction of quiz items may be answerable without video. Since the headline claim depends on quiz/overall performance, the current evidence does not support the conclusion that LMMs are poor at video lecture understanding specifically. The paper can become a useful resource after the authors add transcript-only baselines, human validation of question grounding and automated scoring, and a re-analysis of the performance gap.","major_comments":[{"comment":"The paper's central claim, stated in the Abstract and Section 4.2, that existing models 'generally perform poorly, with accuracy ranging from only 10% to 50%' is not supported for the Quiz track: Table 2 shows that the vision-blind Qwen2.5-72B baseline achieves 40.54% on the quiz with no video input. Since the quiz QA pairs are generated from both detailed captions and transcribed subtitles (Section 3.2), and the supplementary prompt examples in S3 include items such as 'What is the derivative of e^{2t}?' that require no visual evidence, a large fraction of quiz questions appears answerable from language knowledge alone. The authors should report a transcript-only baseline (subtitles without frames), report the share of quiz questions whose gold answer can be derived from the transcript or from general knowledge, and provide human validation that each quiz question requires visual information. Without these, the headline conclusion that LMMs are poor at video lecture understanding is overstated.","section":"Section 4.2 / Table 2"},{"comment":"The quiz generation pipeline uses Claude-3.5-sonnet to produce QA pairs from detailed captions plus transcribed subtitles, and the gold answers inherit any errors or hallucinated content from the captions. The paper reports no human validation of the quiz questions or gold answers, nor of the Qwen2.5-72B judge described in Section 4.1. Given that many evaluated models share Qwen2.5 backbones (Tables 2-4), the automated judge may be systematically biased toward models from the same family. The authors should provide a human-LLM agreement study on a stratified sample of quiz and notebook answers, and quantify judge agreement separately by model family and by OCR-heavy versus reasoning-heavy questions.","section":"Section 3.2 / S3 / Section 4.1"},{"comment":"The conclusion that 'lecture understanding in models relies more on textual content in frames than on animations' is based on discipline-level averages of quiz scores. Since the quiz is not established to be visually grounded (see above), the lower mathematics scores could reflect differences in language-only answerability or question difficulty rather than the presence of animations. The notebook track is more directly visual; the paper should either base this claim on notebook-track evidence with per-question visual-grounding labels, or temper the conclusion until such evidence is provided.","section":"Section 4.2, Finding 2"},{"comment":"The large divergence between Notebook and Quiz scores for several models—for example, Aquila-VL-2B scores 13.78 on Notebook but 34.10 on Quiz, while Mini-InternVL-Chat-4B-V1.5 scores 25.76 on Notebook and 54.20 on Quiz—suggests the two tracks are not measuring a single 'lecture understanding' construct. The paper should discuss whether Quiz scores primarily reflect LLM knowledge and OCR-based text extraction rather than video perception, and should report a per-question or per-video correlation analysis between the two tracks to clarify what each track measures.","section":"Tables 2-4"}],"minor_comments":[{"comment":"The supplementary material lists the ten YouTube channels but does not report how many videos came from each channel; a table or figure with channel-level counts would help assess creator bias in the benchmark.","section":"Section S5"},{"comment":"There are spelling and formatting inconsistencies, including 'imrpove' in Section 2.1 and the alternating 'AuroaCap'/'AuroraCap' spellings in Section S10; these should be corrected.","section":"Section 2.1 / Section S10"},{"comment":"Several cells in Table 1 contain '×' placeholders or missing statistics (e.g., for MVBench, LongVideoBench, and others); please clarify whether these entries are not applicable or simply unavailable, and include the missing values where possible.","section":"Table 1"},{"comment":"The caption reports a Jensen-Shannon distance of 0.668 between the two question-answer distributions; the paper should specify exactly which embedding vectors were used and how the distance was computed, since this number is presented as evidence of task distinctiveness.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are likely to be useful to the video-LMM community, and the scale of the evaluation is a strength. However, the central claim about a video-perception gap rests on the Quiz track, whose construct validity is currently not established; the issues are addressable with additional baselines and human validation, so I recommend major revision rather than rejection. The authors should also more clearly differentiate Video-MMLU from the concurrent Video-MMMU benchmark (reference [65]), since both target professional lecture videos."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know Video-MMLU as a new resource, but the headline finding should not be taken at face value. The dataset is a real contribution: 1,065 lecture videos, roughly 15.7k reasoning QA pairs, a dual-track design with captioning and quiz, and a very broad evaluation of 87+ models. The captioning track is genuinely visual—surface questions ask about colors, positions, formulas—and the pipeline is transparent about how captions and questions were built. That part will get used.\n\nThe soft spot is the quiz track. The paper claims existing models perform poorly on lecture understanding, with accuracy mostly between 10% and 50%. But Table 2 shows the vision-blind Qwen2.5-72B baseline scores 40.54% on the quiz with no video at all. That means roughly two-fifths of the quiz is answerable from world knowledge or language patterns. Since the quiz questions were generated from captions AND transcribed subtitles (Section 3.2), the transcript-only ceiling could be much higher. The paper does not report a transcript-only baseline, which is a serious omission. If a text-only model with subtitles matches or exceeds the best video LMMs on this track, the central conclusion—that LMMs are poor at video lecture understanding—collapses into a general-knowledge test with occasional OCR.\n\nTwo more issues, smaller but real. The judge for correctness is Qwen2.5-72B, with no human validation or second judge; the quiz answers and the judge are both LLM-generated, so artifacts are possible. The video distribution is heavily math-skewed (90.3% math), so the \"multi-discipline\" claim is weaker than the title suggests.\n\nWhat the paper does well: the captioning track looks solid, the token-compression analysis is interesting, and the evaluation breadth is impressive for a new benchmark. The dataset and code appear to be released, which strengthens the empirical contribution.\n\nWho this is for: anyone building or evaluating video LMMs for educational content. The captioning track is usable now; the quiz track needs re-validation before its results should be cited.\n\nRecommendation: send to peer review, but flag the quiz-track construct validity as a major-revision issue. The authors should add a transcript-only baseline, filter or re-label video-independent questions, and validate the judge against human ratings. If they do that, this could be a solid benchmark paper.","headline":"A large and useful captioning benchmark for lecture videos, but the quiz track does not establish a video-perception gap: the vision-blind baseline already scores 40.5%.","tokens_in":38673,"tokens_out":3603,"would_cite":true,"duration_ms":32105,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video-MMLU, a benchmark of 1,065 lecture videos in mathematics, physics, and chemistry, reports that current video models score only 10-50%, with the best model at 69.3%, showing multimodal AI cannot yet reliably perceive and reason over…","keywords":["video understanding benchmark","lecture comprehension","large multimodal models","visual token compression","video question answering","dense video captioning","multi-discipline reasoning","educational video"],"falsifier":"Run the Quiz track with subtitles or transcripts only, feeding no frames to a strong text-only model. The paper's own text baseline already reaches 40.54%, so if a transcript-only model approaches Claude-3.5-sonnet's 72.7% quiz score, the benchmark would be measuring language reasoning rather than video understanding; a human audit answering quiz questions from transcripts alone would settle the same question.","tokens_in":37613,"feed_emoji":"🎓","tokens_out":8985,"duration_ms":70634,"temperature":0.7,"pith_summary":"Video-MMLU is a benchmark built from 1,065 short lecture videos in mathematics, physics, and chemistry that asks multimodal AI models (systems pairing a language model with visual input) to do two things: write detailed captions of what appears on screen, and answer open-ended quiz questions about the material. The paper's central claim is that current video-capable large multimodal models do poorly at both, scoring mostly between 10% and 50%, with the strongest proprietary model reaching 69.3%. The authors interpret this as a genuine capability gap: models can read text that appears in frames, but they struggle to follow formulas, theorem demonstrations, and animated visual arguments, particularly in mathematics. The stakes are practical - if the benchmark measures what it claims, then state-of-the-art multimodal systems cannot yet reliably serve as students of dense educational video, and lecture comprehension needs perception and reasoning methods of its own.","feed_headline":"AI models score just 10-50% on video lecture understanding","feed_subtitle":"A benchmark of 1,065 math, physics, and chemistry lectures shows even the best model only hits 69%.","key_machinery":"The instrument is the benchmark itself: 1,065 lecture videos (average length 109 seconds, 90.3% mathematics, 3.6% physics, 6.1% chemistry) from ten YouTube channels, each paired with a ground-truth caption and 15 quiz questions. Two tracks separate perception from reasoning: the Notebook track scores generated captions against the ground truth using VDCscore, a divide-and-conquer metric that decomposes long captions into short question-answer pairs, while the Quiz track presents 15,746 open-ended reasoning questions with answers capped at 15 words. The ground truth is produced by a multi-stage pipeline in which Aria writes a global video caption, GPT-4o captions individual keyframes, and Claude-3.5-sonnet merges these and cross-checks every formula and number against the video's subtitles, so the entire benchmark rests on these AI-generated captions and answers.","core_discovery":"Evaluating 87 models on Video-MMLU, the paper reports that both proprietary and open-source large multimodal models mostly score between 10% and 50%, with Claude-3.5-sonnet the strongest at 69.3% overall (67.4% on captioning and 72.7% on quiz). Models do better in physics and chemistry, where lectures carry much of their content as on-screen text, and clearly worse in mathematics, where the meaning lives in formulas and changing visualizations; the paper takes this as evidence that current LMMs primarily extract textual content from frames rather than reason over dynamic abstract representations. The study further finds that model size correlates weakly with captioning quality (r = 0.18) but more strongly with quiz performance (r = 0.40), that scaling the language-model backbone helps with diminishing returns, and that video-specialized models do not consistently beat image-based models on this benchmark.","pith_inferences":["A control experiment follows directly from the paper's own numbers: a text-only model scores 40.54% on the Quiz track, so re-running the quiz with subtitles but no frames would separate the visual-perception gap from the language-reasoning gap.","A human audit of a random sample of quiz questions, answered from transcripts alone versus with the video, would test whether the benchmark measures video-specific understanding at all.","The mathematics deficit may be driven by dynamic OCR failure - formulas changing frame to frame - rather than missing reasoning; isolating static formulas from animated derivations would locate the true bottleneck."],"forward_implications":["Even the best model scores only 69.3%, so lecture-video understanding remains an open problem for both proprietary and open-source systems.","Models succeed where lectures put information in text and fail where meaning is carried by formulas and animated demonstrations, with mathematics the weakest discipline (mean 21.8 versus 27.0 for physics and 27.3 for chemistry).","Compressing visual tokens to roughly 16-300 per frame preserves most performance, and some compressed models surpass their full-token base, so efficiency and accuracy are not strictly opposed, while ultra-low token counts (2 per frame) collapse performance.","Scaling the language-model backbone improves quiz performance but with sharply diminishing returns, and architecture choice matters as much as scale.","Video-specialized models do not clearly beat image-based models here, suggesting OCR-heavy image training transfers better to lecture content than video temporal training."],"supporting_citations":[{"why":"Aria generates the global video caption that forms the structural backbone of the annotation pipeline.","marker":"[79]"},{"why":"AuroraCap supplies the VDCscore divide-and-conquer metric used to score the Notebook captioning track.","marker":"[21]"},{"why":"Qwen2.5-72B is the LLM judge that grades every predicted answer on both tracks.","marker":"[143]"},{"why":"Video-MMMU is the closest prior professional-video knowledge benchmark that Video-MMLU extends and compares itself against.","marker":"[65]"},{"why":"MMMU is the static multi-discipline reasoning benchmark whose video counterpart Video-MMLU is built to be.","marker":"[155]"},{"why":"Provides the vision-blind baseline methodology used to control for answering without visual input.","marker":"[26]"},{"why":"Cambrian-1 is the other cited source for the vision-blind baseline setup.","marker":"[117]"}],"fun_headline_variants":["AI scores 10-50% on video lectures; best model 69%","Benchmark: AI can't follow lectures, scores 10-50%","Video lecture benchmark: AI tops out at 69%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quiz questions and their gold answers were generated by an AI model from captions and subtitles and never verified by humans against the videos, so the benchmark's claim to measure video perception depends on those questions actually requiring the video rather than just language and general knowledge.","fun_headline_variants_meta":{"raw":{"variants":["AI scores 10-50% on video lectures; best model 69%","Benchmark: AI can't follow lectures, scores 10-50%","Video lecture benchmark: AI tops out at 69%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3254,"prompt_tokens":853,"completion_tokens":2401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":2339}},"tokens_in":469,"tokens_out":2401,"duration_ms":14771,"temperature":1.0,"reasoning_tokens":2339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:42:32.086399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Quiz track with subtitles or transcripts only, feeding no frames to a strong text-only model. The paper's own text baseline already reaches 40.54%, so if a transcript-only model approaches Claude-3.5-sonnet's 72.7% quiz score, the benchmark would be measuring language reasoning rather than video understanding; a human audit answering quiz questions from transcripts alone would settle the same question.","supporting_citations":[],"review_version":1}