{"id":"092225cd-af15-4c89-92d5-d360b9927457","arxiv_id":"2505.01790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning video-language models on educational content helps only some models, while a transcript-only baseline produces more relevant and answerable questions.","lead":"The paper tests whether current vision-language models can write useful quiz questions from educational videos, with and without fine-tuning. It finds that a text-only model still beats video models, that fine-tuning helps only some models, and that better datasets are needed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'transcript-only baseline is safer' claim rests on a tiny, confounded manual evaluation (Section 4.5) whose Mistral outputs often contain answers; it cannot bear the conclusion.","rationale":"The paper's central claim is comparative and practical: for educational video question generation, current VLMs are not yet reliable and transcript-only text models are the safer choice. This claim depends on the evaluation being valid. The automatic metrics are explicitly acknowledged as limited, the ICD metric (Eq. 1) is unvalidated and cannot capture visual questions, and the manual study is the only direct evidence on relevance and answerability. That manual study has three specific problems: small denominators after filtering, a confound from answer-embedded Mistral outputs, and low inter-rater agreement on Bloom levels. These problems are not merely cosmetic; they affect the direction of the headline comparison. At the same time, the paper is honest about several limitations, releases code, and its broad conclusion is plausible in the current literature, so this is not grounds for rejection. The reader's conditional verdict already captures the need for better evaluation; my analysis agrees and sharpens the concrete remedy. Verdict remains CONDITIONAL (UNCHANGED), with the manual evaluation as the explicit condition for acceptance.","tokens_in":15830,"tokens_out":9866,"duration_ms":98429,"concrete_test":"Run a pre-registered manual evaluation on a stratified random sample of at least 50 outputs per condition (or the full generated set), with three independent raters. First tag each output as 'contains explicit answer/answer text' vs 'clean question only'; then judge relevance, answerability, and Bloom level on the interrogative part only. Report per-condition percentages with Wilson 95% confidence intervals and Krippendorff's alpha. Compare Mistral-7B against fine-tuned PG-Video-LLaVA and Video-LLaMA; if the Mistral advantage is not significant after excluding answer-embedded outputs, and if Video-LLaMA's fine-tuning degradation persists with adequate sample size, the Section 5 claims about transcript-baseline superiority and fine-tuning benefits must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main practical claim—that the transcript-only Mistral-7B baseline currently beats multimodal VLMs for educational question generation—is supported almost entirely by the manual evaluation in Section 4.5 (Table 8), and that evidence is too thin and too confounded to bear the claim. Only 180 outputs were rated across ten model/ablation/prompt conditions; after filtering non-questions, per-condition denominators are tiny (Mistral-7B 93.33% answerability is 14/15; zero-shot Video-LLaMA 75% relevance is 3/4). Section 4.4 and Table 7 report that Mistral-7B generated questions 'often include answers' (average output length 148 chars). If an output contains the answer, the 'answerable' label is not measuring a usable assessment question. The automatic metrics cannot arbitrate: Table 4 gives Mistral lower ROUGE/BERTScore than Video-LLaVA, and the authors state that these metrics penalize valid divergent questions. The Bloom-level analysis has Krippendorff alpha 0.43, so Table 8's cognitive-level distribution is unreliable. In addition, the blanket conclusion that fine-tuning improved performance is internally contradicted by Table 8: Video-LLaMA relevance drops from 75% to 23.5% and answerability from 25% to 5.9% after fine-tuning; the authors attribute this to the tiny zero-shot denominator, which is exactly the sample-size problem. Thus the central ranking is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether current vision-language models (VLMs) can generate learning-oriented questions from educational videos. Using a preprocessed subset of LearningQ (TED-Ed and Khan Academy), the authors evaluate zero-shot and fine-tuned versions of Video-LLaVA, PG-Video-LLaVA, and Video-LLaMA against a text-only Mistral-7B transcript baseline, using ROUGE-L, BERTScore, a newly introduced transcript-alignment metric ICD, and a manual evaluation of 180 outputs from six videos. The main conclusions are that zero-shot VLMs struggle, fine-tuning helps overall but leaves content-irrelevance issues, the text-only baseline performs better than multimodal models, and existing datasets lack temporal annotations and video-style diversity.","tokens_in":16099,"tokens_out":4625,"duration_ms":41319,"significance":"If the central ranking holds, the paper provides a useful cautionary result for educational technology: transcript-based LLM pipelines are currently more reliable than video-language models for question generation, and multimodal fine-tuning still lags. The paper also contributes an empirical analysis of the LearningQ dataset, including preprocessing statistics and ablations of visual and audio modalities, plus a new ICD metric. Strengths include the reproducible setup (code link provided), the use of held-out splits, and the explicit acknowledgment of metric limitations. The evidence, however, is concentrated in a small manual evaluation with low inter-rater reliability on the cognitive-level dimension, so the practical conclusion is currently only weakly supported.","major_comments":[{"comment":"The paper's load-bearing claim that the text-only Mistral-7B baseline outperforms the multimodal VLMs rests on a manual evaluation of 180 outputs from only six videos, and after filtering out non-questions the per-condition denominators are very small: Mistral-7B's 93.33% answerability is 14/15, zero-shot Video-LLaMA's 75% relevance is 3/4, and fine-tuned Video-LLaMA's 23.53% relevance is 4/17. No confidence intervals or significance tests are reported, so the ranking in Section 5 is not statistically established. Please report full contingency tables, per-condition N, and an appropriate test, or soften the conclusion to a qualitative observation.","section":"Section 4.5, Table 8"},{"comment":"The manual evaluation labels Mistral-7B outputs as 'answerable' at 93.33%, but Section 4.4 states that Mistral-7B 'generated questions often include answers' (average output length 148 characters). A response that contains the answer is not a usable assessment question even if it is answerable, so this confound biases the comparison in favor of the transcript baseline. The manual protocol should record whether the generated item is a question without an embedded answer, and the answerability score should be conditional on that.","section":"Section 4.4, Tables 7 and 8"},{"comment":"The conclusion that 'fine-tuning improved performance' is contradicted by the manual evaluation for Video-LLaMA: relevance drops from 75% (zero-shot) to 23.53% (fine-tuned) and answerability from 25% to 5.88%. The authors attribute this to the low zero-shot denominator, which is itself the sample-size problem raised in Section 4.5. The claim needs to be restricted to automatic metrics, or supported by a larger manual sample, and the discrepancy acknowledged.","section":"Section 5, Table 8"},{"comment":"The ICD metric is introduced as a new evaluation measure, but it is not validated against human judgments or an existing metric, and the authors note that text-based measures 'may penalize meaningful questions that differ' and cannot capture visually focused questions. Given that ICD is used in Tables 4 and 5 to support model comparisons, a validation study or an explicit caveat about its exploratory status is needed before it can be cited as evidence for the ranking.","section":"Section 3.4, Eq. (1)"},{"comment":"Krippendorff's alpha for the Bloom's taxonomy levels is 0.43, which the authors themselves characterize as low; nevertheless Table 8 reports the post-discussion distribution of cognitive levels as a main result. With unreliability at this level, the Bloom-level analysis cannot support the conclusion that 'the approaches show a similar distribution.' The manual evaluation should be treated as exploratory only, or the number of raters and items should be increased.","section":"Section 4.5"}],"minor_comments":[{"comment":"The prompt iteration rule—changing 'a question' to 'an additional question' in subsequent iterations—is underspecified; it should state how many iterations are run and how duplicates are handled for the baselines.","section":"Section 3.3"},{"comment":"The table header 'Min Video Length - 0:00:23' would be clearer if the unit (HH:MM:SS) and the fact that these are post-filtering statistics were stated in the caption.","section":"Table 1"},{"comment":"Fine-tuning hyperparameters are incomplete: the paper reports batch size 2 and a learning-rate scaling rule, but not the base learning rate, number of epochs, optimizer, or compute used. This makes the fine-tuning experiments hard to reproduce.","section":"Section 4.2"},{"comment":"Table 4 reports Video-LLaVA 'Question' rates of 100.00% and 99.99%; the 99.99% entry should be reconciled with the statement that Video-LLaVA often reformulates the prompt, and the rounding should be consistent.","section":"Section 4.1, Table 4"},{"comment":"There is a capitalization typo in the sentence following the description of Table 8 ('looking at relevance and answerability' should be 'Looking at relevance and answerability').","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's scope is closer to applied educational NLP than to core computer vision; the editor may want to confirm this fits the journal's scope. I do not see a novelty-disclosure concern: the authors cite their prior NLDB paper on LLM question generation, and the present contribution is a distinct multimodal evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper as a useful but fragile empirical snapshot. It compares several open video-language models (Video-LLaVA, PG-Video-LLaVA, Video-LLaMA) against a text-only Mistral-7B baseline for generating questions from educational videos, under zero-shot, fine-tuned, and ablated-modality conditions. The core message — that a transcript-only baseline currently produces more relevant and answerable questions than the VLMs — is plausible but not established by the evidence as reported.\n\nWhat is genuinely new: a head-to-head comparison of these models on LearningQ, including fine-tuning and single-modality ablations. The authors are also honest that ROUGE/BERTScore penalize valid divergent questions, which is why they introduce the ICD metric (question-to-transcript similarity relative to same-domain distractors). That metric is simple and reasonable, and they release code and data. The paper also earns credit for articulating dataset gaps: no timestamped question annotations and low video-style diversity.\n\nNow the soft spots, in proportion. The central ranking rests almost entirely on the manual evaluation in Section 4.5: 180 outputs across ten conditions, with per-condition denominators of 15 for Mistral and 4 for zero-shot Video-LLaMA. That is too thin to support \"the transcript baseline is safer.\" Worse, Table 7 shows Mistral's outputs average 148 characters and \"often include answers,\" which means its 93% answerability score is not measuring a usable assessment question — an answer key is not a question. The fine-tuning conclusion is also internally contradicted: Video-LLaMA's relevance drops from 75% to 23.5% and answerability from 25% to 5.9% after fine-tuning; the authors attribute this to small zero-shot denominators, which is exactly the problem. The automatic metrics do not rescue the ranking: Mistral has lower ROUGE/BERTScore than Video-LLaVA, and the authors themselves say those metrics penalize meaningful divergent questions. There is no variance estimation or significance testing anywhere.\n\nThe Bloom-level annotation has Krippendorff alpha 0.43, so Table 8's cognitive-level distribution is not reliable. The ICD metric is a contribution, but it is an internal similarity measure and is not validated against human judgment; it cannot carry the comparative claim.\n\nWho should read this: people working on educational question generation or ed-tech evaluation who want a current snapshot of open VLM behavior and a clear list of dataset requirements. It deserves a serious referee because the empirical comparison is new and reproducible, but it needs a major revision: larger or pooled manual evaluation, explicit treatment of the \"question includes answer\" confound, significance testing or at least confidence intervals, and a conclusion that distinguishes \"the transcript baseline is competitive\" from \"the transcript baseline beats VLMs.\"\n\nRecommendation: engage with it as a revise-and-resubmit candidate, not as a desk reject. It is worth bringing to a reading group as a case study in evaluation pitfalls.","headline":"A reproducible but under-powered empirical comparison: the claim that a transcript-only baseline beats VLMs for educational video question generation is plausible but not established by the reported evidence.","tokens_in":16642,"tokens_out":2736,"would_cite":true,"duration_ms":26160,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that video-language models, even fine-tuned, generate less relevant and less answerable quiz questions for educational videos than a transcript-only model, and that the fix is better multimodal training data.","keywords":["question generation","vision-language models","educational videos","fine-tuning","multimodal ablation","Bloom's taxonomy","LearningQ dataset","automatic evaluation"],"falsifier":"Build a test set of questions whose answers live only in the visual channel (diagrams, animations, handwriting), generate questions from the same videos with a fine-tuned video model and with the transcript-only baseline, and have blind raters judge relevance and answerability: if the video model matches or beats the transcript baseline, the paper's ranking is wrong. A cheaper check is whether the ICD metric changes the model ordering on a known-visual question set; since ICD reads only transcripts, a shift in ranking would show that the metric is partly why the transcript model appears to win.","tokens_in":15612,"feed_emoji":"🎓","tokens_out":10130,"duration_ms":87079,"temperature":0.7,"pith_summary":"The paper asks whether vision-language models can turn educational videos into learning-oriented quiz questions, and its answer is that today they cannot do it reliably. Across zero-shot and fine-tuned settings on the LearningQ corpus of TED-Ed and Khan Academy videos, the text-only Mistral-7B model reading the speech transcript produced more relevant and more answerable questions than any video model, whether the video reached the model as frames, audio, or both. Fine-tuning improved the video models' output but left a residue of content-irrelevant questions, and the value of each modality swung with video style: handwritten Khan Academy content degraded the visual channel while audio-only inputs held up. The paper's practical message is that transcript-based pipelines are the safe deployment choice, and that progress depends on datasets with time-stamped questions and a wider variety of presentation styles. That matters because automatically generated comprehension questions could turn passive video watching into active learning at scale, and this comparison tells builders which technology is ready.","feed_headline":"For quiz questions, reading the transcript beats watching the video","feed_subtitle":"A study of four models on 180 quiz questions finds video-language models lag a text-only baseline.","key_machinery":"The argument rests on a controlled comparison: one text-only baseline and three video-language models, run under three prompt formulations, in zero-shot and fine-tuned states, with modality ablation (black frames or silent audio) isolating what each channel contributes. The novel evaluative object is the Inner-Class-Diff (ICD) metric, which scores a generated question as the cosine similarity between its embedding and its own video's transcript minus the average similarity to other transcripts in the same subject domain, so a positive score means the question tracks this video rather than generic domain content. Bloom's taxonomy supplies the qualitative ladder for rating cognitive level, alongside binary human judgments of relevance and answerability. The LearningQ corpus provides the training and evaluation questions, and its lack of timestamped questions is the structural gap the conclusion leans on.","core_discovery":"On the paper's own terms, the discovery is a negative result with a concrete ranking: current video-language models are not dependable generators of learning-oriented questions for educational video. In zero-shot use, Video-LLaVA returns well-formed but shallow questions that often rephrase the prompt itself, PG-Video-LLaVA frequently emits statements or empty strings, and Video-LLaMA produces statements instead of questions. Fine-tuning on LearningQ raises question rates, readability, and automatic scores, yet the manual evaluation shows fine-tuned Video-LLaMA dropping to 23.5% relevance and 5.9% answerability while the text-only Mistral-7B baseline stays at 100% and 93.3%. Modality ablations reveal opposite effects for the two video models, with handwriting-heavy Khan Academy content breaking visual processing. The authors conclude that the visual modality currently adds noise rather than signal for this task and that the bottleneck is data: no existing dataset carries temporal annotations linking questions to the video segments that contain the answers.","pith_inferences":["The transcript-first ordering is probably a snapshot rather than a law: as video grounding improves and training data diversifies, the ranking should reverse specifically for visual-heavy content such as diagrams and demonstrations, which can be tested by rerunning this protocol as models improve.","The ICD logic generalizes beyond questions: any video-grounded generation task, such as captions or summaries, could use similarity-to-own-transcript minus similarity-to-same-domain-transcripts as a cheap hallucination screen.","The data suggest a cheap hybrid for deployers: generate questions from the transcript with a text model, and use a lightweight visual detector only to flag diagram or handwriting segments where human-authored questions are needed, instead of end-to-end multimodal generation.","The models' near-universal preference for 'what' questions, and Video-LLaVA's habit of rephrasing the prompt as its output, hint that surface-form bias in training data is part of the bottleneck; a dataset balanced across question words and Bloom levels would test this directly."],"forward_implications":["Teams building automated comprehension checks for video courses should start from speech transcripts and treat the visual channel as an open research problem rather than a deployable input.","Fine-tuning on educational data measurably raises question quality, so the improvement axis is data and model adaptation, not prompt design.","Without time-stamped questions in training sets, models cannot learn where in a video an answer lives, a blocker the paper states in concrete terms.","A single multimodal architecture will not serve all educational formats, since handwriting-heavy videos degrade visual processing and audio-only inputs sometimes match full multimodal input.","The ICD metric offers a ground-truth-free signal for whether a question is about this video rather than any video in the same domain, which future filtering pipelines can exploit."],"supporting_citations":[{"why":"Supplies the LearningQ dataset of TED-Ed and Khan Academy videos with instructor- and learner-authored questions, used for fine-tuning, evaluation, and the qualitative study.","marker":"[9]"},{"why":"The text-only Mistral-7B baseline which, reading transcripts alone, outperforms all multimodal models on relevance and answerability.","marker":"[20]"},{"why":"Video-LLaVA, the zero-shot video model whose short prompt-rephrasing questions inflate word-overlap scores despite low content alignment.","marker":"[29]"},{"why":"PG-Video-LLaVA, the pixel-grounding model fine-tuned on LearningQ, whose visual channel breaks on handwritten Khan Academy content.","marker":"[38]"},{"why":"Video-LLaMA, the audio-visual model fine-tuned here, whose audio-only setting matches its full multimodal performance.","marker":"[54]"},{"why":"ROUGE-L, the longest-common-subsequence metric that compares generated questions to ground-truth questions.","marker":"[30]"},{"why":"BERTScore, the embedding-based similarity metric that complements ROUGE-L in the automatic evaluation.","marker":"[55]"},{"why":"Provides the all-MiniLM-L6-v2 sentence embeddings on which the paper's new ICD content-alignment metric is computed.","marker":"[43]"},{"why":"Bloom's taxonomy, the framework used to rate the cognitive level of generated questions in the manual evaluation.","marker":"[11]"}],"fun_headline_variants":["Video-language models fail to beat text-only for quiz questions","For generating quiz questions, video adds noise, not signal","Text-only model outshines video-language models for learning questions","AI quiz question generation: transcript beats video every time","Video-language models lag text-only baseline on educational video questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes that resemblance to the dataset's own questions and transcripts is a faithful measure of a question's learning value, an assumption the authors themselves concede may penalize meaningful questions phrased differently.","fun_headline_variants_meta":{"raw":{"variants":["Video-language models fail to beat text-only for quiz questions","For generating quiz questions, video adds noise, not signal","Text-only model outshines video-language models for learning questions","AI quiz question generation: transcript beats video every time","Video-language models lag text-only baseline on educational video questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1409,"prompt_tokens":927,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":401}},"tokens_in":543,"tokens_out":482,"duration_ms":5405,"temperature":1.0,"reasoning_tokens":401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:10:14.923057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of questions whose answers live only in the visual channel (diagrams, animations, handwriting), generate questions from the same videos with a fine-tuned video model and with the transcript-only baseline, and have blind raters judge relevance and answerability: if the video model matches or beats the transcript baseline, the paper's ranking is wrong. A cheaper check is whether the ICD metric changes the model ordering on a known-visual question set; since ICD reads only transcripts, a shift in ranking would show that the metric is partly why the transcript model appears to win.","supporting_citations":[{"cited_title":"In: Association for Computational Linguistics, ACL 2004, Barc elona, Spain, July 21-26, 2004","cited_arxiv_id":null,"evidence_quote":"ROUGE-L, the longest-common-subsequence metric that compares generated questions to ground-truth questions."},{"cited_title":"In: International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 202 0","cited_arxiv_id":null,"evidence_quote":"BERTScore, the embedding-based similarity metric that complements ROUGE-L in the automatic evaluation."}],"review_version":1}