Pith. sign in

REVIEW 5 major objections 5 minor 58 references

Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that video-language models, even fine-tuned, generate less relevant and less answerable quiz questions for educational videos than a transcript-only model, and that the fix is better multimodal training data.

desk verdict A reproducible but under-powered empirical comparison: the claim that a transcript-only baseline beats VLMs for educational video question generation is plausible but not established by the reported evidence. read the letter →

arxiv 2505.01790 v1 pith:HPQCWUMC submitted 2025-05-03 cs.CV cs.CLcs.MM

Markos Stamatakis , Joshua Berger , Christian Wartena , Ralph Ewerth , Anett Hoppe This is my paper · ORCID
classification cs.CVcs.CLcs.MM
keywords questiongenerationvision-languagemodelseducationalvideosfine-tuningmultimodalablationBloom'staxonomyLearningQdatasetautomaticevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether vision-language models can turn educational videos into learning-oriented quiz questions, and its answer is that today they cannot do it reliably. Across zero-shot and fine-tuned settings on the LearningQ corpus of TED-Ed and Khan Academy videos, the text-only Mistral-7B model reading the speech transcript produced more relevant and more answerable questions than any video model, whether the video reached the model as frames, audio, or both. Fine-tuning improved the video models' output but left a residue of content-irrelevant questions, and the value of each modality swung with video style: handwritten Khan Academy content degraded the visual channel while audio-only inputs held up. The paper's practical message is that transcript-based pipelines are the safe deployment choice, and that progress depends on datasets with time-stamped questions and a wider variety of presentation styles. That matters because automatically generated comprehension questions could turn passive video watching into active learning at scale, and this comparison tells builders which technology is ready.

What carries the argument

The argument rests on a controlled comparison: one text-only baseline and three video-language models, run under three prompt formulations, in zero-shot and fine-tuned states, with modality ablation (black frames or silent audio) isolating what each channel contributes. The novel evaluative object is the Inner-Class-Diff (ICD) metric, which scores a generated question as the cosine similarity between its embedding and its own video's transcript minus the average similarity to other transcripts in the same subject domain, so a positive score means the question tracks this video rather than generic domain content. Bloom's taxonomy supplies the qualitative ladder for rating cognitive level, alongside binary human judgments of relevance and answerability. The LearningQ corpus provides the training and evaluation questions, and its lack of timestamped questions is the structural gap the conclusion leans on.

What would settle it

Build a test set of questions whose answers live only in the visual channel (diagrams, animations, handwriting), generate questions from the same videos with a fine-tuned video model and with the transcript-only baseline, and have blind raters judge relevance and answerability: if the video model matches or beats the transcript baseline, the paper's ranking is wrong. A cheaper check is whether the ICD metric changes the model ordering on a known-visual question set; since ICD reads only transcripts, a shift in ranking would show that the metric is partly why the transcript model appears to win.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a negative result with a concrete ranking: current video-language models are not dependable generators of learning-oriented questions for educational video. In zero-shot use, Video-LLaVA returns well-formed but shallow questions that often rephrase the prompt itself, PG-Video-LLaVA frequently emits statements or empty strings, and Video-LLaMA produces statements instead of questions. Fine-tuning on LearningQ raises question rates, readability, and automatic scores, yet the manual evaluation shows fine-tuned Video-LLaMA dropping to 23.5% relevance and 5.9% answerability while the text-only Mistral-7B baseline stays at 100% and 93.3%. Modality ablations reveal opposite effects for the two video models, with handwriting-heavy Khan Academy content breaking visual processing. The authors conclude that the visual modality currently adds noise rather than signal for this task and that the bottleneck is data: no existing dataset carries temporal annotations linking questions to the video segments that contain the answers.

Load-bearing premise

The whole comparison assumes that resemblance to the dataset's own questions and transcripts is a faithful measure of a question's learning value, an assumption the authors themselves concede may penalize meaningful questions phrased differently.

Editorial extensions

If this is right

  • Teams building automated comprehension checks for video courses should start from speech transcripts and treat the visual channel as an open research problem rather than a deployable input.
  • Fine-tuning on educational data measurably raises question quality, so the improvement axis is data and model adaptation, not prompt design.
  • Without time-stamped questions in training sets, models cannot learn where in a video an answer lives, a blocker the paper states in concrete terms.
  • A single multimodal architecture will not serve all educational formats, since handwriting-heavy videos degrade visual processing and audio-only inputs sometimes match full multimodal input.
  • The ICD metric offers a ground-truth-free signal for whether a question is about this video rather than any video in the same domain, which future filtering pipelines can exploit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The transcript-first ordering is probably a snapshot rather than a law: as video grounding improves and training data diversifies, the ranking should reverse specifically for visual-heavy content such as diagrams and demonstrations, which can be tested by rerunning this protocol as models improve.
  • The ICD logic generalizes beyond questions: any video-grounded generation task, such as captions or summaries, could use similarity-to-own-transcript minus similarity-to-same-domain-transcripts as a cheap hallucination screen.
  • The data suggest a cheap hybrid for deployers: generate questions from the transcript with a text model, and use a lightweight visual detector only to flag diagram or handwriting segments where human-authored questions are needed, instead of end-to-end multimodal generation.
  • The models' near-universal preference for 'what' questions, and Video-LLaVA's habit of rephrasing the prompt as its output, hint that surface-form bias in training data is part of the bottleneck; a dataset balanced across question words and Bloom levels would test this directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper investigates whether current vision-language models (VLMs) can generate learning-oriented questions from educational videos. Using a preprocessed subset of LearningQ (TED-Ed and Khan Academy), the authors evaluate zero-shot and fine-tuned versions of Video-LLaVA, PG-Video-LLaVA, and Video-LLaMA against a text-only Mistral-7B transcript baseline, using ROUGE-L, BERTScore, a newly introduced transcript-alignment metric ICD, and a manual evaluation of 180 outputs from six videos. The main conclusions are that zero-shot VLMs struggle, fine-tuning helps overall but leaves content-irrelevance issues, the text-only baseline performs better than multimodal models, and existing datasets lack temporal annotations and video-style diversity.

Significance. If the central ranking holds, the paper provides a useful cautionary result for educational technology: transcript-based LLM pipelines are currently more reliable than video-language models for question generation, and multimodal fine-tuning still lags. The paper also contributes an empirical analysis of the LearningQ dataset, including preprocessing statistics and ablations of visual and audio modalities, plus a new ICD metric. Strengths include the reproducible setup (code link provided), the use of held-out splits, and the explicit acknowledgment of metric limitations. The evidence, however, is concentrated in a small manual evaluation with low inter-rater reliability on the cognitive-level dimension, so the practical conclusion is currently only weakly supported.

major comments (5)
  1. [Section 4.5, Table 8] The paper's load-bearing claim that the text-only Mistral-7B baseline outperforms the multimodal VLMs rests on a manual evaluation of 180 outputs from only six videos, and after filtering out non-questions the per-condition denominators are very small: Mistral-7B's 93.33% answerability is 14/15, zero-shot Video-LLaMA's 75% relevance is 3/4, and fine-tuned Video-LLaMA's 23.53% relevance is 4/17. No confidence intervals or significance tests are reported, so the ranking in Section 5 is not statistically established. Please report full contingency tables, per-condition N, and an appropriate test, or soften the conclusion to a qualitative observation.
  2. [Section 4.4, Tables 7 and 8] The manual evaluation labels Mistral-7B outputs as 'answerable' at 93.33%, but Section 4.4 states that Mistral-7B 'generated questions often include answers' (average output length 148 characters). A response that contains the answer is not a usable assessment question even if it is answerable, so this confound biases the comparison in favor of the transcript baseline. The manual protocol should record whether the generated item is a question without an embedded answer, and the answerability score should be conditional on that.
  3. [Section 5, Table 8] The conclusion that 'fine-tuning improved performance' is contradicted by the manual evaluation for Video-LLaMA: relevance drops from 75% (zero-shot) to 23.53% (fine-tuned) and answerability from 25% to 5.88%. The authors attribute this to the low zero-shot denominator, which is itself the sample-size problem raised in Section 4.5. The claim needs to be restricted to automatic metrics, or supported by a larger manual sample, and the discrepancy acknowledged.
  4. [Section 3.4, Eq. (1)] The ICD metric is introduced as a new evaluation measure, but it is not validated against human judgments or an existing metric, and the authors note that text-based measures 'may penalize meaningful questions that differ' and cannot capture visually focused questions. Given that ICD is used in Tables 4 and 5 to support model comparisons, a validation study or an explicit caveat about its exploratory status is needed before it can be cited as evidence for the ranking.
  5. [Section 4.5] Krippendorff's alpha for the Bloom's taxonomy levels is 0.43, which the authors themselves characterize as low; nevertheless Table 8 reports the post-discussion distribution of cognitive levels as a main result. With unreliability at this level, the Bloom-level analysis cannot support the conclusion that 'the approaches show a similar distribution.' The manual evaluation should be treated as exploratory only, or the number of raters and items should be increased.
minor comments (5)
  1. [Section 3.3] The prompt iteration rule—changing 'a question' to 'an additional question' in subsequent iterations—is underspecified; it should state how many iterations are run and how duplicates are handled for the baselines.
  2. [Table 1] The table header 'Min Video Length - 0:00:23' would be clearer if the unit (HH:MM:SS) and the fact that these are post-filtering statistics were stated in the caption.
  3. [Section 4.2] Fine-tuning hyperparameters are incomplete: the paper reports batch size 2 and a learning-rate scaling rule, but not the base learning rate, number of epochs, optimizer, or compute used. This makes the fine-tuning experiments hard to reproduce.
  4. [Section 4.1, Table 4] Table 4 reports Video-LLaVA 'Question' rates of 100.00% and 99.99%; the 99.99% entry should be reconciled with the statement that Video-LLaVA often reformulates the prompt, and the rounding should be consistent.
  5. [Section 4.5] There is a capitalization typo in the sentence following the description of Table 8 ('looking at relevance and answerability' should be 'Looking at relevance and answerability').

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; empirical benchmark with held-out split and non-load-bearing self-citations.

full rationale

This is an empirical benchmark study, not a derivation chain. The central comparisons (zero-shot VLMs, fine-tuned VLMs, and the transcript-only Mistral-7B baseline) are evaluated on a held-out 80/10/10 split of LearningQ with video IDs randomized at seed 1234, so the test questions are not the same items used for fine-tuning. Automatic scores (ROUGE-L, BERTScore) compare generated questions to the ground-truth questions of the same test video; the qualitative ranking is based on a separate manual evaluation of 180 outputs. The new ICD metric (Eq. 1) is a fixed, parameter-free definitional contrast between similarity to the target transcript and average similarity to same-domain distractors; it is not optimized to produce the paper's conclusions, and the main claim about Mistral-7B's relevance/answerability rests on Section 4.5, not on ICD. The only self-citations (refs [3] and [36]) motivate the task or mention earlier LLM question-generation work; they are not load-bearing premises, no uniqueness theorem or ansatz is imported from them, and no fitted parameter is renamed as a prediction. Concerns about tiny per-condition denominators, low Krippendorff alpha for Bloom levels, and Mistral outputs occasionally containing answers are validity and robustness issues, not circularity, since none of the reported scores reduces to an input by construction. Accordingly, no circular step can be exhibited.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The paper's central claims rest mainly on dataset and metric assumptions rather than on invented entities or theoretical free parameters. The fine-tuned model weights are fitted to the LearningQ training split, and evaluation shares the same benchmark distribution. The ICD metric assumes transcript embeddings represent video content, a limitation the authors acknowledge but do not resolve.

free parameters (1)
  • Fine-tuned VLM weights (PG-Video-LLaVA-13B, Video-LLaMA-13B) = Not specified; trained on LearningQ train split with batch size 2 and learning rate adjusted by linear scaling rule.
    The conclusion that fine-tuning improves question generation rests on these fitted weights, but no checkpoints or complete training schedule are provided.
assumptions (5)
  • domain assumption LearningQ ground-truth questions are valid educational targets
    Training and evaluation use LearningQ's instructor and learner questions; if these are not representative or contain quality issues, all comparisons inherit that bias. The authors filter questions but do not independently validate the remaining set (Section 3.1).
  • domain assumption Transcripts faithfully represent video learning content
    Equation 1 and the text-based metrics compare question embeddings to transcript embeddings, so content shown only visually, such as handwriting or diagrams, is invisible. The authors acknowledge this limitation in Section 3.4 but still use ICD in quantitative comparisons.
  • domain assumption Automatic lexical and semantic metrics reflect question quality
    Tables 4 and 5 rank models by ROUGE-L, BERTScore, and ICD. The authors state in Section 3.4 that these metrics may penalize meaningful questions that differ from ground truth, making this assumption load-bearing for the quantitative conclusions.
  • domain assumption Random split by video ID prevents train and test leakage
    Fine-tuning and evaluation share the LearningQ distribution. The 80/10/10 split with seed 1234 in Section 3.1 is assumed to isolate the test videos, but no duplicate-video or near-duplicate check is reported.
  • domain assumption Manual judgments are reliable despite low Bloom-level agreement
    Two authors coded 180 outputs. Krippendorff alpha was 0.71 and 0.79 for relevance and answerability but only 0.43 for Bloom levels, so conclusions about understanding levels rest on weakly agreed labels that were later resolved by discussion.
invented entities (1)
  • None
    purpose: No new physical, conceptual, or external entities are introduced.
    The only new artifact is the ICD metric, which is a scoring formula, not a postulated entity. No new particle, force, dimension, or conserved quantity appears.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos." pith.science (2026). https://pith.science/paper/HPQCWUMC

@misc{pith2026250501790,
  author       = {Pith},
  title        = {Pith review of: Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPQCWUMC}},
  note         = {Machine review of arXiv:2505.01790}
}
read the original abstract

Web-based educational videos offer flexible learning opportunities and are becoming increasingly popular. However, improving user engagement and knowledge retention remains a challenge. Automatically generated questions can activate learners and support their knowledge acquisition. Further, they can help teachers and learners assess their understanding. While large language and vision-language models have been employed in various tasks, their application to question generation for educational videos remains underexplored. In this paper, we investigate the capabilities of current vision-language models for generating learning-oriented questions for educational video content. We assess (1) out-of-the-box models' performance; (2) fine-tuning effects on content-specific question generation; (3) the impact of different video modalities on question quality; and (4) in a qualitative study, question relevance, answerability, and difficulty levels of generated questions. Our findings delineate the capabilities of current vision-language models, highlighting the need for fine-tuning and addressing challenges in question diversity and relevance. We identify requirements for future multimodal datasets and outline promising research directions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 23 canonical work pages

  1. [1]

    Computers and Education: Ar tificial Intelligence 7, 100298 (2024)

    Al Faraby, S., Romadhony, A., Adiwijaya: Analysis of llms for educational question classification and generation. Computers and Education: Ar tificial Intelligence 7, 100298 (2024). https://doi.org/10.1016/j.caeai.2024.100298

    arXiv 2024

  2. [2]

    Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y. , Zhu, W., Marathe, K., Bitton, Y., Gadre, S.Y., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P.W., Ilharco, G., Wortsman, M., Schmidt, L.: Openflamingo: An open-source fra mework for train- ing large autoregressive vision-language models. arXiv pr eprint abs/2308.01390 (2023). https://doi.org/10.48...

    arXiv doi:10.48550/arxiv.2308.01390 2023

  3. [3]

    , Wartena, C.: Question generation capabilities of “small“ large lang uage models

    Berger, J., Koß, J., Stamatakis, M., Hoppe, A., Ewerth, R. , Wartena, C.: Question generation capabilities of “small“ large lang uage models. In: International Conference on Natural Language & Informatio n Systems, NLDB 2024, Turin, Italy, June 25-27, 2024. pp. 183–194. Spri nger (2024). https://doi.org/10.1007/978-3-031-70242-6_18

    work page doi:10.1007/978-3-031-70242-6_18 2024

  4. [4]

    In: Conference on Computing Communication and Networking Technologies, ICCCNT 2023, Delhi, India, July 6-8, 2023

    Bhanot, G., Pal, B., Sah, B., Chhikara, G.: Generating rel evant question an- swer from video. In: Conference on Computing Communication and Networking Technologies, ICCCNT 2023, Delhi, India, July 6-8, 2023. pp . 1–6. IEEE (2023). https://doi.org/10.1109/ICCCNT56998.2023.10306677

    arXiv 2023

  5. [5]

    In: Conference on User Modeling, Adaptation and Personalization, UMAP 2024, Cagl iari, Italy, July 1-4,

    Biancini, G., Ferrato, A., Limongelli, C.: Multiple-cho ice question generation using large language models: Methodology and educator insights. In: Conference on User Modeling, Adaptation and Personalization, UMAP 2024, Cagl iari, Italy, July 1-4,

    work page 2024

  6. [6]

    O’Reilly Media, In c

    Bird, S., Klein, E., Loper, E.: Natural language processi ng with Python: analyzing text with the natural language toolkit. " O’Reilly Media, In c." (2009) 14 M. Stamatakis et al

    work page 2009

  7. [7]

    Cao, S., Wang, L.: Controllable open-ended question gene ration with A new ques- tion type ontology. In: Annual Meeting of the Association fo r Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virt ual Event, Au- gust 1-6, 2021. pp. 6424–6439. Association for Co...

    doi:10.18653/v1/2021.acl-long.502 2021

  8. [8]

    arXiv preprint abs/2310.14735 (2023)

    Chen, B., Zhang, Z., Langrené, N., Zhu, S.: Unleashing the potential of prompt engineering in large language models: a comprehensive revi ew. arXiv preprint abs/2310.14735 (2023). https://doi.org/10.48550/ARXIV.2310.14735

    arXiv doi:10.48550/arxiv.2310.14735 2023

Show all 58 references
  1. [9]

    In: International Confe rence on Web and Social Media, ICWSM 2018, Stanford, (California), USA, June 25-28 , 2018

    Chen, G., Yang, J., Hauff, C., Houben, G.: Learningq: A larg e-scale dataset for educational question generation. In: International Confe rence on Web and Social Media, ICWSM 2018, Stanford, (California), USA, June 25-28 , 2018. pp. 481–490. AAAI Press (2018). https://doi.org/1...

    2018 doi

  2. [10]

    I n: Language Resources and Evaluation Conference, LREC 2020, Marseille, France, M ay 11-16, 2020

    Colas, A.M., Kim, S., Dernoncourt, F., Gupte, S., Wang, D .Z., Kim, D.S.: Tuto- rialvqa: Question answering dataset for tutorial videos. I n: Language Resources and Evaluation Conference, LREC 2020, Marseille, France, M ay 11-16, 2020. pp. 5450–5455 (2020), https://aclantholog...

    2020

  3. [11]

    Educ ational Horizons 83(3), 154–159 (2005), http://www.jstor.org/stable/42926529

    Conklin, J.: A taxonomy for learning, teaching, and asse ssing: A revision of bloom’s taxonomy of educational objectives complete edition. Educ ational Horizons 83(3), 154–159 (2005), http://www.jstor.org/stable/42926529

    2005

  4. [12]

    Elkins, S., Kochmar, E., Serban, I., Cheung, J.C.K.: How useful are educational questions generated by large language models? In: Artificia l Intelligence in Edu- cation, AIED 2023, Tokyo, Japan, July 3-7, 2023. pp. 536–542 . Springer (2023). https://doi.org/10.1007/978-3-031-3...

    2023 doi

  5. [13]

    Journal of appl ied psychology 32(3), 221 (1948)

    Flesch, R.: A new readability yardstick. Journal of appl ied psychology 32(3), 221 (1948). https://doi.org/10.1037/h0057532

    1948 doi

  6. [14]

    International Journal o f Advanced Corporate Learning 16(1), 19–27 (2023)

    Forkan, A.R.M., Kang, Y., Jayaraman, P.P., Du, H., Thoms on, S., Kollias, E., Wieland, N.: Videodl: Video-based digital learning framew ork using AI question generation and answer assessment. International Journal o f Advanced Corporate Learning 16(1), 19–27 (2023). https://do...

    2023 doi

  7. [15]

    arXiv preprint abs/1706.02677 (2017), http://arxiv.org/abs/1706.02677

    Goyal, P., Dollár, P., Girshick, R.B., Noordhuis, P., We solowski, L., Ky- rola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minib atch SGD: training imagenet in 1 hour. arXiv preprint abs/1706.02677 (2017), http://arxiv.org/abs/1706.02677

    2017 arXiv

  8. [16]

    In: ACM/SIGAPP Sym- posium on Applied Computing, SAC 2019, Limassol, Cyprus, Ap ril 8-12, 2019

    da Graça Campos Pimentel, M., Yaguinuma, C.A., Martins, D.S., Zaine, I.: An- choring interactive points of interest on web-based instru ctional video: effects on students’ interaction behavior and perceived experience. In: ACM/SIGAPP Sym- posium on Applied Computing, SAC 2019, ...

    2019

  9. [17]

    Scientific Dat a 10(1), 158 (2023)

    Gupta, D., Attal, K., Demner-Fushman, D.: A dataset for m edical instructional video classification and question answering. Scientific Dat a 10(1), 158 (2023). https://doi.org/10.1038/s41597-023-02036-y

    2023 doi

  10. [18]

    IEEE Access 11, 20885– 20896 (2023)

    Hadifar, A., Bitew, S.K., Deleu, J., Develder, C., Demee ster, T.: Eduqg: A multi- format multiple-choice dataset for the educational domain . IEEE Access 11, 20885– 20896 (2023). https://doi.org/10.1109/ACCESS.2023.3248790

    2023

  11. [19]

    In: Artificial Intelligence in Education, AIED 2024, Recife , Brazil, July 8-12, 2024

    Hwang, K., Wang, K., Alomair, M., Choa, F., Chen, L.K.: To wards automated mul- tiple choice question generation and evaluation: Aligning with bloom’s taxonomy. In: Artificial Intelligence in Education, AIED 2024, Recife , Brazil, July 8-12, 2024. pp. 389–396. Springer (2024). ...

    2024 doi

  12. [20]

    , Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saul nier, L., Lavaud, L.R., Lachaux, M., Stock, P., Scao, T.L., Lavril, T. , Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b. arXiv preprint abs/2310...

    2023 doi arXiv

  13. [21]

    In: IEEE Conference on Computer Vis ion and Pattern Recognition, CVPR 2017, Honolulu, (Hawaii), USA, July 21-2 6, 2017

    Kembhavi, A., Seo, M.J., Schwenk, D., Choi, J., Farhadi, A., Hajishirzi, H.: Are you smarter than a sixth grader? textbook question answerin g for multimodal machine comprehension. In: IEEE Conference on Computer Vis ion and Pattern Recognition, CVPR 2017, Honolulu, (Hawaii), ...

    2017

  14. [22]

    In: Worksh op on Statistical Ma- chine Translation co-locared with the ACL Conference, WMT@ ACL 2007, Prague, Czech Republic, June 23, 2007

    Lavie, A., Agarwal, A.: METEOR: an automatic metric for M T evaluation with high levels of correlation with human judgments. In: Worksh op on Statistical Ma- chine Translation co-locared with the ACL Conference, WMT@ ACL 2007, Prague, Czech Republic, June 23, 2007. pp. 228–231...

    2007

  15. [23]

    In: Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels, Belgium, Oct ober 31 - Novem- ber 4, 2018

    Lei, J., Yu, L., Bansal, M., Berg, T.L.: TVQA: localized, compositional video question answering. In: Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels, Belgium, Oct ober 31 - Novem- ber 4, 2018. pp. 1369–1379. Association for C...

    2018 doi

  16. [24]

    In: Association for Computation al Linguistics, ACL 2020, Virtual Event, July 5-10, 2020

    Lei, J., Yu, L., Berg, T.L., Bansal, M.: TVQA+: spatio-te mporal grounding for video question answering. In: Association for Computation al Linguistics, ACL 2020, Virtual Event, July 5-10, 2020. pp. 8211–8225. Associ ation for Computa- tional Linguistics (2020). https://doi.or...

    2020 doi

  17. [25]

    I n: HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 20 20

    Leisner, D., Zahn, C.G., Ruf, A., Cattaneo, A.A.P.: Diffe rent ways of interacting with videos during learning in secondary physics lessons. I n: HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 20 20. pp. 284–291. Springer (2020). https://doi.org/10.10...

    2020 doi

  18. [26]

    In: International Conference on Machine Learning, ICM L 2023, Hon- olulu, (Hawaii), USA, July 23-29 2023

    Li, J., Li, D., Savarese, S., Hoi, S.C.H.: BLIP-2: bootst rapping language- image pre-training with frozen image encoders and large lan guage mod- els. In: International Conference on Machine Learning, ICM L 2023, Hon- olulu, (Hawaii), USA, July 23-29 2023. pp. 19730–19742. PM...

    2023

  19. [27]

    , Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y. , Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. arXiv pr eprint abs/2305.06355 (2023). https://doi.org/10.48550/ARXIV.2305.06355

    2023 doi arXiv

  20. [28]

    https://doi.org/10.48550/ARXIV.2501.05220

    Li, Z., Cukurova, M., Bulathwela, S.: A novel approach to scalable and automatic topic-controlled question generation in edu cation (2025). https://doi.org/10.48550/ARXIV.2501.05220

    2025 doi arXiv

  21. [29]

    arXiv preprint abs/2311.10122 (2023)

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L .: Video-llava: Learn- ing united visual representation by alignment before proje ction. arXiv preprint abs/2311.10122 (2023). https://doi.org/10.48550/ARXIV.2311.10122

    2023 doi arXiv

  22. [30]

    In: Association for Computational Linguistics, ACL 2004, Barc elona, Spain, July 21-26, 2004

    Lin, C.Y.: ROUGE: A package for automatic evaluation of s ummaries. In: Association for Computational Linguistics, ACL 2004, Barc elona, Spain, July 21-26, 2004. pp. 74–81. Association for Computational Ling uistics (2004), https://aclanthology.org/W04-1013

    2004

  23. [31]

    arXiv preprint abs/2310.03744 (2023)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines wit h vi- sual instruction tuning. arXiv preprint abs/2310.03744 (2023). https://doi.org/10.48550/ARXIV.2310.03744 16 M. Stamatakis et al

    2023 doi arXiv

  24. [32]

    In: Advances in Neural Information Processing Systems, NeurIPS 2023, New O rleans, (LA), USA, December 10 - 16, 2023

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tun ing. In: Advances in Neural Information Processing Systems, NeurIPS 2023, New O rleans, (LA), USA, December 10 - 16, 2023. vol. 36, pp. 34892–34916. Curran Asso ciates, Inc. (2023), https://proceedings.neurips.cc/paper...

    2023

  25. [33]

    arXiv preprint abs/2306.07207 (2023)

    Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang , T., Wei, Z.: Val- ley: Video assistant with large language model enhanced abi lity. arXiv preprint abs/2306.07207 (2023). https://doi.org/10.48550/ARXIV.2306.07207

    2023 doi arXiv

  26. [34]

    arXiv preprint abs/2306.05424 (2023)

    Maaz, M., Rasheed, H.A., Khan, S.H., Khan, F.S.: Video-c hatgpt: Towards de- tailed video understanding via large vision and language mo dels. arXiv preprint abs/2306.05424 (2023). https://doi.org/10.48550/ARXIV.2306.05424

    2023 doi arXiv

  27. [35]

    https://doi.org/10.1016/j.caeai.2025.100370

    Maity, S., Deroy, A., Sarkar, S.: Can large language mode ls meet the challenge of generating school-level questions? Computers and Educati on: Artificial Intelligence 8, 100370 (2025). https://doi.org/10.1016/j.caeai.2025.100370

    2025

  28. [36]

    https://doi.org/10.1016/j.compedu.2021.104355

    Merkt, M., Hoppe, A., Bruns, G., Ewerth, R., Huff, M.: Push ing the button: Why do learners pause online videos? Computers & Education 176, 104355 (2022). https://doi.org/10.1016/j.compedu.2021.104355

    2022

  29. [37]

    , de Souza, A.F., Oliveira- Santos, T.: Automatic multiple-choice question generatio n and evaluation sys- tems based on LLM: A study case with university resolutions

    Mucciaccia, S.S., Paixão, T.M., Mutz, F.W., Badue, C.S. , de Souza, A.F., Oliveira- Santos, T.: Automatic multiple-choice question generatio n and evaluation sys- tems based on LLM: A study case with university resolutions. In: Confer- ence on Computational Linguistics, COLING...

    2025

  30. [38]

    arXiv preprint abs/2311.13435 (2023)

    Munasinghe, S., Thushara, R., Maaz, M., Rasheed, H.A., K han, S., Shah, M., Khan, F.S.: Pg-video-llava: Pixel grounding l arge video-language models. arXiv preprint abs/2311.13435 (2023). https://doi.org/10.48550/ARXIV.2311.13435

    2023 doi arXiv

  31. [39]

    arXiv preprint abs/2303.08774 (2023)

    OpenAI: GPT-4 technical report. arXiv preprint abs/2303.08774 (2023). https://doi.org/10.48550/ARXIV.2303.08774

    2023 doi arXiv

  32. [40]

    Education and Information Technologies 24(1), 805–823 (2019)

    Palaigeorgiou, G., Papadopoulou, A.: Promoting self-p aced learning in the el- ementary classroom with interactive video, an online cours e platform and tablets. Education and Information Technologies 24(1), 805–823 (2019). https://doi.org/10.1007/S10639-018-9804-5

    2019 doi

  33. [41]

    In: Association for Com putational Linguistics, ACL 2002, Philadelphia, (PA), USA, July 6-12, 2002

    Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a metho d for automatic evaluation of machine translation. In: Association for Com putational Linguistics, ACL 2002, Philadelphia, (PA), USA, July 6-12, 2002. pp. 311– 318. ACL (2002). https://doi.org/10.3115/1073083.1073135

    2002

  34. [42]

    In: Conference on E mpirical Methods in Natural Language Processing, EMNLP 2016, Austin , Texas, USA

    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ ques- tions for machine comprehension of text. In: Conference on E mpirical Methods in Natural Language Processing, EMNLP 2016, Austin , Texas, USA. pp. 2383–2392. The Association for Computational Ling uistics ...

    2016 doi

  35. [43]

    In: Conference on Empirical Methods in Natural La nguage Processing, EMNLP 2019, Hong Kong, China, November 3-7, 2019

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence emb eddings using siamese bert- networks. In: Conference on Empirical Methods in Natural La nguage Processing, EMNLP 2019, Hong Kong, China, November 3-7, 2019. pp. 3980–3 990. Association for Computational Linguistics (2019)....

    2019 doi

  36. [44]

    ACM Computing Surveys 55(2), 26:1–26:39 (2023)

    Sai, A.B., Mohankumar, A.K., Khapra, M.M.: A survey of ev aluation met- rics used for NLG systems. ACM Computing Surveys 55(2), 26:1–26:39 (2023). https://doi.org/10.1145/3485766

    2023 doi

  37. [45]

    In: Artificial Intelligenc e in Education, Using VLMs to Generate Questions for Educational Videos 17 AIED 2024, Recife, Brazil, July 8-12, 2024

    Scaria, N., Chenna, S.D., Subramani, D.N.: Automated ed ucational ques- tion generation at different bloom’s skill levels using larg e language mod- els: Strategies and evaluation. In: Artificial Intelligenc e in Education, Using VLMs to Generate Questions for Educational Videos...

    2024 doi

  38. [46]

    Artificial Intel ligence Review 57(10), 271 (2024)

    Shahin, N., Ismail, L.: From rule-based models to deep le arning transformers archi- tectures for natural language processing and sign language translation systems: sur- vey, taxonomy and performance evaluation. Artificial Intel ligence Review 57(10), 271 (2024). https://doi.or...

    2024 doi

  39. [47]

    In: Ar tificial Intelligence in Ed- ucation, AIED 2023, Tokyo, Japan, July 3-7, 2023, Proceedin gs

    Shimmei, M., Bier, N.L., Matsuda, N.: Machine-generate d questions attract in- structors when acquainted with learning objectives. In: Ar tificial Intelligence in Ed- ucation, AIED 2023, Tokyo, Japan, July 3-7, 2023, Proceedin gs. pp. 3–15. Springer (2023). https://doi.org/10.1...

    2023 doi

  40. [48]

    In: Annual Confere nce on Infor- mation Technology Education, SIGITE 2017, New York, USA, Oc tober 4 - 7, 2017

    Shoufan, A., Mohamed, F.: On the likes and dislikes of you tube’s ed- ucational videos: A quantitative study. In: Annual Confere nce on Infor- mation Technology Education, SIGITE 2017, New York, USA, Oc tober 4 - 7, 2017. p. 127–132. Association for Computing Machinery (2017)....

    2017

  41. [49]

    : Automatic question- naire and interactive session generation from videos

    Skanda, V.C., Jayaram, R., Bukitagar, V.C., Kumar, N.S. : Automatic question- naire and interactive session generation from videos. In: C omputational Intelligence in Data Science, ICCIDS 2020, Chennai, India, February 20-2 2, 2020. pp. 205–212. Springer (2020). https://doi.or...

    2020 doi

  42. [50]

    In: ACM on Multimedia Conference, MM 2017, Mountain View, (CA), USA , October 23-27,

    Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., Zhuan g, Y.: Video question answering via gradually refined attention over appearance a nd motion. In: ACM on Multimedia Conference, MM 2017, Mountain View, (CA), USA , October 23-27,

    2017

  43. [51]

    In: Màrquez, L., Callison-Burch, C., Su, J., Pig hin, D., Marton, Y

    Yang, Y., Yih, W., Meek, C.: Wikiqa: A challenge dataset f or open-domain question answering. In: Màrquez, L., Callison-Burch, C., Su, J., Pig hin, D., Marton, Y. (eds.) Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-...

    2015 doi

  44. [52]

    arXiv prep rint abs/2410.13191 (2024)

    Yao, Z., Parashar, A., Zhou, H., Jang, W.S., Ouyang, F., Y ang, Z., Yu, H.: Mcqg- srefine: Multiple choice question generation and evaluatio n with iterative self- critique, correction, and comparison feedback. arXiv prep rint abs/2410.13191 (2024). https://doi.org/10.48550/ARX...

    2024 doi arXiv

  45. [53]

    : Activitynet-qa: A dataset for understanding complex web videos via question answering

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D. : Activitynet-qa: A dataset for understanding complex web videos via question answering. In: The Thirty-Third AAAI Conference on Artificial Intelligence, A AAI 2019, The Thirty- First Innovative Applications of Artifi...

    2019

  46. [54]

    In: Conference on Empi rical Methods in Nat- ural Language Processing - System Demonstrations, EMNLP 20 23, Singapore, De- cember 6-10, 2023

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction -tuned audio-visual lan- guage model for video understanding. In: Conference on Empi rical Methods in Nat- ural Language Processing - System Demonstrations, EMNLP 20 23, Singapore, De- cember 6-10, 2023. pp. 543–553. Ass...

    2023 doi

  47. [55]

    In: International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 202 0

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with BERT. In: International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 202 0. OpenReview.net (2020), https://openreview.net/forum?i...

    2020

  48. [2017]

    1645–1653

    pp. 1645–1653. ACM (2017). https://doi.org/10.1145/3123266.3123427

    2017

  49. [2024]

    https://doi.org/10.1145/3631700.3665233

    ACM (2024). https://doi.org/10.1145/3631700.3665233

    2024

  50. [5384]

    https://doi.org/10.1109/CVPR.2017.571

    IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.571

    2017 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.