Pith. sign in

REVIEW 3 major objections 6 minor 60 references

YouLeQD: Decoding the Cognitive Complexity of Questions and Engagement in Online Educational Videos from Learners' Perspectives

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new dataset of 57,242 learner questions from YouTube lecture comments, labeled by Bloom's Taxonomy, shows most questions sit at the lowest cognitive level, Knowledge, while harder questions are less liked but more likely to draw replies.

desk verdict YouLeQD is a useful new dataset, but the paper's central claim about knowledge-level dominance outruns the classifier's measured recall. read the letter →

arxiv 2501.11712 v1 pith:7W3COBMV submitted 2025-01-20 cs.CL

Nong Ming , Sachin Sharma , Jiho Noh This is my paper · ORCID
classification cs.CL
keywords Bloom'sTaxonomyquestionclassificationlearnerquestionsYouTubeeducationalvideosRoBERTaknowledgedistillationstudentengagementcognitivecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces YouLeQD, a dataset of 57,242 learner-posed questions extracted from comments on 1,762 YouTube lecture videos across five STEM subjects. It trains two RoBERTa-based classifiers: one detects which comments are questions, and one assigns each question a Bloom's Taxonomy cognitive level. The paper's central finding is that most learner questions are classified as Knowledge, the lowest level, with higher levels such as Analysis and Synthesis comparatively rare. It also reports an inverse relationship between question popularity (likes normalized by views) and cognitive level, alongside a positive correlation between interaction rate (replies) and higher levels from Analysis to Synthesis. If these findings hold, they suggest that online learners mostly ask factual recall questions, and that cognitively demanding questions are what drive discussion rather than endorsement.

What carries the argument

The machinery is a two-stage classification pipeline. A fine-tuned RoBERTa model, trained with knowledge distillation from GPT-4o soft labels on an existing interrogative-sentence dataset, detects question sentences in comments. A second RoBERTa model, fine-tuned on the DASQBT question dataset (a small multi-class question dataset labeled by Bloom's levels) and augmented by GPT-4o-generated questions, assigns each question to Bloom's six levels plus an 'Irrelevant' class, with out-of-distribution detection based on low softmax probabilities. The Bloom's classifier then produces the distributional and engagement results across all 57,242 questions.

What would settle it

Take a stratified random sample of roughly 1,000 to 2,000 extracted questions, have three or more independent raters assign Bloom's levels, then recompute the level distribution and the likes-per-view and replies-per-view correlations on the human labels; if the ordering or sign of the correlations differs materially, the central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that large-scale learner questioning on YouTube is dominated by low-cognitive questions: Knowledge accounts for about 44 percent to 48 percent of questions across subjects, whereas higher-level categories (Analysis, Evaluation, Synthesis) together range from roughly 14.6 percent to 27.6 percent. The paper further claims that popularity, measured by likes per view, falls as Bloom's level rises, while interaction rate, measured by replies per view, increases for the higher levels. These patterns are presented as evidence about how learners actually engage with educational videos, and as a resource for aligning AI question-generation systems with genuine learner behavior.

Load-bearing premise

The central claim depends on the Bloom classifier's labels being accurate across all 57,242 questions; the human checkpoint covers only 300 questions, and per-class recall is very low for some categories, so the reported distribution and engagement correlations could change if model errors were corrected.

Editorial extensions

If this is right

  • If the distributional findings are representative, AI question-generation systems should be calibrated to the reality that learners mostly ask recall-level questions, and should aim to prompt deeper questions rather than assume they occur naturally.
  • The dataset provides a public corpus of naturally occurring learner questions with Bloom labels, enabling comparisons across biology, chemistry, math, physics, and computer science.
  • The engagement result implies that reply-based discussion is more likely to be stimulated by higher-order questions, whereas like-based popularity tracks simpler content, which matters for how instructors and platforms surface comments.
  • The classification method demonstrates a workflow for using LLM-generated labels and knowledge distillation to annotate noisy social-media text at scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's headline percentages are only as trustworthy as the classifier's per-class recall, which human evaluation shows is low for Comprehension and Analysis; correcting the dataset-wide distribution with human labels could shift the reported ranking of cognitive levels.
  • If the engagement pattern is causal rather than merely correlational, instructional designers could use cognitive-level labels to identify high-value discussion threads and route them to instructors or answer engines.
  • A testable extension the paper does not pursue would be to compare the same questions' Bloom levels as assigned by multiple human raters versus the model; the current 300-sample evaluation with majority-vote labels does not measure rater disagreement.
  • Because the 'Irrelevant' class dominates human labels (214 of 300 samples), many YouTube comments classified as questions are not genuine learning questions, so educational AI built on this data should filter or reweight that class.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces YouLeQD, a dataset of 57,242 learner-posed questions extracted from comments on 1,762 YouTube educational lecture videos across five STEM subjects. It also trains two RoBERTa-based classifiers: one detects interrogative sentences in comments (via knowledge distillation from GPT-4o), and one assigns Bloom's Taxonomy (BT) cognitive levels to the extracted questions, using GPT-4o-generated augmentation and an added 'Irrelevant' class. The authors report that most learner questions fall under the lowest BT level, Knowledge (44–48% across subjects), that higher BT levels correlate positively with interaction rate (replies) but negatively with popularity (likes), and that subject-level distributions differ. The dataset and code are publicly released. The central empirical claims rest on applying the BT classifier to the full corpus, with a human evaluation of only 300 examples used as an external check.

Significance. If the distributional and engagement findings are correct, YouLeQD would be a useful public resource for studying learner questioning behavior in informal online education, and the paper's pipeline (LLM-assisted augmentation, knowledge distillation, OOD-aware classification) could inform similar annotation efforts. The strengths are the scale of the released dataset, the public code, the use of an independent human-labeled test set (albeit small), and the explicit attempt to handle out-of-distribution questions. However, the headline conclusions are currently only as strong as the classifier's predictions, and the human evaluation shows substantial per-class error, especially for Comprehension and Analysis. Because the paper does not correct for this measurement error, the reported BT distribution and engagement correlations must be treated as provisional.

major comments (3)
  1. [§IV-B, §IV-C, Fig. 2] The headline result that 44–48% of learner questions are Knowledge-level is computed from RoBERTa predictions on the full 57,242 questions, but the only human check, Table VI, shows per-class recall of 0.360 for Knowledge, 0.148 for Comprehension, and 0.176 for Analysis. The relevant-class supports are also very small (7 for Application, 3 for Synthesis). The paper does not use the human confusion matrix to re-estimate the dataset-level BT distribution or report confidence intervals for Figure 2. Because the model systematically undercounts middle-level classes, the reported Knowledge share could be inflated by misclassified Comprehension and Analysis questions. Please add a confusion-matrix-based correction (or equivalent re-weighting) and uncertainty bounds, and show how the Section V conclusion changes under plausible label-error scenarios.
  2. [§III-C.2, §IV-C, Table VI] The construction of the Irrelevant class is circular: the top-500 lowest-confidence predictions from the first-stage model are labeled Irrelevant and added to the training set, and the same model family is later used to label the full corpus from which the distributional claims are drawn. Table VI complicates interpretation further: human annotators labeled 214 of 300 (71.3%) extracted questions as Irrelevant, yet Figure 2 reports six BT-level percentages with no Irrelevant category mentioned. The paper must state explicitly whether Figure 2 excludes Irrelevant predictions. If it excludes them, the claim that 'most questions fall under the lowest cognitive level' is conditional on the non-irrelevant subset and should be reframed; if it includes them, the figure is inconsistent with the human evaluation. In either case, the self-training construction should be validated with human labels rather than the model's own confidence scores.
  3. [§IV-E, Fig. 3] The engagement analysis in Figure 3 is based on the same noisy model labels and reports no statistical tests, confidence intervals, or effect sizes. With per-class recall as low as 0.148 (Comprehension), the inverse popularity/cognitive-level relationship and the positive interaction-rate correlation from Analysis to Synthesis could be artifacts of label noise. Please report the correlation estimates with uncertainty (for example, via bootstrap or using a bias-corrected label distribution), state the effective sample size per BT level after grouping by video, and clarify exactly how the rates are normalized by view counts.
minor comments (6)
  1. [Table II] The table title contains a typo: 'YoulQD' should be 'YouLeQD'.
  2. [§IV-A] The question-detection model is reported as achieving 99.42% F1, but the test-set size and split details are not given; please specify them.
  3. [Eq. (1)] The knowledge-distillation loss is not fully defined: the argument of the exponential and the summation indices over k should be made explicit, and the choices of τ = 2 and α = 2.5 should be justified relative to the cited reference.
  4. [§II-B] The sentence 'Table I lists the cognitive levels of Bloom’s taxonomy and provides examples illustrative examples of questions' contains a duplicated phrase; it should read 'illustrative examples'.
  5. [Table III] The reported GPT-4o agreement of 0.988 on DASQBT does not state how many examples were evaluated; please add the sample size and any filtering criteria.
  6. [Figure 1] The caption says 'Confusion Matrix of Human Annotators and Model Prediction', but only panel (a) shows human annotator agreement; panel (b) compares human labels with model predictions. The caption should describe each panel separately.

Circularity Check

1 steps flagged · score 4.0 of 10

The reported BT distribution and engagement correlations are partly circular because the 'Irrelevant' class used to train the final classifier is pseudo-labeled from the same model's lowest-confidence predictions on the same YouTube questions, so the model's uncertainty is fed back as a 'discovery'.

  1. self definitional [Section III-C2, 'Detecting Out-of-Distribution Examples and Evaluation Strategy']
    "We identified the top 500 examples with low maximum/predicted class probability from the softmax distributions and labeled them as belonging to the 'Irrelevant' class. In the following stage, we trained the same type of model on the expanded dataset, which includes additional question samples classified as 'Irrelevant,' and evaluated its performance on the reserved test dataset."

    The 'Irrelevant' labels are not independent annotations; they are produced by the first BT model's low-confidence predictions on the very corpus that the final model is later used to analyze. After retraining with these self-generated labels, the final model classifies all 57,242 extracted questions and those predictions drive the central findings: the BT-level distribution in Section IV-C/Fig. 2 and the popularity/interaction correlations in Section IV-E/Fig. 3. Consequently, part of the reported distribution is a re-description of the initial model's confidence scores rather than a measurement of learner questions.

full rationale

The central empirical claims (Knowledge-dominant questions; inverse popularity vs. BT level; positive interaction correlation for higher levels) are derived entirely from RoBERTa predictions on 57,242 questions. One training ingredient is circular: the 'Irrelevant' class is constructed by taking the initial model's top-500 lowest-confidence predictions on those same questions and using them as gold labels, then retraining the model and using its predictions as evidence about the corpus. This is a self-training loop; the resulting 'Irrelevant' share and the relative BT shares are partly determined by the initial model's own uncertainty rather than by external ground truth. However, the paper does provide an independent human-labeled evaluation of 300 questions (weighted F1=0.721), and the human confusion matrix gives the model partial support, so the central claim is not tautological. The low per-class recalls for Comprehension (0.148) and Analysis (0.176) are a serious correctness risk for the Knowledge-dominance conclusion, but that is a validity problem rather than a definitional circularity. No load-bearing self-citation chain or uniqueness-imported-from-authors pattern is present. Overall: partial circularity via pseudo-labeled 'Irrelevant' training, score 4.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the model's BT labels, which depend on the DASQBT training set, GPT-4o augmentation, the number of self-labeled 'Irrelevant' examples, and data filtering thresholds. These choices shape the resulting distribution and engagement correlations.

free parameters (5)
  • Number of low-confidence examples labeled Irrelevant = 500
    The paper selects the top 500 examples with the lowest maximum softmax probability from the model's predictions on YouTube questions and labels them as 'Irrelevant' (Section III-C2). This threshold directly shapes the final model's behavior and the resulting BT distribution.
  • Question token length cutoffs = 3 to 50 tokens
    Comments/questions outside this length are removed before and after extraction (Sections III-A and III-B), affecting which questions enter the dataset and analysis.
  • Comment limit per video = 1000
    The paper limits comments per video to 1000 'to avoid bias towards popular videos' (Section III-A), which changes the sample composition for videos with more than 1000 comments.
  • KD temperature tau and loss weight alpha = tau=2, alpha=2.5
    Hyperparameters for the knowledge-distillation training of the question detection model (Equation 1, Section III-B). They affect which comments are classified as questions, though the impact on the central claims is indirect.
  • Number of GPT-4o generated augmentation examples = approx. 3,246 (from 2,533 to 5,779)
    Data augmentation expands the DASQBT training set via GPT-4o; the exact number per class is not specified, and the generated questions change the training distribution (Section III-C1).
assumptions (6)
  • domain assumption Bloom's Taxonomy levels are a valid and meaningful way to categorize the cognitive complexity of learner questions.
    The entire analysis uses BT as the framework (Section I, Table I). If BT does not capture the cognitive complexity of informal YouTube questions, the distributional claims lose meaning.
  • domain assumption The DASQBT dataset labels are correct and representative of educational questions.
    The BT classifier is fine-tuned on DASQBT (Section III-C); label errors or unrepresentative examples would propagate to the YouTube question labels.
  • domain assumption Softmax probabilities can effectively identify out-of-distribution examples.
    The paper relies on Hendrycks and Gimpel [56] to label the top-500 low-confidence predictions as 'Irrelevant' (Section III-C2).
  • domain assumption GPT-4o generated questions and labels are consistent with human judgments.
    Data augmentation uses GPT-4o to generate labeled questions (Section III-C1). The paper reports high agreement on the DASQBT test set (Table III), but this does not guarantee the generated questions resemble YouTube comments.
  • domain assumption Likes and replies normalized by views are meaningful engagement metrics.
    The engagement analysis (Section IV-E, Figure 3) uses these normalized rates as proxies for popularity and interaction.
  • domain assumption The YouTube videos selected are representative of online educational content.
    The sample is drawn from popular channels and playlists (Section III-A); chemistry videos have far fewer views (37K) than biology (782K), so representativeness is uncertain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of YouLeQD: Decoding the Cognitive Complexity of Questions and Engagement in Online Educational Videos from Learners' Perspectives." pith.science (2026). https://pith.science/paper/7W3COBMV

@misc{pith2026250111712,
  author       = {Pith},
  title        = {Pith review of: YouLeQD: Decoding the Cognitive Complexity of Questions and Engagement in Online Educational Videos from Learners' Perspectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7W3COBMV}},
  note         = {Machine review of arXiv:2501.11712}
}
read the original abstract

Questioning is a fundamental aspect of education, as it helps assess students' understanding, promotes critical thinking, and encourages active engagement. With the rise of artificial intelligence in education, there is a growing interest in developing intelligent systems that can automatically generate and answer questions and facilitate interactions in both virtual and in-person education settings. However, to develop effective AI models for education, it is essential to have a fundamental understanding of questioning. In this study, we created the YouTube Learners' Questions on Bloom's Taxonomy Dataset (YouLeQD), which contains learner-posed questions from YouTube lecture video comments. Along with the dataset, we developed two RoBERTa-based classification models leveraging Large Language Models to detect questions and analyze their cognitive complexity using Bloom's Taxonomy. This dataset and our findings provide valuable insights into the cognitive complexity of learner-posed questions in educational videos and their relationship with interaction metrics. This can aid in the development of more effective AI models for education and improve the overall learning experience for students.

Figures

Figures reproduced from arXiv: 2501.11712 by the authors.

Figure 1
Figure 1. Confusion Matrix of Human Annotators and Model [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Question Type Distribution by Cognitive Complexity [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Popularity and Interaction Rate vs. Bloom’s Taxonomy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 54 canonical work pages

  1. [1]

    Enhancing peer interaction and learning in the classroom through reciprocal questioning,

    A. King, “Enhancing peer interaction and learning in the classroom through reciprocal questioning,” American educational research journal, vol. 27, no. 4, pp. 664–687, 1990

    work page 1990

  2. [2]

    Student-generated questions: A meaningful aspect of learning in science,

    C. Chin and D. E. Brown, “Student-generated questions: A meaningful aspect of learning in science,” International Journal of Science Educa- tion, vol. 24, no. 5, pp. 521–549, 2002

    work page 2002

  3. [3]

    Teaching students to generate questions: A review of the intervention studies,

    B. Rosenshine, C. Meister, and S. Chapman, “Teaching students to generate questions: A review of the intervention studies,” Review of educational research, vol. 66, no. 2, pp. 181–221, 1996

    work page 1996

  4. [4]

    Improved learning based on genai,

    A. R. Borah, T. Nischith, and S. Gupta, “Improved learning based on genai,” in 2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) . IEEE, 2024, pp. 1527–1532

    work page 2024

  5. [5]

    Generative ai as a learning buddy and teach- ing assistant: Pre-service teachers’ uses and attitudes,

    M. Nyaaba, L. Shi, M. Nabang, X. Zhai, P. Kyeremeh, S. A. Ayoberd, and B. N. Akanzire, “Generative ai as a learning buddy and teach- ing assistant: Pre-service teachers’ uses and attitudes,” arXiv preprint arXiv:2407.11983, 2024

    arXiv 2024

  6. [6]

    How does genai affect trust in teacher-student relationships? insights from students’ assessment experiences,

    J. Luo, “How does genai affect trust in teacher-student relationships? insights from students’ assessment experiences,” Teaching in Higher Education, pp. 1–16, 2024

    work page 2024

  7. [7]

    The process of developing a rubric to assess the cognitive complexity of student- generated multiple choice questions in medical education,

    R. Grainger, E. Osborne, W. Dai, and D. Kenwright, “The process of developing a rubric to assess the cognitive complexity of student- generated multiple choice questions in medical education,” The Asia Pacific Scholar, vol. 3, no. 2, p. 19, 2018

    work page 2018

  8. [8]

    Student-generated pre-exam questions is an effective tool for partici- patory learning: A case study from ecology of waterborne pathogens course,

    M. Teplitski, T. Irani, C. J. Krediet, M. Di Cesare, and M. Marvasi, “Student-generated pre-exam questions is an effective tool for partici- patory learning: A case study from ecology of waterborne pathogens course,” Journal of Food Science Education , vol. 17, no. 3, pp. 76–84, 2018

    work page 2018

Show all 60 references
  1. [9]

    Coaching for questioning: A study on the impact of questioning,

    N. A. Irons, “Coaching for questioning: A study on the impact of questioning,” Unpublished Capstone Action Research Project). Fielding Graduate University, Santa Barbara, CA , 2014

    2014

  2. [10]

    Gagne’s nine events of instruction: An introduction,

    K. Kruse, “Gagne’s nine events of instruction: An introduction,” Re- trieved the, vol. 10, 2009

    2009

  3. [11]

    Kolb’s experiential learning model: critique from a modelling perspective,

    H. Bergsteiner, G. C. Avery, and R. Neumann, “Kolb’s experiential learning model: critique from a modelling perspective,” Studies in continuing education, vol. 32, no. 1, pp. 29–46, 2010

    2010

  4. [12]

    First principles of instruction,

    M. D. Merrill, “First principles of instruction,” Educational technology research and development, vol. 50, pp. 43–59, 2002

    2002

  5. [13]

    Interacting with educational chatbots: A systematic review,

    M. A. Kuhail, N. Alturki, S. Alramlawi, and K. Alhejori, “Interacting with educational chatbots: A systematic review,” Education and Infor- mation Technologies, vol. 28, no. 1, pp. 973–1018, 2023

    2023

  6. [14]

    Developing an ai-based chatbot for practicing responsive teaching in mathematics,

    D. Lee and S. Yeo, “Developing an ai-based chatbot for practicing responsive teaching in mathematics,” Computers & Education, vol. 191, p. 104646, 2022

    2022

  7. [15]

    Generating multiple choice questions for computing courses using large language models,

    A. Tran, K. Angelikas, E. Rama, C. Okechukwu, D. H. Smith, and S. MacNeil, “Generating multiple choice questions for computing courses using large language models,” in 2023 IEEE Frontiers in Education Conference (FIE) . IEEE, 2023, pp. 1–8

    2023

  8. [16]

    An automatic question generation system using rule-based approach in bloom’s tax- onomy,

    D. Gnanasekaran, R. Kothandaraman, and K. Kaliyan, “An automatic question generation system using rule-based approach in bloom’s tax- onomy,” Recent Advances in Computer Science and Communications (Formerly: Recent Patents on Computer Science) , vol. 14, no. 5, pp. 1477–1487, 2021

    2021

  9. [17]

    Evaluating ai- generated questions: a mixed-methods analysis using question data and student perceptions,

    R. Van Campenhout, M. Hubertz, and B. G. Johnson, “Evaluating ai- generated questions: a mixed-methods analysis using question data and student perceptions,” in International Conference on Artificial Intelli- gence in Education . Springer, 2022, pp. 344–353

    2022

  10. [18]

    Towards automated generation and evaluation of questions in educational domains

    S. Bhat, H. A. Nguyen, S. Moore, J. C. Stamper, M. Sakr, and E. Nyberg, “Towards automated generation and evaluation of questions in educational domains.” in EDM, 2022

    2022

  11. [19]

    Evaluation of youtube videos for patients’ education on periradicular surgery,

    A. Jamleh, M. Nassar, H. Alissa, and A. Alfadley, “Evaluation of youtube videos for patients’ education on periradicular surgery,” Plos one, vol. 16, no. 12, p. e0261309, 2021

    2021

  12. [20]

    Can youtube enhance student nurse learning?

    A. Clifton and C. Mann, “Can youtube enhance student nurse learning?” Nurse education today , vol. 31, no. 4, pp. 311–313, 2011

    2011

  13. [21]

    The use of youtube video in encouraging speaking skill,

    A. Wahyuni, A. R. Utami, and E. Education, “The use of youtube video in encouraging speaking skill,” Pustakailmu. id, vol. 7, no. 3, pp. 1–9, 2021

    2021

  14. [22]

    Investigating the effectiveness of youtube as a learning tool among efl students at baghdad university,

    N. Fadhil Abbas and T. Ali Qassim, “Investigating the effectiveness of youtube as a learning tool among efl students at baghdad university,” Arab World English Journal (AWEJ) Special Issue on CALL, no. 6, 2020

    2020

  15. [23]

    An empirical analysis of knowledge co- construction in youtube comments,

    I. Dubovi and I. Tabak, “An empirical analysis of knowledge co- construction in youtube comments,” Computers & Education , vol. 156, p. 103939, 2020

    2020

  16. [24]

    A comment on learning: Media literacy practices in youtube,

    E. M. Meyers, “A comment on learning: Media literacy practices in youtube,” international Journal of Learning and Media , vol. 4, no. 3-4, 2014

    2014

  17. [25]

    Making sense of comments on youtube educational videos: A self-directed learning perspective,

    C. S. Lee, H. Osop, D. H.-L. Goh, and G. Kelni, “Making sense of comments on youtube educational videos: A self-directed learning perspective,” Online information review , vol. 41, no. 5, pp. 611–625, 2017

    2017

  18. [26]

    Sight: A large annotated dataset on student insights gathered from higher education transcripts,

    R. E. Wang, P. Wirawarn, N. Goodman, and D. Demszky, “Sight: A large annotated dataset on student insights gathered from higher education transcripts,” arXiv preprint arXiv:2306.09343 , 2023

    2023 arXiv

  19. [27]

    Using student-generated questions for student-centred assessment,

    T. Papinczak, R. Peterson, A. S. Babri, K. Ward, V . Kippers, and D. Wilkinson, “Using student-generated questions for student-centred assessment,” Assessment & Evaluation in Higher Education , vol. 37, no. 4, pp. 439–452, 2012

    2012

  20. [28]

    J. T. Dillon, Questioning and teaching: A manual of practice . Wipf and Stock Publishers, 2004

    2004

  21. [29]

    Survey of the state of the art in natural language generation: Core tasks, applications and evaluation,

    A. Gatt and E. Krahmer, “Survey of the state of the art in natural language generation: Core tasks, applications and evaluation,” Journal of Artificial Intelligence Research , vol. 61, pp. 65–170, 2018

    2018

  22. [30]

    Evaluation methodologies in automatic question generation 2013-2018,

    J. Amidei, P. Piwek, and A. Willis, “Evaluation methodologies in automatic question generation 2013-2018,” in Proceedings of the 11th International Conference on Natural Language Generation , 2018, pp. 307–317

    2013

  23. [31]

    A systematic review of automatic question generation for educational purposes,

    G. Kurdi, J. Leo, B. Parsia, U. Sattler, and S. Al-Emari, “A systematic review of automatic question generation for educational purposes,” International Journal of Artificial Intelligence in Education , vol. 30, pp. 121–204, 2020

    2020

  24. [32]

    Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications,

    N. Mulla and P. Gharpure, “Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications,” Progress in Artificial Intelligence , vol. 12, no. 1, pp. 1–32, 2023

    2023

  25. [33]

    Using full-information item analysis to improve item quality,

    T. M. Haladyna and M. C. Rodriguez, “Using full-information item analysis to improve item quality,”Educational Assessment, vol. 26, no. 3, pp. 198–211, 2021

    2021

  26. [34]

    Evaluating the psychometric characteristics of gen- erated multiple-choice test items,

    M. J. Gierl, H. Lai, D. Pugh, C. Touchie, A.-P. Boulais, and A. De Champlain, “Evaluating the psychometric characteristics of gen- erated multiple-choice test items,” Applied Measurement in Education , vol. 29, no. 3, pp. 196–210, 2016

    2016

  27. [35]

    Exploring student understanding of the engineering design process using distractor analysis,

    S. A. Wind, M. Alemdar, J. A. Lingle, R. Moore, and A. Asilkalkan, “Exploring student understanding of the engineering design process using distractor analysis,” International Journal of STEM Education , vol. 6, pp. 1–18, 2019

    2019

  28. [36]

    Developing the test of inquiry skills: measuring the level of inquiry skills among pupils in slovakia,

    D. ˇSmida, E. ˇCipkov´a, and M. Fuchs, “Developing the test of inquiry skills: measuring the level of inquiry skills among pupils in slovakia,” International Journal of Science Education , vol. 46, no. 1, pp. 73–108, 2024

    2024

  29. [37]

    Item analysis: the impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items,

    A. A. Rezigalla, A. M. E. S. A. Eleragi, A. B. Elhussein, J. Alfaifi, M. A. ALGhamdi, A. Y . Al Ameer, A. I. O. Yahia, O. A. Mohammed, and M. I. E. Adam, “Item analysis: the impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice ite...

    2024

  30. [38]

    Handbook i: cognitive domain,

    B. S. Bloom, M. D. Engelhart, E. Furst, W. H. Hill, and D. R. Krathwohl, “Handbook i: cognitive domain,” New York: David McKay, pp. 483–498, 1956

    1956

  31. [39]

    Understanding by design: A framework for effecting curricular development and assessment,

    D. Roth, “Understanding by design: A framework for effecting curricular development and assessment,” CBE—Life Sciences Education , vol. 6, no. 2, pp. 95–97, 2007

    2007

  32. [40]

    Implementing standards-based edu- cation. student assessment series

    R. J. Marzano and J. S. Kendall, “Implementing standards-based edu- cation. student assessment series.” 1998

    1998

  33. [41]

    L. D. Fink, Creating significant learning experiences: An integrated approach to designing college courses . John Wiley & Sons, 2013

    2013

  34. [42]

    Question classification based on bloom’s taxonomy cognitive domain using modified tf-idf and word2vec,

    M. Mohammed and N. Omar, “Question classification based on bloom’s taxonomy cognitive domain using modified tf-idf and word2vec,” PloS one, vol. 15, no. 3, p. e0230442, 2020

    2020

  35. [43]

    Classification-driven intelligent system for automated evaluation of higher education exam paper quality,

    H. M. Ahmed and S. E. Sorour, “Classification-driven intelligent system for automated evaluation of higher education exam paper quality,” Education and Information Technologies , pp. 1–27, 2024

    2024

  36. [44]

    Identification of cognitive learning complexity of assessment questions using multi-class text classification,

    S. Das, S. K. D. Mandal, and A. Basu, “Identification of cognitive learning complexity of assessment questions using multi-class text classification,” Contemporary Educational Technology , vol. 12, no. 2, p. ep275, 2020

    2020

  37. [45]

    A bibliometric review of large language models research from 2017 to 2023,

    L. Fan, L. Li, Z. Ma, S. Lee, H. Yu, and L. Hemphill, “A bibliometric review of large language models research from 2017 to 2023,” arXiv preprint arXiv:2304.02020, 2023

    2017 arXiv

  38. [46]

    Large language models as financial data annotators: A study on effectiveness and efficiency,

    T. Aguda, S. Siddagangappa, E. Kochkina, S. Kaur, D. Wang, C. Smiley, and S. Shah, “Large language models as financial data annotators: A study on effectiveness and efficiency,” arXiv preprint arXiv:2403.18152, 2024

    2024 arXiv

  39. [47]

    Annollm: Making large language models to be better crowdsourced annotators,

    X. He, Z. Lin, Y . Gong, A. Jin, H. Zhang, C. Lin, J. Jiao, S. M. Yiu, N. Duan, W. Chen et al., “Annollm: Making large language models to be better crowdsourced annotators,” arXiv preprint arXiv:2303.16854 , 2023

    2023 arXiv

  40. [48]

    Is a large language model a good annotator for event extraction?

    R. Chen, C. Qin, W. Jiang, and D. Choi, “Is a large language model a good annotator for event extraction?” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, 2024, pp. 17 772– 17 780

    2024

  41. [49]

    E-qgen: Educational lecture abstract-based question generation system,

    M.-S. Chen and A.-Z. Yen, “E-qgen: Educational lecture abstract-based question generation system,” arXiv preprint arXiv:2404.13547 , 2024

    2024 arXiv

  42. [50]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  43. [51]

    Questions vs statements classification,

    S. Khan, “Questions vs statements classification,” May 2021. [Online]. Available: https://www.kaggle.com/datasets/shahrukhkhan/ questions-vs-statementsclassificationdataset

    2021

  44. [52]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  45. [53]

    The spaadia annotation scheme,

    G. Leech and M. Weisser, “The spaadia annotation scheme,” 2013

    2013

  46. [54]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  47. [55]

    Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,

    H. Zhou and L. Song, “Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,” in Proceedings of International Conference on Learning Representations (ICLR) , 2021

    2021

  48. [56]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks,

    D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations , 2022. APPENDIX A CONFUSING STATEMENTS IN BT Q UESTIONS CLASSIFICATION The log probabilities are u...

    2022

  49. [57]

    Removal of the ending question marks

  50. [58]

    Incomplete sentences

  51. [59]

    Use of wh-words in non-interrogative sentences

  52. [60]

    Could you guys please upload the problem sets for 5.111 on the OCW web page?

    Mix of interrogative and non-interrogative sentences in one statement. TABLE VIII Confusing Statements with Low Confidence Scores from GPT-4o Text Logprobs Houston was founded in 1836 on land near the banks of Buffalo Bayou (now known as Allen’s Landing) and incorporated as a ...

    1952

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.