{"id":"f463db6e-5bf8-4bd9-aa74-440fc799f03c","arxiv_id":"2501.11712","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Learner questions on YouTube lecture videos mostly fall into the lowest Bloom's Taxonomy level, and harder questions receive fewer likes but more replies.","lead":"This paper introduces a dataset of 57,242 learner questions scraped from comments on YouTube lecture videos, labeled by cognitive difficulty using Bloom's Taxonomy. It also analyzes how question difficulty relates to likes and replies on the videos, which could help improve AI tutoring systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BT distribution and engagement correlations are computed from model predictions without correcting for the classifier's low recall on Comprehension (0.148) and Analysis (0.176), so the 'most questions are Knowledge' claim may be an artifact of label bias.","rationale":"Agree with the reader's weakest assumption. The classifier's human-evaluated recall values are strikingly low for Comprehension and Analysis, the very categories that would matter for the 'low-cognitive-level' narrative. The paper's own Table VI shows that at human level, Comprehension is nearly as frequent as Knowledge among the relevant questions (27 vs 25), yet the model recalls only 14.8% of Comprehension. Without a correction, the model's predicted counts cannot be trusted. The proposed test—a larger human sample with confusion-matrix adjustment—is the standard way to settle whether the distributional claim survives. The engagement correlations depend on the same labels, so they inherit the same risk. The paper deserves credit for releasing data and code and for including a human evaluation, but the evaluation is used only as a headline accuracy number, not as input to the estimation. The conditional verdict is appropriate: the claims should be accepted only after the corrected distribution is reported.","tokens_in":13924,"tokens_out":5120,"duration_ms":49847,"concrete_test":"Draw a new stratified random sample of 600 questions (120 per subject) from the 57,242 extracted questions, annotate them with the same BT protocol, and compute the model's confusion matrix on this sample. Use the inverse of that matrix (or a Bayesian latent-class estimator) to correct the model's predicted counts on the full dataset, yielding an unbiased BT distribution with confidence intervals. If the corrected Knowledge proportion is not significantly larger than the next-largest class (or if it drops by more than 10 percentage points relative to the reported ~46%), the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that most learner questions fall under the Knowledge level and that higher BT levels correlate with interaction rate—is derived from RoBERTa predictions on all 57,242 extracted questions. The only independent check (Section IV-B, Table VI) evaluates the model on 300 human-labeled questions and reports weighted F1=0.721, with per-class recall of 0.148 for Comprehension and 0.176 for Analysis, and 0.360 for Knowledge. These low recalls mean the classifier systematically undercounts middle-level questions. The paper does not use the human confusion matrix to re-estimate or adjust the dataset-wide BT distribution, nor does it report confidence intervals for the Figure 2 percentages. As a result, the reported 44–48% Knowledge share could be inflated by misclassified Comprehension/Analysis questions, and the inverse popularity/cognitive-level relationship in Section IV-E could reflect label noise rather than a true engagement pattern. The 'Irrelevant' class, constructed from the model's own low-confidence predictions, may further bias the distribution by absorbing difficult genuine questions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces YouLeQD, a dataset of 57,242 learner-posed questions extracted from comments on 1,762 YouTube educational lecture videos across five STEM subjects. It also trains two RoBERTa-based classifiers: one detects interrogative sentences in comments (via knowledge distillation from GPT-4o), and one assigns Bloom's Taxonomy (BT) cognitive levels to the extracted questions, using GPT-4o-generated augmentation and an added 'Irrelevant' class. The authors report that most learner questions fall under the lowest BT level, Knowledge (44–48% across subjects), that higher BT levels correlate positively with interaction rate (replies) but negatively with popularity (likes), and that subject-level distributions differ. The dataset and code are publicly released. The central empirical claims rest on applying the BT classifier to the full corpus, with a human evaluation of only 300 examples used as an external check.","tokens_in":14181,"tokens_out":4151,"duration_ms":43813,"significance":"If the distributional and engagement findings are correct, YouLeQD would be a useful public resource for studying learner questioning behavior in informal online education, and the paper's pipeline (LLM-assisted augmentation, knowledge distillation, OOD-aware classification) could inform similar annotation efforts. The strengths are the scale of the released dataset, the public code, the use of an independent human-labeled test set (albeit small), and the explicit attempt to handle out-of-distribution questions. However, the headline conclusions are currently only as strong as the classifier's predictions, and the human evaluation shows substantial per-class error, especially for Comprehension and Analysis. Because the paper does not correct for this measurement error, the reported BT distribution and engagement correlations must be treated as provisional.","major_comments":[{"comment":"The headline result that 44–48% of learner questions are Knowledge-level is computed from RoBERTa predictions on the full 57,242 questions, but the only human check, Table VI, shows per-class recall of 0.360 for Knowledge, 0.148 for Comprehension, and 0.176 for Analysis. The relevant-class supports are also very small (7 for Application, 3 for Synthesis). The paper does not use the human confusion matrix to re-estimate the dataset-level BT distribution or report confidence intervals for Figure 2. Because the model systematically undercounts middle-level classes, the reported Knowledge share could be inflated by misclassified Comprehension and Analysis questions. Please add a confusion-matrix-based correction (or equivalent re-weighting) and uncertainty bounds, and show how the Section V conclusion changes under plausible label-error scenarios.","section":"§IV-B, §IV-C, Fig. 2"},{"comment":"The construction of the Irrelevant class is circular: the top-500 lowest-confidence predictions from the first-stage model are labeled Irrelevant and added to the training set, and the same model family is later used to label the full corpus from which the distributional claims are drawn. Table VI complicates interpretation further: human annotators labeled 214 of 300 (71.3%) extracted questions as Irrelevant, yet Figure 2 reports six BT-level percentages with no Irrelevant category mentioned. The paper must state explicitly whether Figure 2 excludes Irrelevant predictions. If it excludes them, the claim that 'most questions fall under the lowest cognitive level' is conditional on the non-irrelevant subset and should be reframed; if it includes them, the figure is inconsistent with the human evaluation. In either case, the self-training construction should be validated with human labels rather than the model's own confidence scores.","section":"§III-C.2, §IV-C, Table VI"},{"comment":"The engagement analysis in Figure 3 is based on the same noisy model labels and reports no statistical tests, confidence intervals, or effect sizes. With per-class recall as low as 0.148 (Comprehension), the inverse popularity/cognitive-level relationship and the positive interaction-rate correlation from Analysis to Synthesis could be artifacts of label noise. Please report the correlation estimates with uncertainty (for example, via bootstrap or using a bias-corrected label distribution), state the effective sample size per BT level after grouping by video, and clarify exactly how the rates are normalized by view counts.","section":"§IV-E, Fig. 3"}],"minor_comments":[{"comment":"The table title contains a typo: 'YoulQD' should be 'YouLeQD'.","section":"Table II"},{"comment":"The question-detection model is reported as achieving 99.42% F1, but the test-set size and split details are not given; please specify them.","section":"§IV-A"},{"comment":"The knowledge-distillation loss is not fully defined: the argument of the exponential and the summation indices over k should be made explicit, and the choices of τ = 2 and α = 2.5 should be justified relative to the cited reference.","section":"Eq. (1)"},{"comment":"The sentence 'Table I lists the cognitive levels of Bloom’s taxonomy and provides examples illustrative examples of questions' contains a duplicated phrase; it should read 'illustrative examples'.","section":"§II-B"},{"comment":"The reported GPT-4o agreement of 0.988 on DASQBT does not state how many examples were evaluated; please add the sample size and any filtering criteria.","section":"Table III"},{"comment":"The caption says 'Confusion Matrix of Human Annotators and Model Prediction', but only panel (a) shows human annotator agreement; panel (b) compares human labels with model predictions. The caption should describe each panel separately.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The dataset release and the general approach are valuable, but the central empirical claims are currently not supported by the evaluation as presented. The missing bias correction for the classifier and the ambiguity around the Irrelevant class are load-bearing issues that require a substantial reanalysis. I would be willing to review a revision that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the YouLeQD dataset is a real resource, but the headline finding that most learner questions sit at the Knowledge level is not backed by the classifier evidence as presented. The paper deserves a serious referee, but the central empirical claims need a re-analysis with human labels in the loop.\n\nWhat's new and good: a corpus of 57,242 learner questions from YouTube STEM lecture comments, with transcripts and engagement metadata, publicly available. That's a genuinely useful resource for educational NLP. The engagement analysis (likes/replies versus Bloom's level) is a reasonable first look, and the authors were transparent enough to report a human evaluation on 300 samples with a confusion matrix, plus code and a repo. Credit where due: they also report negative results from data augmentation, which is more than many papers do.\n\nThe soft spots: the BT classifier's human-evaluated recall is very low for Comprehension (0.148) and Analysis (0.176), while Knowledge recall is 0.360. The dataset-level distribution in Figure 2 comes straight from model predictions, with no correction using the human labels and no confidence intervals. If the model systematically undercounts middle levels, the '44-48% Knowledge' claim could be a label-bias artifact. The 'Irrelevant' class is built from the model's own low-confidence predictions, so it can swallow difficult genuine questions. That is a real circularity. A straightforward fix is to re-estimate the distribution from a larger human-labeled random sample, or at least apply a confusion-matrix-based correction and report uncertainty. The engagement correlations in Figure 3 are also purely descriptive, with no significance tests or effect sizes.\n\nOne mismatch: the conclusion admits the small annotation sample as a limitation, but the abstract states the finding unconditionally. That should be toned down until the re-analysis is done.\n\nBottom line: this is a useful dataset and a promising analysis pipeline, but the empirical claims outrun the validation. I'd send it to peer review because the resource matters and the issues are fixable.","headline":"YouLeQD is a useful new dataset, but the paper's central claim about knowledge-level dominance outruns the classifier's measured recall.","tokens_in":14672,"tokens_out":2876,"would_cite":true,"duration_ms":27596,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new dataset of 57,242 learner questions from YouTube lecture comments, labeled by Bloom's Taxonomy, shows most questions sit at the lowest cognitive level, Knowledge, while harder questions are less liked but more likely to draw replies.","keywords":["Bloom's Taxonomy","question classification","learner questions","YouTube educational videos","RoBERTa","knowledge distillation","student engagement","cognitive complexity"],"falsifier":"Take a stratified random sample of roughly 1,000 to 2,000 extracted questions, have three or more independent raters assign Bloom's levels, then recompute the level distribution and the likes-per-view and replies-per-view correlations on the human labels; if the ordering or sign of the correlations differs materially, the central claim is falsified.","tokens_in":13744,"feed_emoji":"🎓","tokens_out":4976,"duration_ms":47499,"temperature":0.7,"pith_summary":"This paper introduces YouLeQD, a dataset of 57,242 learner-posed questions extracted from comments on 1,762 YouTube lecture videos across five STEM subjects. It trains two RoBERTa-based classifiers: one detects which comments are questions, and one assigns each question a Bloom's Taxonomy cognitive level. The paper's central finding is that most learner questions are classified as Knowledge, the lowest level, with higher levels such as Analysis and Synthesis comparatively rare. It also reports an inverse relationship between question popularity (likes normalized by views) and cognitive level, alongside a positive correlation between interaction rate (replies) and higher levels from Analysis to Synthesis. If these findings hold, they suggest that online learners mostly ask factual recall questions, and that cognitively demanding questions are what drive discussion rather than endorsement.","feed_headline":"Learner questions on YouTube cluster at lowest cognitive level","feed_subtitle":"Analysis of 57,242 comments finds most are recall-level, yet harder questions spur more replies.","key_machinery":"The machinery is a two-stage classification pipeline. A fine-tuned RoBERTa model, trained with knowledge distillation from GPT-4o soft labels on an existing interrogative-sentence dataset, detects question sentences in comments. A second RoBERTa model, fine-tuned on the DASQBT question dataset (a small multi-class question dataset labeled by Bloom's levels) and augmented by GPT-4o-generated questions, assigns each question to Bloom's six levels plus an 'Irrelevant' class, with out-of-distribution detection based on low softmax probabilities. The Bloom's classifier then produces the distributional and engagement results across all 57,242 questions.","core_discovery":"On the paper's own terms, the discovery is that large-scale learner questioning on YouTube is dominated by low-cognitive questions: Knowledge accounts for about 44 percent to 48 percent of questions across subjects, whereas higher-level categories (Analysis, Evaluation, Synthesis) together range from roughly 14.6 percent to 27.6 percent. The paper further claims that popularity, measured by likes per view, falls as Bloom's level rises, while interaction rate, measured by replies per view, increases for the higher levels. These patterns are presented as evidence about how learners actually engage with educational videos, and as a resource for aligning AI question-generation systems with genuine learner behavior.","pith_inferences":["The paper's headline percentages are only as trustworthy as the classifier's per-class recall, which human evaluation shows is low for Comprehension and Analysis; correcting the dataset-wide distribution with human labels could shift the reported ranking of cognitive levels.","If the engagement pattern is causal rather than merely correlational, instructional designers could use cognitive-level labels to identify high-value discussion threads and route them to instructors or answer engines.","A testable extension the paper does not pursue would be to compare the same questions' Bloom levels as assigned by multiple human raters versus the model; the current 300-sample evaluation with majority-vote labels does not measure rater disagreement.","Because the 'Irrelevant' class dominates human labels (214 of 300 samples), many YouTube comments classified as questions are not genuine learning questions, so educational AI built on this data should filter or reweight that class."],"forward_implications":["If the distributional findings are representative, AI question-generation systems should be calibrated to the reality that learners mostly ask recall-level questions, and should aim to prompt deeper questions rather than assume they occur naturally.","The dataset provides a public corpus of naturally occurring learner questions with Bloom labels, enabling comparisons across biology, chemistry, math, physics, and computer science.","The engagement result implies that reply-based discussion is more likely to be stimulated by higher-order questions, whereas like-based popularity tracks simpler content, which matters for how instructors and platforms surface comments.","The classification method demonstrates a workflow for using LLM-generated labels and knowledge distillation to annotate noisy social-media text at scale."],"supporting_citations":[{"why":"Supplies Bloom's Taxonomy, the cognitive-complexity framework the whole classification and analysis rests on.","marker":"[38]"},{"why":"Supplies DASQBT, the small multi-class question dataset used to train the Bloom's Taxonomy classifier.","marker":"[44]"},{"why":"Provides RoBERTa, the base model fine-tuned for both question detection and Bloom classification.","marker":"[50]"},{"why":"Supplies the interrogative-sentence classification dataset used to train the question-detection model.","marker":"[51]"},{"why":"Provides the precedent of using GPT-4 as a scaling annotator on educational video comments, a method this paper adapts.","marker":"[26]"},{"why":"Introduces knowledge distillation, the learning framework used to improve the question-detection model.","marker":"[54]"},{"why":"Provides the soft-label loss design used in the knowledge-distillation objective.","marker":"[55]"},{"why":"Supplies the softmax-probability baseline for detecting out-of-distribution examples, used to add the 'Irrelevant' class.","marker":"[56]"}],"fun_headline_variants":["YouTube learner questions skew to recall level, YouLeQD shows","Higher Bloom's level questions earn more replies on YouTube","YouLeQD: learners ask simple questions, but complex ones engage","New educational dataset reveals question complexity in videos","Most YouTube lecture questions are basic, but harder ones get replies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the Bloom classifier's labels being accurate across all 57,242 questions; the human checkpoint covers only 300 questions, and per-class recall is very low for some categories, so the reported distribution and engagement correlations could change if model errors were corrected.","fun_headline_variants_meta":{"raw":{"variants":["YouTube learner questions skew to recall level, YouLeQD shows","Higher Bloom's level questions earn more replies on YouTube","YouLeQD: learners ask simple questions, but complex ones engage","New educational dataset reveals question complexity in videos","Most YouTube lecture questions are basic, but harder ones get replies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":1922,"prompt_tokens":846,"completion_tokens":1076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":995}},"tokens_in":462,"tokens_out":1076,"duration_ms":9416,"temperature":1.0,"reasoning_tokens":995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:55:45.128747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified random sample of roughly 1,000 to 2,000 extracted questions, have three or more independent raters assign Bloom's levels, then recompute the level distribution and the likes-per-view and replies-per-view correlations on the human labels; if the ordering or sign of the correlations differs materially, the central claim is falsified.","supporting_citations":[{"cited_title":"Handbook i: cognitive domain,","cited_arxiv_id":null,"evidence_quote":"Supplies Bloom's Taxonomy, the cognitive-complexity framework the whole classification and analysis rests on."},{"cited_title":"Identification of cognitive learning complexity of assessment questions using multi-class text classification,","cited_arxiv_id":null,"evidence_quote":"Supplies DASQBT, the small multi-class question dataset used to train the Bloom's Taxonomy classifier."},{"cited_title":"Questions vs statements classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the interrogative-sentence classification dataset used to train the question-detection model."},{"cited_title":"SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts","cited_arxiv_id":"2306.09343","evidence_quote":"Provides the precedent of using GPT-4 as a scaling annotator on educational video comments, a method this paper adapts."},{"cited_title":"Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective,","cited_arxiv_id":null,"evidence_quote":"Provides the soft-label loss design used in the knowledge-distillation objective."},{"cited_title":"A baseline for detecting misclassified and out-of-distribution examples in neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the softmax-probability baseline for detecting out-of-distribution examples, used to add the 'Irrelevant' class."}],"review_version":1}