{"id":"270ae935-6a6c-4d3a-94a8-6d2db0b68e56","arxiv_id":"2502.10408","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SQKT, a knowledge tracing model that embeds student questions and auto-extracted Python skills, predicts next-problem success more accurately than code-only baselines on four Python courses from a Korean e-learning platform.","lead":"A new AI model for programming courses predicts whether a student will solve the next coding problem by also analyzing the questions they ask between attempts, and it reports much higher accuracy than code-only baselines. The result matters because student questions carry confusion signals that code submissions alone miss, which could make adaptive tutoring systems more responsive.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The success-label threshold is computed on the full course, including test students, so test labels are not independent of the training data. This contaminates the reported AUC gains and directly threatens the central claim that student questions improve prediction.","rationale":"The paper's central claim is that student questions and automatically extracted skill information improve prediction of subsequent problem success. All evidence for this claim is based on AUC and related metrics, and those metrics depend entirely on how success is labeled. Section 4.2 explicitly states that the submission-count threshold is the average across all students in the course. Since the data are split by student into train, validation, and test, test students are included in that average, so the label for a test student's attempt depends on a statistic that includes that same student and other test students. This is a form of transductive label construction: the test labels are not generated by a rule fixed before seeing the test set. The effect may be small in practice, but the paper provides no analysis showing it is negligible, and the claimed gains are large enough that even a modest label shift could change the ranking of models. Moreover, the abstract's '33.1% absolute improvement' is selected against the weakest baseline, not the best baseline (Section 5.1 reports 12.6-20.8 versus KTMFF+), so the headline number already overstates the improvement relative to the strongest competitor. The reader correctly identified the threshold leakage as the weakest assumption. The paper's other problems, such as contradictory sample sizes in Section 4.2 versus Tables 1 and 7, missing error bars, and the lack of released code and data, all reinforce the need for a rerun of the evaluation under a cleaner protocol, but they are secondary to the label-construction issue. A training-only threshold is a minimal, concrete fix that would directly test whether the central claim survives. Because the requested rerun is feasible and the concern is addressable, the existing CONDITIONAL verdict is appropriate and does not need to be changed.","tokens_in":11611,"tokens_out":7151,"duration_ms":78338,"concrete_test":"Re-run the full Table 2 evaluation with the success threshold computed exclusively from the training split, and as a sensitivity check also with threshold = training_mean ± 1 submission. If the SQKT versus KTMFF+ AUC gap changes by more than about 2 absolute points on any course, the headline improvement is substantially an artifact of the leakage; if the gap is stable, this particular concern does not invalidate the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines success as scoring 100 within a threshold equal to the average number of submissions across all students in each course. Because this threshold is computed on the entire course, test students contribute to the statistic that determines their own labels, breaking the train/test independence that the 8:1:1 student split is meant to guarantee. If the threshold shifts by even one submission, a nontrivial fraction of test labels flip. Since SQKT's input history includes the number and content of prior submissions through the k-row student history tensors, the model can exploit features that correlate with the label-construction rule. The reported 33.1-point absolute AUC improvement over the weakest baseline (OKT, Table 2) and the 12.6-20.8-point improvement over the strongest baseline (KTMFF+) could therefore be inflated by label-construction artifacts rather than by the question and skill signals the paper claims are central. The absence of error bars and the internal inconsistencies in cross-domain sample sizes (Section 4.2 reports 5,858 and 1,674 samples, while Tables 1 and 7 give far larger training/test counts) further weaken confidence, but the threshold issue is the most direct threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SQKT, a knowledge tracing model for programming education that augments code submissions and problem descriptions with student questions, educator responses, and GPT-extracted skill labels. The model is evaluated on four Python courses from a Korean online learning platform using in-domain and cross-domain settings, with ablations and error analyses. The authors report large improvements in AUC, accuracy, and F1 over existing code-based KT baselines, and claim that student questions and automatically extracted skills provide predictive signals beyond code submissions.","tokens_in":11825,"tokens_out":6005,"duration_ms":57836,"significance":"If the reported results are reliable, this is a useful contribution: it demonstrates that student-educator question interactions carry information for knowledge tracing that is not available in code submissions alone, and it proposes a practical way to obtain skill labels without full manual tagging. The paper includes a new real-world dataset, a plausible model architecture, and a range of experiments including cross-domain transfer to data-scarce courses. The main limitations are methodological: the success-label construction depends on test-set statistics, the experiments lack error bars, and the reported dataset sizes are internally inconsistent. These issues directly affect the credibility of the central quantitative claims.","major_comments":[{"comment":"The success threshold is defined as the average number of submissions across all students in each course, and this average is computed over the full course data, including students in the test split. Because test students contribute to the statistic that determines their own success labels, the test labels are not independent of the training data. This violates the student-level split intended by the 8:1:1 partitioning and can let the model exploit features correlated with the label-construction rule (e.g., the number of submissions encoded in the Ui tensors). Please recompute all success labels using a threshold estimated from the training students only (or from a nested cross-validation over students), report the resulting metrics for all models, and include a sensitivity analysis to the threshold choice.","section":"Section 4.2 (Evaluation Metrics)"},{"comment":"No error bars, confidence intervals, or repeated-seed results are reported anywhere in the paper. All conclusions rest on single point estimates from one run per configuration, and the margins in some ablations are very small (e.g., Table 4 shows AUC differences of 0.8-1.4 points). This makes the claimed improvements, including the 33.1% absolute AUC gain in the abstract, statistically unverifiable. Please run at least five seeds per model and report means and standard deviations, and apply a significance test (e.g., paired bootstrap over students) for the key comparisons in Table 2 and the ablation tables.","section":"Section 5 (Results, Tables 2-4, Figure 4)"},{"comment":"The sample sizes for the cross-domain settings do not match the dataset statistics in Tables 1 and 7. The text states that the 'Python Introduction' course has 5,858 samples and 'First Python' has 1,674, but Table 7 lists 4,135 and 11,302 test problems, respectively; the '9,390 samples' for the combined training set also does not correspond to any sum in Tables 1/7. Please clarify what unit is being counted (students, problem attempts, or submissions) and correct the tables and text so that the experimental setup is reproducible.","section":"Section 4.2 (Cross-Domain Experimental Setup)"},{"comment":"The text says the ablation results are on the 'Python Basic' course, but the table caption says the study is on the 'Python Introduction' course. Additionally, the claim that 'the same pattern is observed in other courses' is not supported by any reported results. Please correct the caption-to-text mismatch and provide ablation results for all three courses, either in the main text or in an appendix.","section":"Section 5.1, Table 3 (Ablation Study)"}],"minor_comments":[{"comment":"The abstract's '33.1% absolute improvement in AUC compared to baseline models' is ambiguous, because Table 2 shows that this is the gain over OKT (the weakest baseline on Python Introduction), not over the strongest baseline (KTMFF+). Please state explicitly that the gain is relative to a specific baseline or report the comparison to the best-performing baseline.","section":"Abstract and Section 5.1"},{"comment":"The validation of the skill extractor uses 100 questions annotated by a co-author, and the reported Cohen's kappa of 0.98 is between that co-author and a graduate student, not against an independent gold standard. This is a weak external anchor for the claim that the extractor is reliable. Please provide more detail on the annotation protocol or temper the reliability claim.","section":"Section 3.1 (Skill Extraction)"},{"comment":"The caption says 'Impact of response and triplet loss functions,' but the text refers to 'question loss.' Use consistent terminology for this auxiliary loss.","section":"Table 4 caption"},{"comment":"Table 7 reports 'Python Basic # of problems' as 2,665, 362, 412, while Table 1 lists 48 unique problems with 474 submissions per problem. These numbers are not reconcilable as given; please clarify whether the rows in Table 7 count problem attempts (student-problem pairs) and use a consistent term throughout.","section":"Tables 1 and 7"},{"comment":"The sentence 'excluding one due to insufficient data' is vague; please name the excluded course (Algorithm, as implied by Table 2) and state the criterion used for 'insufficient.'","section":"Section 4.2 (In-Domain)"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and addresses a meaningful gap, but the evaluation is not yet trustworthy. The threshold leakage in Section 4.2 is the most serious issue; if the authors cannot re-run the experiments with a training-only threshold, the central claim about the benefit of student questions would not be supported. The inconsistencies in the dataset statistics also need thorough correction before the paper can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid first cut at something that's genuinely missing: KT for programming that uses student questions as a signal. The architecture is assembled from known pieces, but the combination is new, and the ablations give me confidence that the questions themselves carry predictive information beyond code submissions. The skill extraction via a GPT-4o-generated rule-based script is a pragmatic choice, and the validation, though thin, is at least an attempt at grounding it.\n\nThe main problem is the success label. Section 4.2 defines success as scoring 100 within a threshold equal to the average number of submissions across all students in the course. Since the split is by student, the test students' submission counts feed into that average, so the test labels are not independent of the training distribution. The model sees submission counts in its input history, which makes it plausible that some of the large AUC gains come from learning the label rule rather than from the question and skill signals. The ablations still show a similar pattern, so I don't think the qualitative conclusion collapses, but the headline numbers are not trustworthy as reported. They need to recompute the threshold using only training students and report variance across seeds.\n\nThere are smaller issues: no error bars anywhere; the cross-domain sample counts in Section 4.2 (5,858 and 1,674) don't match Table 1 or Table 7; the OKT+ row in Table 2 repeats identical numbers for two different courses, which looks like a copy-paste error. The skill extractor was validated on 100 questions annotated by a co-author; that's a weak anchor, though the kappa is high. Code and data are not released.\n\nOverall, the paper deserves a serious referee because the idea is timely and the experiments, once cleaned up, could support a real contribution. I'd want to see the label issue fixed and the numbers reconciled before believing the 33-point claim, but the direction is worth engaging with.","headline":"A promising but compromised first attempt at using student questions for knowledge tracing; the label leakage needs to be fixed before the headline numbers can be trusted.","tokens_in":12364,"tokens_out":2705,"would_cite":false,"duration_ms":26592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Student questions, combined with automatically extracted skill labels, predict a student's success on the next programming problem more accurately than code submissions alone.","keywords":["knowledge tracing","programming education","student questions","skill extraction","code embeddings","transformer","AUC","adaptive learning"],"falsifier":"Re-run the in-domain experiments with the success threshold computed only from training students, or set to a fixed per-problem value, and compare AUC across models; if SQKT's 33.1-point gain over the baseline shrinks dramatically, the reported improvement is partly an artifact of using test data in the threshold.","tokens_in":11372,"feed_emoji":"💬","tokens_out":7340,"duration_ms":69778,"temperature":0.7,"pith_summary":"This paper argues that in programming courses, the questions students ask between submissions reveal knowledge gaps that code alone does not capture, and that a knowledge tracing model can exploit those questions to predict whether a student will succeed on the next problem. It introduces SQKT, which embeds student questions, code submissions, problem statements, and skill labels extracted from questions and reference solutions, then feeds the fused history through attention layers. The authors report that this question-aware model beats code-only baselines by large margins and generalizes across courses with different difficulty levels and limited data. If correct, the result would let adaptive learning systems in computer science use naturally occurring Q&A forum activity, instead of manually tagged skills, to decide what to teach next.","feed_headline":"Student questions sharpen knowledge tracing by up to 33 AUC points","feed_subtitle":"A new model mines Q&A forum questions and auto-extracted Python skills to predict a student's next coding result.","key_machinery":"The carrying component is the question-embedding channel, built on CodeT5 and fine-tuned to predict educator responses, combined with the GPT-generated skill extractor and a triplet-loss fusion layer. The skill extractor is a rule-based script produced by GPT-4o that maps noisy Korean-and-code question text to 36 Python concepts and 19 error types; the same script runs on reference solutions to obtain the skills each target problem requires. The fusion layer projects question, code, problem, and skill embeddings into a shared space using triplet loss, so the attention layers can compare a student's historical questions and submissions with the next problem's requirements.","core_discovery":"SQKT is, the authors claim, the first knowledge tracing model for programming that treats student questions as a first-class input. The model encodes each question with CodeT5, an encoder-decoder trained on code and natural language, and strengthens that embedding by training the encoder to generate the educator's response as an auxiliary task. A GPT-4o-derived rule-based skill extractor labels both the student's question and the target problem's reference solution with a fixed set of 36 Python concepts and 19 error types; these skill labels are embedded with BERT and combined with CodeBERT code embeddings and BERT problem embeddings. A triplet-loss fusion layer projects all embeddings into one 512-dimensional space, and six multi-head self-attention layers trace the student's history to predict success on the target problem. In in-domain tests SQKT reaches AUC 87.1-93.4, beating the strongest baseline by 12.6-20.8 percentage points and the OKT baseline by up to 33.1 points on one course; in cross-domain tests question input yields large gains, including an 11.4-point gain over in-domain training on a data-scarce algorithm course.","pith_inferences":["Editorial inference: the 33.1-point headline gain is measured against OKT, which has low AUC on these data; against the stronger KTMFF+ baseline the gain is 12.6-20.8 points, so the practical improvement over a strong code-based model is large but smaller than the headline number suggests.","Editorial inference: because success labels depend on a course-wide average submission count that includes test students, re-computing with a training-only threshold could change all reported gains; the model's relative ranking might survive, but the magnitude is unverified.","Editorial inference: the rule-based, GPT-generated skill extractor was validated on 100 questions from one course, so its generalization to other languages, frameworks, or question styles is untested, and a machine-learned extractor might improve scalability.","Editorial inference: the error analysis indicates that complexity from mixed Korean-English code is the most common failure mode, so multilingual tokenization or normalization is a natural next step that could push accuracy further."],"forward_implications":["Knowledge tracing for programming can work without expert-created knowledge-component tags: skills are extracted automatically from questions and reference solutions.","Student questions are a usable behavioral signal even in noisy real classroom data, since the model did not preprocess the questions.","The approach can transfer across courses with different content structures and to advanced courses with too little data to train in-domain models.","The auxiliary task of predicting educator responses improves question embeddings, turning Q&A forum data into a training signal rather than just an input.","SQKT's question embeddings can be appended to existing KT models, such as KTMFF+ and OKT+, and improve their accuracy too."],"supporting_citations":[{"why":"Supplies the Bayesian knowledge tracing framework and the task definition of predicting next-problem performance.","marker":"Corbett and Anderson, 1994"},{"why":"Introduces deep knowledge tracing, the deep-learning approach that the code-based baselines and sequence modeling build on.","marker":"Piech et al., 2015"},{"why":"Provides CodeT5, the encoder-decoder used to embed students' mixed natural-language/code questions.","marker":"Wang et al., 2021"},{"why":"Provides BERT, used to embed problem descriptions and extracted skill labels.","marker":"Devlin et al., 2018"},{"why":"Provides CodeBERT, used to embed code submissions.","marker":"Feng et al., 2020"},{"why":"OKT, the open-ended knowledge tracing baseline that SQKT is compared against.","marker":"Liu et al., 2022"},{"why":"KTMFF, the multi-feature fusion baseline whose question-augmented variant is SQKT's strongest competitor.","marker":"Xiao et al., 2023"}],"fun_headline_variants":["Student questions boost knowledge tracing by 33 AUC points","First KT model to mine student questions for coding success","SQKT: Questions as first-class input lift coding skill prediction","Ask and learn: Student Q&A improves knowledge tracing 33 points","Student questions as input: SQKT lifts coding prediction by 33 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A student is labeled successful only if they reach a perfect score within the average number of submissions taken by all students in the course, and that average includes the students whose performance is being predicted.","fun_headline_variants_meta":{"raw":{"variants":["Student questions boost knowledge tracing by 33 AUC points","First KT model to mine student questions for coding success","SQKT: Questions as first-class input lift coding skill prediction","Ask and learn: Student Q&A improves knowledge tracing 33 points","Student questions as input: SQKT lifts coding prediction by 33 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000531,"raw_usage":{"total_tokens":2567,"prompt_tokens":968,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1514}},"tokens_in":584,"tokens_out":1599,"duration_ms":13261,"temperature":1.0,"reasoning_tokens":1514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:40:07.986633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the in-domain experiments with the success threshold computed only from training students, or set to a fixed per-problem value, and compare AUC across models; if SQKT's 33.1-point gain over the baseline shrinks dramatically, the reported improvement is partly an artifact of using test data in the threshold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian knowledge tracing framework and the task definition of predicting next-problem performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces deep knowledge tracing, the deep-learning approach that the code-based baselines and sequence modeling build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OKT, the open-ended knowledge tracing baseline that SQKT is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KTMFF, the multi-feature fusion baseline whose question-augmented variant is SQKT's strongest competitor."}],"review_version":1}