{"id":"db151da0-fd2b-4454-980c-49f4a8e90939","arxiv_id":"2501.05415","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"UKT models student knowledge as Gaussian distributions, uses Wasserstein self-attention and an aleatory-uncertainty contrastive loss, and reports improved knowledge tracing prediction on six datasets.","lead":"This paper presents UKT, a knowledge tracing model that represents a student's knowledge state as a probability distribution with a mean and an uncertainty, and that uses a Wasserstein-based attention mechanism to track how that distribution changes. It also adds a contrastive learning step meant to make predictions robust to careless mistakes and lucky guesses, and it reports higher AUC than several deep learning baselines on six education datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The response-flipping negative samples alter the full learning history, not isolated aleatory events, so the claimed epistemic/aleatory separation is unverified and the robustness claim depends on it.","rationale":"The paper is a serious engineering contribution: it provides a code repository, carefully evaluates on six benchmarks, and the ablations show each component helps. However, its headline claim is twofold: (1) UKT surpasses deep KT baselines; (2) it handles interaction uncertainty by separating epistemic from aleatory components. The AUC evidence for (1) is real but mostly within about 0.5-1% of strong baselines like AKT and simpleKT, and it loses on AS2015, so 'significantly surpasses' is not statistically tested. The deeper issue is (2). The response-flipping negative sampling is not a valid operationalization of aleatory uncertainty as defined in the paper. Flipping all previous correct/incorrect responses changes the entire learning trajectory; the contrastive loss then makes the model invariant to broad historical changes, not specifically to careless errors or lucky guesses. The paper offers only qualitative heatmaps and a table where injecting AU hurts all models, to support the decomposition. This is exactly the reader's weakest assumption. If a labeled or quasi-experimental test fails, the uncertainty-handling claim collapses, even though the benchmark AUC might survive. The malformed equations (Eqs. 4 and 7) add a reproduction barrier, but they are fixable with code access; the conceptual flaw in negative sampling is more fundamental. Hence the verdict remains conditional: the work is promising, but the central robustness claim needs a direct measurement before acceptance beyond corrections.","tokens_in":12247,"tokens_out":5806,"duration_ms":55554,"concrete_test":"Label as 'likely aleatory' any interaction where the same student gives inconsistent responses to the same KC across attempts in ASSISTments2009 or NIPS34. Compare UKT's predicted covariance at these interactions against matched consistent ones, and measure accuracy on the labeled subset with and without the contrastive loss. If covariance does not differ significantly or the loss does not improve accuracy on those items, the Eq. 7 separation claim fails. Analytically, compute the Eq. 4 distance for flipping only the last response versus flipping all prior responses; if the latter dominates, the negatives are not targeted at aleatory noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is the claimed separation of epistemic from aleatory uncertainty via (i) Gaussian covariance embeddings and (ii) the aleatory-aware contrastive loss (Eq. 7) with response-flipping negative samples (Sec. 'Aleatory Uncertainty-Aware Contrastive Learning'). This separation is load-bearing for the 'handling uncertainty' claim, but it is asserted, not measured. The negative-sequence construction is especially problematic: if the last response is correct, all previous correct responses are inverted; if incorrect, all previous incorrect responses are inverted (keeping the last response). This produces a sequence whose entire history differs from the positive, not a controlled perturbation of a single aleatory event. Optimizing against such negatives cannot isolate 'careless mistakes' or 'lucky guesses' as defined in the Preliminaries; it may simply regularize the representation to be insensitive to any response flips, potentially discarding real learning signal. No evidence ties the learned covariance to labeled aleatory events: the only support is the heatmaps in Figs. 5-6 and Table 3, where adding 'AU' always hurts performance for every model, including UKT (just less). Thus the central claim that UKT retains epistemic uncertainty while downweighting aleatory uncertainty is unsupported by the experiments; the AUC improvements could be due to the extra stochastic embeddings or the contrastive regularizer regardless of aleatory semantics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Summary: The paper proposes UKT, a knowledge tracing model that represents each student's knowledge state as a Gaussian distribution with learned mean and covariance embeddings, uses a Wasserstein-based self-attention mechanism to update these distributions, and adds an aleatory uncertainty-aware contrastive loss. The contrastive negatives are constructed by flipping previous correct or incorrect responses depending on the last response. The paper reports AUC and accuracy on six datasets, ablations, sensitivity to the contrastive weight, and covariance heatmaps, and claims that UKT outperforms existing deep KT baselines and better handles aleatory uncertainty.","tokens_in":12505,"tokens_out":10086,"duration_ms":88531,"significance":"If the central claims were established, UKT would be a useful contribution to knowledge tracing: stochastic state representations with calibrated uncertainty could support more reliable mastery assessment, and a training scheme that downweights careless mistakes and lucky guesses would be practically valuable. The paper has concrete strengths: it evaluates on six benchmark datasets, compares with six recent baselines, reports ablations for each architectural component, and provides a public code repository. Against that, the uncertainty-decomposition story is currently asserted rather than measured, the contrastive loss in Eq. (7) has an apparent sign inconsistency, and the 'consistently outperforms' claim is contradicted by the ASSIST2015 row of Table 1.","major_comments":[{"comment":"Eq. (4) is not a well-defined 2-Wasserstein distance as printed. The trace term `M^sigma + E^sigma - 2 ((M^sigma)^{1/2} M^sigma (E^sigma)^{1/2})^{1/2}` mixes the matrices and their square roots without indicating the intended square root of a product; the standard closed forms are `trace(M^sigma + E^sigma - 2 (M^sigma E^sigma)^{1/2})` for commuting matrices or the Bures form `trace(M^sigma + E^sigma - 2 (M^sigma^{1/2} E^sigma M^sigma^{1/2})^{1/2})`. In addition, `R_t` in `W2(M_t, R_t)` is never defined. The attention mechanism cannot be implemented from the paper as written.","section":"Methodology, Eq. (4)"},{"comment":"The contrastive objective in Eq. (7) is internally inconsistent. Because W2 is a distance, the positive-pair term should use `exp(-W2(h_i, h_i^-))` if the flipped sequence is the positive view; as written, minimizing `L_CL` maximizes `W2(h_i, h_i^-)`, pushing the two views apart. Moreover, qs_i^- is called a negative sample, but it appears in the numerator as the positive anchor, and the denominator uses in-batch sequences as negatives. The paper must clarify whether the flipped sequence is a positive or negative view and must correct the sign convention; the current formulation does not express the stated goal of robustness to aleatory uncertainty.","section":"Aleatory Uncertainty-Aware Contrastive Learning, Eq. (7)"},{"comment":"The construction of qs_i^- is not a controlled manipulation of aleatory uncertainty. When the last answer is correct, the rule flips all previous correct responses; when the last is incorrect, it flips all previous incorrect responses. This changes the student's entire learning history, so the resulting sequence does not isolate a careless mistake or a lucky guess. No experiment validates the claimed epistemic/aleatory separation against labeled, simulated, or otherwise known uncertainty events; Figures 5-6 only display averaged covariance values, and Table 3 reports aggregate accuracy differences. The load-bearing claim that UKT retains epistemic uncertainty while suppressing aleatory uncertainty is therefore not supported by the evidence presented.","section":"Aleatory Uncertainty-Aware Contrastive Learning (negative construction)"},{"comment":"The summary of RQ1 contains a direct contradiction. The text says 'UKT consistently outperforms the other baselines in AUC scores across all datasets,' but Table 1 shows UKT (0.7267 +/- 0.0007) below AKT (0.7282 +/- 0.0004) and DKT (0.7271 +/- 0.0005) on ASSIST2015; the same paragraph later acknowledges that UKT 'only slightly underperforms baselines on the ASSIST2015 dataset.' The benchmark claim should be corrected to 'best on five of six datasets,' and in view of the 0.001 to 0.009 AUC margins on the other datasets, the paper should include paired significance tests (per-student bootstrap or repeated cross-validation paired tests) to support the word 'significantly.'","section":"Experiments, RQ1 (Table 1)"},{"comment":"Table 3 is the only quantitative evidence for the robustness advantage, but it is not convincing. The table shows that adding AU reduces simpleKT by 1.14%, AKT by 1.33%, and UKT by 0.83%, yet it reports no confidence intervals or significance tests; with differences at the 0.01-AUC scale, the conclusion that UKT is 'less affected' is fragile. The table also does not define precisely what 'with AU' changes in the data or training, and the UKT values in Table 3 (0.8501/0.8430) differ from the UKT value in Table 1 (0.8563), so the protocol underlying Table 3 must be clarified before the robustness claim can be evaluated.","section":"Uncertainty Analysis (RQ3), Table 3"}],"minor_comments":[{"comment":"The notation in Eq. (1) is under-specified: `W1_c` and `W2_c` are used but never defined, and it is unclear how `e^sigma_qj` and `e^mu_qj` relate to the one-hot response vector `e_qj` introduced in the text.","section":"Methodology, Eq. (1)"},{"comment":"The description of WassersteinSelfAttention never states how the score in Eq. (4) is converted into attention weights (e.g., softmax over keys), so the attention mechanism is not fully specified.","section":"Methodology, Eq. (5)"},{"comment":"The captions and text say the 'mean of the covariance matrix' is plotted, but it is not stated whether the average is over sequence positions, students, or batch elements, and the heatmap axes are not labeled.","section":"Uncertainty Analysis, Figures 5-6"},{"comment":"The column headers 'w/o AU' and 'with AU' are ambiguous; the text should specify what the AU manipulation consists of and reconcile the UKT AUC with Table 1.","section":"Table 3"},{"comment":"The abbreviations 'Stocemb' and 'W.dist' are not defined; they should be expanded to stochastic embedding and Wasserstein distance, and the ablation rows need significance tests or confidence intervals to support component contributions.","section":"Table 4"},{"comment":"The POJ dataset URL in footnote 5 contains a space and is not clickable; please provide a corrected link and cite the original source for POJ.","section":"Datasets"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a learning-analytics or educational-AI venue, and the benchmark studies are useful. The biggest risk is overclaiming the uncertainty decomposition; I recommend major revision rather than rejection because the AUC results, the ablations, and the code release give the authors a basis to fix the issues with additional experiments and clarifications. I saw no evidence of citation manipulation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"UKT is a genuine effort to bring uncertainty modeling into deep knowledge tracing, and the empirical package is mostly solid: code link, six datasets, consistent if small AUC gains, ablations supporting each component. That part deserves review. The epistemic/aleatory separation story, by contrast, is asserted and illustrated rather than measured, and the two key equations are not correctly written.\n\nNew here is the combination of Gaussian stochastic embeddings, Wasserstein self-attention, and a contrastive loss with response-flipped negatives for KT. That's a legitimate new application of ideas borrowed from sequential recommendation. The benchmark is honest: UKT wins on five of six datasets by 0.001–0.009 AUC and loses on ASSIST2015.\n\nThe soft spots are real. Equation (4) is not a proper 2-Wasserstein distance as printed; the matrix terms are muddled. Equation (7) appears to invert the contrastive objective, pushing positive pairs apart rather than together. Both are fixable, but the paper is not reproducible from the text alone. The deeper issue is the aleatory semantics. The negative-sampling scheme flips all previous correct or incorrect responses, not a single careless mistake or lucky guess, so the learned regularizer can't be said to isolate aleatory events. There is no labeled-uncertainty evaluation; the heatmaps show only that covariance embeddings are stable within a sequence. And Table 3 shows that adding the aleatory \"AU\" term hurts all models, including UKT—that's a robustness effect, not evidence that UKT 'retains epistemic uncertainty' as a separate quantity. The claim of being 'the first to explore uncertainty in KT' also overstates: Bayesian Knowledge Tracing is the classic probabilistic KT model, and should have been cited and discussed.\n\nStill, the main prediction result is benchmarked against external baselines, so no circularity there. The empirical story will likely hold up once the equations are fixed and a proper uncertainty evaluation is added. This is a substantive contribution worth peer review, but the revisions are major: correct the equations, tone down the novelty claim, and add a direct measurement of aleatory vs epistemic uncertainty. I'd send it out, expecting heavy revision.","headline":"UKT is a solid empirical package with fixable equation errors, but the epistemic/aleatory separation claim is unmeasured and the 'first in KT' phrasing ignores BKT.","tokens_in":13045,"tokens_out":3530,"would_cite":false,"duration_ms":33638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modeling student knowledge as a Gaussian beats deep KT baselines","keywords":["knowledge tracing","uncertainty quantification","Gaussian embeddings","Wasserstein self-attention","contrastive learning","aleatory uncertainty","epistemic uncertainty","student performance prediction"],"falsifier":"A concrete test: take a dataset in which some responses are annotated as careless mistakes or guesses, then flip only those annotated responses to build negative samples and compare with flipping the same number of random responses. If the annotated flips change predictions no more than random flips, or if covariance magnitudes do not differ between students with known different mastery levels, the claimed separation between epistemic and aleatory uncertainty is not doing the work the paper assigns to it.","tokens_in":12005,"feed_emoji":"🎓","tokens_out":8331,"duration_ms":69837,"temperature":0.7,"pith_summary":"The paper's central claim is that knowledge tracing models should not represent a student's knowledge state as a single point, because careless errors and lucky guesses make any one interaction an unreliable sample of true mastery. The authors propose to represent each state as a Gaussian distribution whose mean encodes baseline knowledge and whose covariance encodes uncertainty, and to track how these distributions change with a Wasserstein-distance self-attention mechanism. They add an aleatory-uncertainty-aware contrastive loss that builds negative sequences by flipping earlier responses while keeping the last outcome, simulating lucky guesses and careless mistakes, and trains the model to move encoded states away from these negative views. On six real-world datasets the model reports higher AUC than the deep sequential, memory-augmented, adversarial, graph, and attention baselines it compares with, and it shows smaller performance drops than two strong attention baselines when aleatory noise is injected. The authors take this as evidence that modeling uncertainty, not just the expected knowledge state, improves both prediction and resilience against noisy interactions.","feed_headline":"Gaussian student states beat deep KT baselines","feed_subtitle":"Uncertainty-aware embeddings lift prediction on six real-world datasets and give per-student uncertainty estimates.","key_machinery":"The central machinery is the Gaussian state representation paired with Wasserstein-based self-attention. A mean embedding and a covariance embedding define each interaction and knowledge component as a multidimensional Gaussian, with the covariance kept positive definite by applying ELU plus one to the diagonal. The attention score between two states is the negative 2-Wasserstein distance $W_2$, so the model retrieves historical knowledge states that overlap with the current state distributionally rather than only by pointwise embedding similarity. The contrastive loss completes the mechanism: flipped-response sequences serve as negative anchors in an InfoNCE-style objective expressed through Wasserstein distances between encoded distributions, and the final loss is a weighted sum of the prediction loss and this contrastive loss. This machinery is what turns uncertainty from a modeling byproduct into an explicit, trainable quantity.","core_discovery":"UKT's core discovery is that stochastic distribution embeddings can be learned for knowledge tracing end to end. Each interaction and each knowledge concept is mapped to a mean embedding and a covariance embedding that together define a multivariate Gaussian; the mean represents the student's mastery and the covariance represents the uncertainty of that assessment. A self-attention layer compares these Gaussian states with the 2-Wasserstein distance, which accounts for both mean difference and covariance overlap, and retrieves historical states accordingly. The aleatory-uncertainty-aware contrastive loss flips all correct responses in sequences ending in a correct response and flips all incorrect responses in sequences ending in an incorrect response, creating negative samples that mimic lucky guesses and careless mistakes; treating those flips as negatives is what makes the model resilient to aleatory noise while preserving epistemic uncertainty. Because every component contributes in the ablations and because the model outperforms DKT, SAKT, SAINT, ATKT, AKT, and SimpleKT on most of the six benchmarks, the paper argues that the uncertainty-aware representation itself is the reason for the gains.","pith_inferences":["Beyond the paper: the Wasserstein self-attention could transfer to adaptive quiz selection, where a content item's difficulty profile and a student's Gaussian knowledge state naturally call for a distributional overlap score.","Beyond the paper: if datasets with explicit annotations of careless mistakes or guesses exist, the response-flipping negative sampling could be validated directly by checking whether those annotated interactions behave like the constructed negatives.","Beyond the paper: the heatmap evidence that within-student covariance stays fairly stable over time suggests the learned covariance could be used as a student trait signal, for instance to flag students who rush or guess, though the paper does not test this against external labels."],"forward_implications":["If UKT's reported results hold, uncertainty-aware Gaussian embeddings become a viable replacement for deterministic embeddings in deep knowledge tracing, since the model wins or ties on AUC across all six datasets.","The covariance embeddings give educators a per-student, per-concept uncertainty signal in addition to a mastery score, so a correct answer accompanied by high covariance can be interpreted differently from a confident correct answer.","The contrastive negative-sampling strategy offers a way to make knowledge tracing models resist careless mistakes and lucky guesses without requiring labels for which interactions are noisy.","The ablations imply that each of the three components, the stochastic embeddings, the Wasserstein distance, and the contrastive loss, adds predictive value, so a simpler deterministic model would not reproduce the result.","The reported single-GPU training time suggests the uncertainty-aware architecture is practical for large-scale online learning platforms, not just small benchmarks."],"supporting_citations":[{"why":"Supplies the DKT baseline and the deterministic fixed-embedding approach the paper argues against.","marker":"(Piech et al. 2015)"},{"why":"Supplies the AKT baseline and the context-aware embedding structure from which UKT's mean and covariance embeddings are developed.","marker":"(Ghosh, Heffernan, and Lan 2020)"},{"why":"Supplies the SimpleKT baseline and the standard dot-product self-attention whose computational cost UKT's Wasserstein attention is compared with.","marker":"(Liu et al. 2023)"},{"why":"Supplies the data preprocessing steps, evaluation protocol, and benchmark baselines the experiments follow.","marker":"(Liu et al. 2022)"},{"why":"Supplies the SAKT baseline, an early self-attention knowledge tracing model used as a comparison point.","marker":"(Pandey and Karypis 2019)"},{"why":"Supplies the SAINT baseline, an encoder-decoder attention model included in the comparison.","marker":"(Choi et al. 2020)"},{"why":"Supplies the stochastic self-attention idea for sequential data that UKT adapts to knowledge tracing.","marker":"(Fan et al. 2022)"},{"why":"Supplies the categorical cross-entropy contrastive objective used in the aleatory-uncertainty-aware contrastive learning loss.","marker":"(Oord, Li, and Vinyals 2018)"}],"fun_headline_variants":["Stochastic states model uncertainty in knowledge tracing","Wasserstein attention captures uncertainty in KT","UKT uses Gaussian embeddings to beat deep baselines","Aleatory-aware loss makes KT robust to noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Gaussian mean-and-covariance representation, together with response-flipping negatives, separates epistemic uncertainty, the genuine differences in what students know, from aleatory uncertainty, careless mistakes and lucky guesses, well enough that training against the negatives improves prediction without throwing away useful learning signal.","fun_headline_variants_meta":{"raw":{"variants":["Stochastic states model uncertainty in knowledge tracing","Wasserstein attention captures uncertainty in KT","UKT uses Gaussian embeddings to beat deep baselines","Aleatory-aware loss makes KT robust to noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000335,"raw_usage":{"total_tokens":1849,"prompt_tokens":927,"completion_tokens":922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":864}},"tokens_in":543,"tokens_out":922,"duration_ms":8770,"temperature":1.0,"reasoning_tokens":864,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:14:18.918340+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take a dataset in which some responses are annotated as careless mistakes or guesses, then flip only those annotated responses to build negative samples and compare with flipping the same number of random responses. If the annotated flips change predictions no more than random flips, or if covariance magnitudes do not differ between students with known different mastery levels, the claimed separation between epistemic and aleatory uncertainty is not doing the work the paper assigns to it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SAKT baseline, an early self-attention knowledge tracing model used as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic self-attention idea for sequential data that UKT adapts to knowledge tracing."}],"review_version":1}