{"id":"bc8116b5-68e6-4290-b967-8a767f212e38","arxiv_id":"2502.10396","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DASKT uses behavioral logs, K-means clustering, and graph attention to simulate student affect and includes it in an LSTM knowledge tracer, claiming state-of-the-art results.","lead":"A new knowledge tracing model, DASKT, tries to infer students' emotional states (frustration, concentration, boredom, confusion) from their click and answer data, then feeds those emotions into an LSTM to predict future performance. The paper reports large accuracy gains over prior models on two ASSISTments datasets, but the affect features may leak future answers into the model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. 1-9 define affective factors from whole-sequence student statistics and Eq. 12 applies them to the training and test sets, so future responses leak into predictions at time t; the reported AUC gains may be a leakage artifact.","rationale":"The reader's weakest assumption identifies the same load-bearing concern, and the manuscript text provides direct support: Eq. 12 is explicitly applied to the training and test sets, and the affective factors in Eqs. 1-9 are defined without a time index, so they use the entire student sequence. Even if cluster centers were learned only on the training split, a test student's individual factors still encode future correctness, response times, attempts, and intervals. Since the affect embedding enters the LSTM input at every time step, the model can use this future information to increase prediction accuracy without learning a genuine affect-to-knowledge relationship. This is not a disagreement about whether affect can matter; it is a concrete correctness risk in the experimental protocol. The proposed prefix-only recomputation is the natural check. If DASKT's advantages persist under that protocol, the central claim would gain support; if they vanish, Table 4 and Table 5 are explained by leakage. No other concern is more load-bearing, because the entire claim of state-of-the-art predictive performance rests on the validity of the time-t affect features.","tokens_in":20826,"tokens_out":7448,"duration_ms":78848,"concrete_test":"Re-run DASKT with prefix-only affective factors: for each prediction of r_{t+1}, recompute A_si_k, AT_si_k, PR_si, Att_si_k, and IT_si in Eqs. 1-9 using only interactions with timestamps strictly before t, or only ICA segments that end before t, while keeping K-means centers and population statistics fixed from the training folds. Report AUC, ACC, RMSE, and r^2 on ASSIST2012 and ASSISTchall. If the gap to the best baseline shrinks to below the t-test threshold or reverses, the reported gains are a leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim requires that DASKT's affect input at time t is computable from past interactions only. As written, it is not. In Section 3.2, confidence uses A_si_k, the average correctness of student si on KC k over the full exercise sequence, and AT_si_k from the whole sequence (Eqs. 1-3); interest uses the total sequence length and total time span (Eqs. 4-6); effort sums attempts and intervals over the entire sequence (Eqs. 7-9). These features have no time index and are therefore identical at every t, encoding information about responses at t and later. Section 4.2 compounds the problem: after Eq. 12, the text states that |sn| is the set of all students in the training and test sets, so the cluster assignment used to label affect at time t can depend on test-set data as well. Because the resulting affect embedding is concatenated into the LSTM input at time t (Eq. 16) to predict r_{t+1}, the network has a direct path to future information. The ablation in Table 5 attributes roughly five AUC points to affect components, but those gains may reflect leakage rather than a causal affect effect. This is the load-bearing issue: if the affect features are made strictly causal, the headline outperformance claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DASKT, a knowledge tracing model that simulates four affective states (frustration, concentration, boredom, confusion) from behavioral records without explicit affect labels. Affective factors (confidence, interest, effort) are computed from whole-sequence statistics, clustered into four states via K-means, and then processed by a two-layer graph attention network to form a dynamic affect trajectory. The affect embedding is concatenated with problem, KC, and response embeddings into an LSTM to predict future performance. Experiments on ASSIST2012 and ASSISTchall report AUC of 0.7925 and 0.8529, outperforming 12 baselines, with an ablation study attributing roughly five AUC points to the affect components.","tokens_in":21152,"tokens_out":6259,"duration_ms":58791,"significance":"If the claims held, the paper would offer a low-cost, label-free way to incorporate affect into knowledge tracing and improve both accuracy and interpretability. The manuscript also states that data and code are released, which is a strength for reproducibility. However, the central performance claim is undermined by a severe information-leakage problem in the affective feature construction, and the affect validation is incomplete and partly ambiguous. As written, the reported gains cannot be attributed to a causal effect of affect, so the significance of the contribution is not established.","major_comments":[{"comment":"The affective factors are computed from whole-sequence statistics that include future responses. Specifically, Eq. (1) defines A_{s_i}^k as the average correctness over all occurrences of KC k in the student's entire sequence; Eq. (4) uses |el_{s_i}|, the full sequence length, to define participation rate; Eqs. (7)-(8) sum attempts and intervals over the whole sequence. These quantities have no time index and are therefore identical at every time step, encoding information about responses at time t and later. Eq. (12) then assigns an affect state to each segment using these same global factors, and Eq. (16) concatenates the resulting affect embedding into the LSTM input at time t to predict r_{t+1}. Consequently, the model has a direct path to future information, so the performance gains in Table 4 and the ablation findings in Table 5 may be leakage artifacts rather than evidence of a causal affect effect. The authors must reformulate the affective factors so that the value at time t depends only on interactions before t, or they must demonstrate that the reported results are unchanged under such a causal formulation.","section":"Section 3.2, Eqs. (1)-(9); Section 4.2, Eq. (12); Section 4.3, Eq. (16)"},{"comment":"The paper never specifies how the four K-means clusters are mapped to the emotion labels frustration, concentration, boredom, and confusion. Section 3.3.2 simply states that the affect embedding has four states, and Section 5.4 compares DASE to the affect detector of Pardos et al. [35]. Without an explicit mapping procedure (for example, matching clusters to detector labels on the training set), the consistency rates in Table 3 and the qualitative affect interpretations in Figures 4 and 5 are not reproducible, and the claim that no affect labels are used in training is difficult to verify.","section":"Section 4.2, Eq. (11)"},{"comment":"The per-affect AUC values (e.g., 15.27, 16.22, 20.17, 21.37) are implausible for an AUC metric, which must lie between 0 and 1 (or 0 and 100). The relationship between these per-affect values and the \"Total\" row (73.16, 75.24, etc.) is unexplained, and the notation \"↑0.12\" is ambiguous without a baseline definition. Moreover, the validation compares DASE with the affect detector on the same dataset used to develop the model; if the cluster-to-label mapping is chosen to maximize agreement with the detector, the consistency rates would be inflated. Please clarify the metric definitions, the mapping procedure, and whether detector labels are used only for evaluation.","section":"Table 3"},{"comment":"No standard deviations or confidence intervals are reported despite the stated use of 5-fold cross-validation. The significance asterisks in Table 4 are not accompanied by variance information, and the ablation in Table 5 also lacks error bars. Given the leakage issue in the affective features, the reported improvements cannot be judged as statistically reliable, and the comparisons with baselines may not be meaningful as presented.","section":"Tables 4 and 5"}],"minor_comments":[{"comment":"The notation AT_{s_i}^k = att_{s_i}^{t_k} is unclear because t_k is not defined; it appears to mix a time index with a KC index, and the equation should specify how response time is aggregated.","section":"Section 3.2.1, Eq. (1)"},{"comment":"The phrase \"|sn| is the set of all students in the training and test sets\" is confusing because Eq. (11) uses |sm| for the training set; the text should clarify how test students are assigned to clusters and whether any test-set statistics are used in building the clusters.","section":"Section 4.2, Eq. (12)"},{"comment":"The statement \"no real affect labels were used in training\" should be reconciled with the use of the affect detector in Section 5.4; please state explicitly whether the detector labels are used only for evaluation or also for choosing the cluster-to-emotion mapping.","section":"Section 5.1 and Section 5.4"},{"comment":"The heading \"Konwledge Tracing\" contains a typo; it should read \"Knowledge Tracing.\"","section":"Section 2.1 heading"},{"comment":"Figure 2 is dense and the subfigure labels (a), (b), (c) are not clearly separated in the rendered PDF, making it hard to map the text description to the diagram.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"The leakage issue in the affective feature construction is fundamental and invalidates the central empirical claim. Even though the authors could in principle reformulate the features to be causal, that would change the method and likely alter the results; the current manuscript does not support the claimed contribution. The idea of simulating affect from behavioral data is interesting, and the release of code and data is commendable, but the validation of the affect simulation itself (cluster-to-label mapping, Table 3) is also too incomplete to justify a revision without substantial rework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: DASKT is a genuinely new combination of affect simulation and knowledge tracing, but the headline AUC gains look like an artifact of data leakage. The authors build affect features from whole-sequence statistics and then feed them into an LSTM at each time step, so information about responses at time t and later is available when predicting r_{t+1}. That is a load-bearing flaw, not a nitpick.\n\nWhat's new and good: the architecture—mining confidence/interest/effort from behavioral logs, K-means to assign affect states, graph attention over the affect trajectory, then LSTM for KT—is a reasonable way to inject affect without labels. The idea of validating simulated affect against an existing affect detector on ASSIST2012 is smart, and the consistency rates in Table 3 (72-80%) are the most interesting result in the paper. The authors also promise code and data, which is commendable.\n\nThe soft spots, in order of severity:\n\n1. Leakage. Eqs. 1-9 define confidence, interest, and effort using averages and sums over the student's entire exercise sequence (and, in Eq. 2 and 5, over all students). Eq. 12 then assigns affect to every segment using cluster centers from the training and test sets. As written, the affect input at time t encodes future responses and even test-set information. The ablation in Table 5 credits affect components with ~5 AUC points, but those points may just be leakage. The paper's limitation section mentions the lack of gold-standard affect data but does not mention this.\n\n2. The cluster-to-emotion mapping is never explained. The K-means step produces four clusters; the paper simply calls them frustration, concentration, boredom, and confusion. Without a mapping rule, the affect labels are uninterpretable.\n\n3. Table 3's per-affect AUC values (10-25%) are implausible as AUC numbers; they look like a scaling or thresholding mistake. This makes the affect quality comparison hard to trust.\n\n4. No error bars anywhere, so the claimed improvements over 12 baselines are not shown to be stable.\n\nThe core idea is worth taking seriously, but the current evidence does not support the performance claim. This is fixable: recompute the factors per segment using only past interactions, specify the cluster-label mapping, fix the metric reporting, and rerun. If the gains survive, it's a useful contribution to educational data mining. As written, I wouldn't cite the numbers, and I wouldn't accept the paper.\n\nI'd still send it to reviewers—not to rubber-stamp, but because a careful referee can flag the leakage and the paper could be rehabilitated. For a reading group, it's a decent case study in how leakage hides in plain sight.","headline":"A novel affect-aware KT architecture undone, as written, by future-looking affect features; the core performance claim is not credible yet.","tokens_in":21671,"tokens_out":3476,"would_cite":false,"duration_ms":34361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Students' affective states can be simulated from ordinary answer logs, and injecting them into knowledge tracing improves prediction of future performance.","keywords":["knowledge tracing","affective computing","affect simulation","student performance prediction","graph attention network","LSTM","confidence-interest-effort","educational data mining"],"falsifier":"Recompute every affective factor (confidence, interest, effort) using only interactions strictly before the predicted time, retrain DASKT on the same data splits, and compare AUC; if the advantage disappears, the reported gains come from information leakage rather than from modeling affect.","tokens_in":20627,"feed_emoji":"🎓","tokens_out":13171,"duration_ms":111163,"temperature":0.7,"pith_summary":"Knowledge tracing predicts a student's next answer from their exercise history, and the paper claims that knowing how the student feels—frustrated, concentrated, bored, or confused—makes that prediction better and more interpretable. DASKT is a pipeline that derives three affective factors (confidence, interest, effort) from ordinary behavioral logs such as answer correctness, response time, participation frequency, attempts, and inter-question intervals; clusters students into four affective states; and models how affect changes over time using a graph attention network. On two public datasets, ASSIST2012 and ASSISTchall, the reported area under the ROC curve (AUC) is 0.7925 and 0.8529 respectively, above all twelve baselines, and the simulated affect agrees with an external affect detector on 72.74% to 80.29% of questions on ASSIST2012. If the claim holds, affect can be brought into intelligent tutoring systems at no extra sensing cost.","feed_headline":"Mining affect from answer logs boosts knowledge tracing accuracy","feed_subtitle":"The DASKT model outpredicts 12 baselines on two public datasets using no affect labels or sensors.","key_machinery":"The load-bearing mechanism is the affect trajectory graph built on interval-computed affect labels. DASKT first normalizes each student's confidence, interest, and effort vectors (the MAF module), clusters them into four affect states with K-means, and assigns an affect label to each interval of LSeg exercises (the ICA module). These interval labels become nodes of a graph in which each node is connected to its predecessor and successor, and a two-layer graph attention network—AT-GAT followed by A-GAT—produces a smoothed, coherent affect representation at each time step. That representation is concatenated with the problem, knowledge-concept, and response embeddings and fed into a long short-term memory (LSTM) network, so the student's knowledge state is updated under the influence of the simulated affect.","core_discovery":"The paper's central claim is that affect in online learning can be simulated from non-affect-oriented behavioral logs, without sensors, surveys, or gold-standard affect labels, and that the simulated affect improves knowledge tracing in two ways. On prediction, DASKT reports AUC of 0.7925 on ASSIST2012 and 0.8529 on ASSISTchall, beating all twelve baselines; on interpretability, the affect-conditioned knowledge states show the expected pattern that concentrated students gain more and frustrated or bored students lose more for the same answers. The simulated affect also agrees with the affect detector on ASSIST2012 for 72.74% to 80.29% of questions, and feeding the simulated affect into the DKT and GKT models improves their AUC, which the paper reads as evidence that the affect signal is accurate and transferable.","pith_inferences":["Editorial inference: The reported gains may shrink if affective factors are recomputed causally, since the current formulas use whole-sequence averages and therefore include information from the outcome being predicted; a strictly causal version would separate true affect signal from leakage.","Editorial inference: The same confidence–interest–effort pipeline could be tested in other education tasks without affect labels, such as cognitive diagnosis or dropout prediction, though the paper only evaluates knowledge tracing.","Editorial inference: Segment length (LSeg) and the number of affect clusters are free parameters, so a sensitivity analysis would show how much of the gain is robust and how much depends on these choices.","Editorial inference: Because the affect cluster centers are learned from the training population, deploying DASKT on a new platform or school would likely require recalibration of the affect states to that population."],"forward_implications":["Affect simulation can run at scale from logs alone, so intelligent tutoring systems without affect labels or sensors can still incorporate affect into their student models.","Knowledge-state estimates become affect-sensitive: the paper shows positive affect raises mastery trajectories and negative affect lowers them, which makes predicted mastery more interpretable and opens the door to affect-targeted interventions.","The affect component transfers across architectures: feeding the simulated affect into DKT and GKT raises their AUC on ASSIST2012, indicating it is not tied to a single model design.","Long-sequence prediction benefits most, since the largest reported gain is on ASSISTchall, a longitudinal dataset, suggesting affect matters more when learning histories are long.","Without the affect modules the model's prediction quality drops (AUC from 0.7925 to 0.7404 on ASSIST2012 in the full ablation), so the affect pathway is load-bearing for the reported performance."],"supporting_citations":[{"why":"Supplies the four affect states (frustration, concentration, boredom, confusion) and the affect detector used as the consistency baseline for the simulated affect.","marker":"[35]"},{"why":"Provides the base LSTM sequence model that DASKT extends and serves as a primary baseline in all experiments.","marker":"[4]"},{"why":"Provides the interval-based dynamic student classification idea that the interval computation of affect (ICA) adapts to affect labels.","marker":"[5]"},{"why":"Serves as a learning-process-consistent knowledge tracing baseline without affect, against which DASKT's process modeling is compared.","marker":"[2]"},{"why":"Provides the graph attention network aggregation used to model affect coherence on the affect trajectory graph.","marker":"[54]"},{"why":"Proposes the confidence-interest-effort framework of affective factors that the MAF module computes from behavior.","marker":"[40]"},{"why":"Grounds the practice of deriving affective factors from observable behavior in educational settings.","marker":"[41]"},{"why":"Motivates the accuracy-based confidence measure used in the affective factor formulas.","marker":"[42]"},{"why":"Grounds the response-time component of confidence through the established link between response time and diagnostic accuracy.","marker":"[43]"},{"why":"Motivates using number of attempts and inter-question intervals as the effort factor.","marker":"[45]"}],"fun_headline_variants":["Affect from logs: better KT without sensors or surveys","Simulated student affect from answer logs beats 12 KT baselines","No affect labels? This KT model simulates them and wins","Clickstreams to emotions: DASKT lifts KT performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the affect features used to predict an answer are honest predictors, even though they are computed from statistics over the student's entire exercise history, including the answer being predicted; recomputing them from only past interactions might erase the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Affect from logs: better KT without sensors or surveys","Simulated student affect from answer logs beats 12 KT baselines","No affect labels? This KT model simulates them and wins","Clickstreams to emotions: DASKT lifts KT performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1693,"prompt_tokens":947,"completion_tokens":746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":675}},"tokens_in":563,"tokens_out":746,"duration_ms":7703,"temperature":1.0,"reasoning_tokens":675,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:01:55.626454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute every affective factor (confidence, interest, effort) using only interactions strictly before the predicted time, retrain DASKT on the same data splits, and compare AUC; if the advantage disappears, the reported gains come from information leakage rather than from modeling affect.","supporting_citations":[{"cited_title":"Classical test theory,","cited_arxiv_id":null,"evidence_quote":"Motivates the accuracy-based confidence measure used in the affective factor formulas."},{"cited_title":"Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes,","cited_arxiv_id":null,"evidence_quote":"Supplies the four affect states (frustration, concentration, boredom, confusion) and the affect detector used as the consistency baseline for the simulated affect."},{"cited_title":"Deep knowledge tracing and dynamic student classification for knowl- edge tracing,","cited_arxiv_id":null,"evidence_quote":"Provides the interval-based dynamic student classification idea that the interval computation of affect (ICA) adapts to affect labels."},{"cited_title":"Learning process-consistent knowledge tracing,","cited_arxiv_id":null,"evidence_quote":"Serves as a learning-process-consistent knowledge tracing baseline without affect, against which DASKT's process modeling is compared."},{"cited_title":"Diagnosing and acting on student affect: the tutor’s perspective,","cited_arxiv_id":null,"evidence_quote":"Proposes the confidence-interest-effort framework of affective factors that the MAF module computes from behavior."},{"cited_title":"Knowledge elicitation methods for affect modelling in education,","cited_arxiv_id":null,"evidence_quote":"Grounds the practice of deriving affective factors from observable behavior in educational settings."},{"cited_title":"The relationship between response time and diagnostic accuracy,","cited_arxiv_id":null,"evidence_quote":"Grounds the response-time component of confidence through the established link between response time and diagnostic accuracy."},{"cited_title":"Effort-based tutoring: An empirical approach to intelligent tutoring,","cited_arxiv_id":null,"evidence_quote":"Motivates using number of attempts and inter-question intervals as the effort factor."}],"review_version":1}