{"id":"d2b3b143-2e7c-4c46-98ef-0d6408b3c98f","arxiv_id":"2611.16323","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Semantic text representations of questions do not strongly recover the relation structure learned by Deep Knowledge Tracing, indicating DKT's relations are not primarily semantic.","lead":"This paper tests whether the relations between knowledge components learned by Deep Knowledge Tracing (DKT) are semantic, i.e., based on the text of the questions. It finds that text-derived semantic representations align only weakly with DKT's learned relations and do not consistently improve prediction accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"DKT relation graph stability is unquantified; weak alignment may reflect noise in the target rather than absence of semantic content.","rationale":"The reader's weakest_assumption correctly identifies that the stability of the DKT relation graph is unquantified. This is the most load-bearing concern because the central claim depends on the DKT graph being a valid and stable representation of the relations DKT uses. The paper acknowledges this gap (b138, b389) but does not address it. The concrete test (retraining with different seeds and computing pairwise correlations) would directly assess stability. If the DKT graph is unstable, the weak alignment with semantic structures could be an artifact, weakening the central claim. The reader's verdict is ACCEPT with high confidence, but this concern suggests that the central claim is conditional on the stability of the DKT graph. Therefore, the verdict should be CONDITIONAL, pending the proposed stability check. The agreement is 'agree' because the reader identified the same weakest assumption.","tokens_in":22774,"tokens_out":548,"duration_ms":6299,"concrete_test":"Retrain the same DKT model (same data, same hyperparameters) 5–10 times with different random seeds. For each retraining, extract the DKT relation matrix using the influence-score method. Compute the pairwise Pearson correlations between these matrices. If the mean correlation is low (e.g., <0.5), the DKT graph is not stable and the weak alignment with semantic structures may be due to noise in the target. If the mean correlation is high (e.g., >0.8), then the weak alignment is more likely a genuine property of DKT's learned relations.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (DKT relations are not primarily semantic) rests on comparing DKT-derived relation graphs with text-derived semantic graphs. The paper explicitly states it does not estimate the stability of the DKT relation matrices across retraining runs (b138, b389). If the DKT graph is itself noisy or not reproducible, then the weak alignment with semantic structures may reflect instability in the target rather than a genuine lack of semantic content. The influence-score method (J_ij = y(j|i) / sum_k y(j|k)) is a specific procedure that may not capture all aspects of DKT's learned relations. The paper acknowledges this limitation but does not address it, making the central claim less secure. Without a stability estimate, the observed weak alignment could be an artifact of the extraction method rather than a property of DKT's learned representations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper asks whether the KC relations learned by Deep Knowledge Tracing (DKT) are semantic. Using three ASSISTments benchmarks restricted to text-linked subsets, the authors construct KC-level semantic representations through nine pipelines (varying KC construction, response-state encoding, and embedding backbone, including fine-tuned BERT), then run two complementary tests: (RQ1) graph alignment between the DKT-derived relation matrix (via the original Piech et al. influence-score procedure, J_ij = y(j|i)/sum_k y(j|k)) and the cosine-similarity structure induced by text embeddings; and (RQ2) predictive comparison of semantic-initialized DKT against the pyKT baseline in full-data, reduced-data, and KC-supervised-encoder settings. RQ1 reports only weak and inconsistent alignment (best Pearson 0.0894 on ASSISTments 2012, 0.0573 on 2009, 0.0247 on 2017). RQ2.1 finds no validation-selected semantic configuration exceeding baseline test AUC; RQ2.2 finds a favorable ASSISTments 2017 trend (8/9 positive splits) but weak or unstable effects on 2009 and 2012; fine-tuned BERT gives small, not-stable-gain AUC differences. The paper concludes that DKT-derived relations are not primarily semantic under the tested text-derived semantic structures and graph-extraction procedure, while explicitly scoping the claim and acknowledging unquantified target stability.","tokens_in":22974,"tokens_out":5538,"duration_ms":36530,"significance":"If the result holds, it is a useful negative result for the KT community: it constrains the common intuition that question-text semantics should recover DKT's latent KC dependencies or improve prediction by replacing the embedding table. The paper is methodologically careful in ways that deserve explicit credit: (i) the nine-pipeline design plus multiple embedding backbones tests the robustness of the core comparison rather than a single representation choice; (ii) the training-sequence co-exposure comparator is a well-chosen non-text reference that largely recovers the DKT graph on two of three datasets, indicating the DKT graph carries structure that text semantics fails to capture; (iii) the graph comparison respects the dataset-specific KC ordering (keyid2idx.json); and (iv) the claims are consistently and explicitly scoped to the influence-score extraction procedure, the text-linked subsets, and the evaluated pipelines, with descriptive rather than inferential framing.","major_comments":[{"comment":"The RQ1 conclusion is explicitly comparative and appropriately hedged, but its principal evidence — the weak alignment between text-semantic and DKT-derived graphs — is uncalibrated because the reproducibility of the DKT relation matrix is not measured. The paper states this (b138, b389), and the co-exposure comparator partially mitigates the concern on ASSISTments 2009 and 2012, where it aligns more strongly than the best semantic configuration. However, on ASSISTments 2017 that comparator is negative (Pearson -0.0322, Spearman -0.0469, §4.1), so on that dataset weak semantic alignment is as consistent with an unreproducible or underdetermined target graph (compounded by the small 638-question text pool) as with a genuinely non-semantic DKT relation structure. The paper's central claim remains defensible because it is scoped to the tested procedure and supported by the independent RQ2 tests, but a cheap stability probe — retraining the baseline DKT with, say, three to five seeds and reporting graph-to-graph Pearson/Spearman correlations and top-k overlap against the reported matrix — would materially strengthen the central claim by distinguishing target instability from absent semantic content. I recommend adding this as a strengthening request rather than treating it as a defect.","section":"§3.4, §5.1","authors_acknowledge":"We do not estimate how stable the extracted DKT relation matrices are across retraining runs, so the observed alignment magnitudes are not calibrated against the reproducibility of the DKT-derived graph itself.","authors_acknowledge_p":0.95},{"comment":"The selection protocol for the full-data RQ2.1 sweep reports the validation-selected configuration per condition, and the paper correctly notes the semantic rows are maxima selected by validation AUC while the baseline is a single run family. This asymmetry is acknowledged and acceptable for a descriptive comparison. What would strengthen the presentation is one sentence quantifying the spread of test AUC across the evaluated configurations per dataset (e.g., range or standard deviation across validation-selected alternatives), since most reported AUC deltas are within 0.001–0.005 of the baseline and the study explicitly does not quantify run-to-run variation (b381). Without such context, the reader cannot judge whether the reported shortfalls are within configuration noise; this is a presentation point, not a correctness issue.","section":"Table 5 / §4.2","authors_acknowledge":"The study does not quantify run-to-run variation for those selected full- data runs or variability across alternative reduced-data samples, so the favorable values should not be read as statistically established effects.","authors_acknowledge_p":0.76}],"minor_comments":[{"comment":"Under ASSISTments 2017, two rows are both labeled 'Semantic Output Layer' with the same embedding model (text-embedding-3-large) and Pipeline 1, but report different test AUC (0.7195 and 0.7193) and test accuracy (0.6785 and 0.6798). This appears to be a duplicated or mislabeled export; please clarify what distinguishes the two rows (or merge them).","section":"Table 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The semantic RQ1 graph is built from the 'first N rows of each 2N-row pretrained interaction tensor.' For the mirror pipelines this yields the base KC vector, but for the stack-antonym pipelines the first N rows are the doubly-dimensional concatenated vectors including a response marker. Please state explicitly, for each of the three response-state families, which response-specific vector (correct vs. incorrect) the first N rows correspond to and how the response-marker segment enters the cosine comparison; this determines the semantic structure being compared against the DKT graph.","section":"§3.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Typos in the title ('Knowledge T racing') and affiliation line ('T echnology') should be fixed.","section":"Title and author line","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The ASSISTments 2017 pretrained-initialization export (S16) has only 113 usable runs out of 796 exported. Given that this dataset drives the favorable reduced-data trend, please add a sentence on why so few runs completed and whether the low pass rate could bias the validation-selected configuration.","section":"Table 13","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Kia,\n\nThis paper asks a clean question—are the KC relations DKT learns semantic?—and answers it with a systematic empirical campaign. That alone is a service to the subfield. The nine-pipeline design, the use of multiple embedding models, and the two complementary tests (graph alignment and initialization) are well thought out. The paper is honest about its limitations, especially the incomplete text coverage and the fact that the DKT relation graph's own stability is unquantified. The writing is direct and the claims are appropriately bounded.\n\nThe central result—weak and inconsistent alignment between text-derived semantic graphs and DKT-derived influence-score graphs—is credible as a descriptive finding. The correlations are tiny (Pearson 0.02–0.09), and the pattern holds across datasets and pipelines. The reduced-data experiments on ASSISTments 2017 show a hint of a sample-efficiency benefit, but the authors themselves note it's not stable across datasets. The KC-supervised BERT analysis is a nice mechanism check, and the fact that even near-perfect KC classification (2009) yields only a 0.0029 AUC gain reinforces the main conclusion.\n\nNow the soft spot, and it's the one the stress-test note flags: the target of the comparison is itself uncalibrated. The DKT relation graph is extracted via the original influence-score method, but the paper explicitly states it does not estimate the stability of these matrices across retraining runs (b138, b389). If the DKT graph is noisy or not reproducible, then weak alignment could reflect instability in the target rather than absence of semantic content. The authors acknowledge this, but they don't address it. A simple stability estimate—even a small-scale retraining check—would substantially strengthen the claim. Without it, the central argument has a load-bearing gap.\n\nThat said, the paper is still a solid empirical contribution. It's for researchers working on interpretability of KT models or on incorporating side information into DKT. The negative result is useful: it redirects attention toward behavioral, temporal, and structural signals. The paper deserves a serious referee. I'd recommend acceptance with a request for a stability analysis of the DKT relation graphs, or at least a more explicit discussion of how this gap affects the interpretation.\n\nBest,\n[Your name]","headline":"A careful negative result: DKT's learned KC relations are not primarily semantic, but the central comparison is weakened by unquantified stability of the DKT relation graph itself.","tokens_in":23524,"tokens_out":570,"duration_ms":6778},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"DKT does not learn primarily semantic relations between knowledge components","keywords":["deep knowledge tracing","knowledge tracing","semantic embeddings","knowledge components","question text","influence score","graph alignment","ASSISTments"],"falsifier":"If a future study demonstrated that a text-derived KC graph (e.g., from a fine-tuned educational language model) achieved a Pearson correlation >0.3 with the DKT-derived graph on the same dataset, or if semantic initialization consistently improved DKT's test AUC by >0.01 across multiple datasets and runs with quantified stability, the paper's central claim that DKT relations are not primarily semantic would be contradicted.","tokens_in":24478,"feed_emoji":"📚","tokens_out":1657,"duration_ms":15345,"temperature":0.7,"pith_summary":"This paper tests whether the relations between knowledge components (KCs) learned by Deep Knowledge Tracing (DKT) are semantic—that is, whether they align with similarity patterns captured by text embeddings of the associated questions. The authors construct nine pipelines to produce text-derived KC representations and compare these to the KC relation graph extracted from DKT via the original influence-score method. Across three ASSISTments datasets, the alignment is weak and inconsistent (Pearson correlations 0.0247–0.0894). The paper also finds that replacing DKT's learned interaction embeddings with these pretrained semantic embeddings does not reliably improve prediction accuracy. The central claim is that DKT-derived relations are not primarily semantic; the results constrain what text similarity alone can explain in knowledge tracing.","feed_headline":"DKT's KC relations are not primarily semantic","feed_subtitle":"Text-derived embeddings align only weakly with DKT's learned relation graph across three datasets.","key_machinery":"The central object is the DKT-derived KC relation graph, extracted via the original influence-score method: for each source–target KC pair (i,j), J_ij = y(j|i) / sum_k y(j|k), where y(j|i) is the predicted correctness probability for target KC j given a correct response to source KC i. This directed weighted graph is symmetrized and compared to semantic KC graphs built from pairwise cosine similarities of text-derived KC embeddings. The nine semantic pipelines vary in how question text is embedded (e.g., sentence transformers, OpenAI, Cohere, fine-tuned BERT), how multiple questions per KC are aggregated (averaging, sampling, stuffing), and how response correctness (correct/incorrect) is incorporated (mirror polarity, stack antonym, add words).","core_discovery":"The paper establishes that the KC relation structure extracted from DKT by the original influence-score procedure aligns only weakly with semantic structures derived from question text via nine different embedding pipelines. The highest Pearson correlation between DKT-derived and semantic KC graphs is 0.0894 (ASSISTments 2012), and the alignment varies across datasets, embedding models, and pipeline choices. Furthermore, initializing DKT with pretrained semantic embeddings does not consistently outperform the standard randomly initialized baseline in either full-data or reduced-data settings. The authors conclude that DKT's learned relations are not primarily semantic, suggesting that behavioral, temporal, structural, or psychometric signals may play a larger role.","pith_inferences":["The paper's reliance on the original influence-score method for extracting DKT relations is itself a choice; alternative extraction methods (e.g., the repeated-response method proposed in Hong et al. 2025) might yield different relation structures and potentially different alignment results. The authors acknowledge this limitation but do not explore its impact.","The nine semantic pipelines, while diverse, all treat questions as static text. In real classrooms, the semantics of a question may be mediated by its position in a curriculum, the teacher's instruction, or the student's prior exposure—factors not captured by text embeddings. A testable extension would be to incorporate contextualized embeddings that account for sequence-level or student-level met","The paper's negative result has a positive implication: it suggests that DKT's predictive power comes from patterns that are not easily captured by off-the-shelf language models. This opens the door for developing KT-specific representation learning that jointly models text, behavior, and structure, rather than relying on transfer from NLP.","The authors do not estimate the stability of the DKT relation matrix across retraining runs. If the matrix is highly variable, the weak alignment could be due to noise in the target rather than a genuine absence of semantic content. A natural follow-up would be to measure the run-to-run reproducibility of the influence-score graph and calibrate alignment magnitudes against that baseline."],"forward_implications":["If DKT's KC relations are not primarily semantic, then architectures that explicitly model behavioral co-occurrence, temporal order, or prerequisite structure may be more appropriate than those relying solely on text similarity.","The weak alignment suggests that text-derived embeddings alone are unlikely to serve as a strong inductive bias for DKT, even in data-sparse regimes where semantic priors might seem beneficial.","The finding that a simple KC co-exposure matrix (based on training sequences) aligns more strongly with DKT's relations than any text-derived graph on two of three datasets implies that behavioral patterns carry more weight than semantic content in DKT's learned representations.","The study's methodology—comparing DKT influence-score graphs to cosine-similarity graphs—provides a general diagnostic framework for probing what kinds of structure KT models actually learn.","Future KT work that incorporates question text may need to combine it with other signals (difficulty, temporal context, curriculum structure) rather than using text as a standalone input."],"supporting_citations":[{"why":"Introduces the DKT model and the influence-score method for extracting KC relation graphs; the paper's central comparison target is built using this method.","marker":"Piech et al., 2015"},{"why":"Provides BERT, the base model for the fine-tuned KC classifiers used in the semantic pipelines and for general-purpose embeddings.","marker":"Devlin et al., 2019"},{"why":"Provides Sentence-BERT, the backbone for several of the sentence-transformer embedding models used in the nine pipelines.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides SimCSE, a contrastive learning method for sentence embeddings; cited as a modern text representation method relevant to KC embedding.","marker":"Gao et al., 2021"},{"why":"Provides pyKT, the library whose DKT implementation is used as the baseline and modified for semantic initialization.","marker":"Liu et al., 2023"},{"why":"Shows that for KC imputation, contextual information from neighboring problems can be as predictive as problem text alone; used to motivate why the semantic hypothesis might fail.","marker":"Pardos and Dadu, 2017"},{"why":"Introduces Context-Aware Attentive Knowledge Tracing, cited as an example of a later KT model that couples question information with additional signals beyond text.","marker":"Ghosh et al., 2020"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The DKT-derived KC relation graph, extracted via the original influence-score method (J_ij = y(j|i) / sum_k y(j|k)), is a valid and stable representation of the relations DKT actually uses for prediction; the paper acknowledges it does not estimate the stability of these matrices across retraining runs.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.2e-05,"raw_usage":{"cost":9.1546e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.1546e-05,"upstream_inference_prompt_cost":9.1182e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":1004,"prompt_tokens":1002,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1002,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":2263,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T06:38:51.441614+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a future study demonstrated that a text-derived KC graph (e.g., from a fine-tuned educational language model) achieved a Pearson correlation >0.3 with the DKT-derived graph on the same dataset, or if semantic initialization consistently improved DKT's test AUC by >0.01 across multiple datasets and runs with quantified stability, the paper's central claim that DKT relations are not primarily semantic would be contradicted.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}