{"id":"42bdebdd-09a8-41cd-89fc-3ddf886449ce","arxiv_id":"1908.02146","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"KQN predicts student correctness as the dot product of a student knowledge vector and a skill vector, and claims the distances between learned skill vectors reveal how related skills are.","lead":"This paper introduces KQN, a neural knowledge tracing model that predicts whether a student will answer a problem correctly by taking the dot product of a student knowledge vector and a skill vector. It claims better accuracy than earlier models on four educational datasets and proposes 'probabilistic skill similarity' for visualizing skill relationships.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy claim lacks a controlled comparison: baselines are cited values, not rerun; without paired significance, the 'outperforms all existing' assertion is unsupported.","rationale":"The paper's most consequential assertion is that KQN outperforms all existing KT models. That assertion is load-bearing because it is the first itemized claim in the abstract and the basis for the paper's headline contribution. The reader's weakest_assumption focuses on the representational premise that skills can be fixed unit vectors and knowledge a single vector in the same space; that is a legitimate modeling concern, but it would leave KQN as a plausible architecture even if the assumption is only approximate. The empirical comparison concern is more directly tied to the central claim: if the baselines were not run under identical conditions and no significance test was performed, the 'outperforms all' statement is not supported by the reported evidence. The paper does provide several runs for KQN and reports standard deviations, but no comparable variance information for the cited baselines. I am not suggesting any misrepresentation; the issue is that the evaluation protocol as reported is insufficient to sustain the comparative claim. Since the reader already returned CONDITIONAL and my concern is consistent with that verdict, I recommend no change to the verdict. A controlled re-evaluation would either strengthen the claim to ACCEPT or require the authors to soften the language to a qualified competitive result.","tokens_in":14106,"tokens_out":5079,"duration_ms":53507,"concrete_test":"Re-run all neural baselines (DKT, DKVMN, and DKT+KQN) using the exact same data splits, preprocessing, and early-stopping protocol as KQN, with at least 10 random seeds, and compute paired bootstrap confidence intervals for each AUC difference. Also release per-dataset hyperparameters and code. If any baseline is not significantly worse, or if DKT+KQN is significantly better on ASSIST2015, rewrite the abstract's 'outperforms all existing' claim as a qualified claim of competitive accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section 6.1 is that KQN outperforms all existing KT models on four public datasets. The evidence in Table 3 does not support this at the claimed strength. Section 5.2 states that KQN and DKT were implemented in TensorFlow and that data splits from the DKVMN repository were used, but the comparison numbers for DKT, DKVMN, BKT, and IRT variants are not produced by the same evaluation pipeline: they are cited from other papers [10, 21, 22, 26], with no error bars and no paired tests. The reported KQN advantages are roughly 0.34–0.75 AUC points, which is within the range of preprocessing and seed variability. In addition, the final per-dataset hyperparameters and d_opt values are not reported, so the reported KQN results are not independently reproducible. Finally, on ASSIST2015 the authors' own DKT+KQN variant slightly exceeds KQN (73.41 vs 73.40), so the headline claim is not even uniformly true within the paper's own experiments. The model may well be competitive, but the current evidence does not establish the strong comparative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Knowledge Query Network (KQN), a knowledge tracing model in which a student's knowledge state vector and skill vectors are embedded into the same d-dimensional space, and the logit of a correct response at time t+1 is the dot product of the knowledge state and the skill vector (Section 3.5). The knowledge state is produced by an RNN encoder and a linear map, and the skill vectors by an MLP with L2 normalization onto the positive orthant unit sphere (Sections 3.6 and 3.7). The authors claim that KQN outperforms all existing KT models on four datasets (Section 6.1), that the learned skill vectors support a novel 'probabilistic skill similarity' relating cosine/Euclidean distances to odds ratios (Section 4), that the knowledge interaction can be visualized, and that the model is stable across embedding dimensionalities (Section 6.4).","tokens_in":14361,"tokens_out":6594,"duration_ms":63956,"significance":"If the claims were fully supported, KQN would offer a rare combination of state-of-the-art prediction and interpretable skill relationships in a single model. The core idea of placing knowledge states and skills in one vector space with a dot-product interaction is clean and pedagogically appealing, and the sensitivity analysis using Mantel tests is a thoughtful robustness check. The algebraic identity in Section 4.2 is correct as a manipulation, and the use of four public datasets with shared splits is a good starting point. However, the two central contributions—the accuracy superiority and the probabilistic skill similarity interpretation—are not established by the evidence presented. The baseline comparison is uncontrolled, and the distance–odds-ratio relation is shown to depend on the knowledge state vector in a way that undermines the paper's interpretive claim. The model may well be competitive, but the manuscript currently overstates what is demonstrated.","major_comments":[{"comment":"The derivation of the squared log odds ratio is algebraically correct, but the conclusion that 'two skills are probabilistically similar if they are close' does not follow from it. The displayed chain ends with (log o1/o2)^2 = (KS·Δ_{1,2})^2 × 2 d_cosine(s1,s2), where Δ_{1,2} is the unit vector along s1−s2. The factor (KS·Δ_{1,2})^2 depends on the knowledge state vector KS, which varies across students and time steps. For any fixed pair of skill vectors, the odds ratio can be made arbitrarily large by aligning KS with Δ, and zero by making KS orthogonal to Δ, regardless of how far apart s1 and s2 are. Thus the pairwise distance between skill vectors alone does not determine or even bound the odds ratio, and the phrase 'relates the pairwise cosine and Euclidean distances ... to the odds ratios' in the abstract is misleading. To support the probabilistic skill similarity interpretation, the authors need to either define a quantity that marginalizes over KS or explicitly state and test conditions under which the KS-dependent factor can be ignored. This issue is load-bearing for contribution (3) and for the interpretability claim, and it should be addressed before publication.","section":"Section 4.2"},{"comment":"The headline claim that 'KQN outperforms all the existing KT models based on prediction accuracy' is not supported by the evidence. As stated in Section 5.2, the IRT, BKT, DKT, and DKVMN numbers are cited from other papers rather than produced by the same evaluation pipeline; no error bars are available for these baselines under the same preprocessing, no paired significance tests are reported, and the KQN advantages are roughly 0.34–0.75 AUC points, which is within typical run-to-run variability for KT benchmarks. Moreover, the paper's own DKT+KQN variant achieves 73.41% on ASSISTments 2015, slightly above KQN's 73.40%, so the unqualified 'outperforms all' phrasing is stronger than the paper's own results. To make the central accuracy claim credible, the authors should rerun the baselines with the same data splits, the same early-stopping and epoch protocols, and the same number of seeds, and report significance tests or confidence intervals. Without this, the empirical contribution is only 'competitive', not 'state-of-the-art'.","section":"Section 6.1, Table 3"},{"comment":"The motivation for the positive-orthant constraint on skill vectors is incomplete. The paper argues that this constraint 'assures that learning on one skill does not decrease learning on another', but this conclusion implicitly assumes that the knowledge state vector KS has nonnegative coordinates. In KQN, KS is computed as KS_t = W_h,KS h_t + b_h,KS (Section 3.6), with no positivity constraint, so its entries can be negative. With a negative coordinate in KS, an increase in one component of KS can decrease the dot product with a positive-orthant skill vector, causing the probability of a correct response for another skill to fall even when the skill vectors are orthogonal or similar in the cosine sense. The example in Section 3.1 uses only positive KS vectors and therefore does not establish the general claim. This affects the interpretability narrative that motivates the model architecture, and the authors should either constrain KS to the positive orthant as well, or modify the motivation to acknowledge that the monotonicity property is not guaranteed.","section":"Sections 3.1 and 3.5"}],"minor_comments":[{"comment":"In the sentence 'we used the same ones 1 used by DKVMN', there is a stray '1' that appears to be a typographical artifact; it should read 'the same ones used by DKVMN'.","section":"Section 5.2"},{"comment":"The text 'TensorFlow 1.56' should likely read 'TensorFlow 1.5', since the footnote references the TensorFlow 1.5 release page.","section":"Footnote 6"},{"comment":"For Statics2011, the text says 'higher than the previous highest by 0.34%', but 83.20 − 82.84 = 0.36 percentage points; the reported number should be corrected.","section":"Section 6.1"},{"comment":"The text says '30 Ordering Functions and 62 Ordering Real Numbers' were assigned to the fourth cluster, but Table 4 lists skill 30 as 'Ordering Fractions'; the skill name in the sentence should be corrected.","section":"Section 6.3"},{"comment":"The statement that p-values are omitted 'since they were 0.001 in all cases' is not a proper report of a permutation-based Mantel test. With 999 permutations, the minimum attainable p-value is 0.001, so reporting '0.001' for every test likely collapses different values. The authors should report the actual permutation p-values or use 'p < 0.001' if all are at the minimum.","section":"Table 9"},{"comment":"The optimal hyperparameters, including the optimal embedding dimensionality d_opt, are never reported. Since Section 6.4 defines d_opt and reports sensitivity results relative to it, omitting the actual values prevents independent reproduction of both the main accuracy results and the sensitivity analysis. Please include a table of the selected hyperparameters for each dataset.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The core model is reasonable and the topic is timely, but the two main claims are currently overreaching. The probabilistic skill similarity claim is mathematically incomplete as stated, and the accuracy comparison lacks a controlled baseline evaluation. Both issues are fixable within the scope of a revision: the first can be addressed by reframing the contribution or adding the necessary qualification/test, and the second by rerunning baselines and adding significance tests. I would be willing to review a revised version. I would also gently suggest that the authors release code or at least report hyperparameters, as the current level of detail makes reproduction difficult."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: KQN is a tidy architecture with one genuinely new piece—the positive-orthant unit-sphere skill embedding and dot-product query—and the interpretability story is a derived identity, not a discovery. The accuracy claim is real but over-sold.\n\nWhat's new: DKT and DKVMN already learn per-skill output vectors or key memories; KQN replaces the softmax layer with an explicit dot product between a single knowledge-state vector and a learned skill vector, with the skill constrained to the positive orthant. That constraint is a sensible inductive bias for skills in a single subject area. The probabilistic skill similarity is exactly the algebra in Section 4.2: the squared log-odds difference factors into (KS·Δ)^2 times the cosine distance. That is a tautology, but it does give a nice way to think about why close skill vectors behave similarly for a fixed KS, and the clustering experiments show the learned skill vectors are coherent.\n\nWhat the paper does well: the model description is clear; the authors use the same data splits as DKVMN, which allows some comparability; they report standard deviations for their own runs; the sensitivity analysis around dimensionality is a thoughtful stability check. The visualization in Figure 3 is honest enough to note a counter-intuitive case.\n\nWhere it gets soft: Table 3's baseline numbers are cited from other papers, not rerun in the same pipeline. No error bars, no significance tests. The margins are small—0.3 to 0.8 AUC points—well within seed and preprocessing noise. On ASSIST2015 the authors' own DKT+KQN variant beats KQN by 0.01, which undermines the abstract's claim that KQN outperforms all existing models. That specific sentence should be softened to \"competitive with or better than existing models.\" Also, final hyperparameters are not reported and code is not released, so the results are not independently reproducible. These are fixable with more careful evaluation and a public release.\n\nShould this get a serious referee? Yes. The architecture is worth knowing about, the experiments are extensive (four datasets, clustering, sensitivity), and the interpretability claim, while definitional, is still useful for a practitioner. The paper needs a stronger empirical section, not a redesign. I'd send it to review, but I would not accept the headline accuracy claim on current evidence.","headline":"KQN's dot-product query and positive-orthant skill vectors are a clean idea, but the 'outperforms all' claim rests on cited baselines and should be softened.","tokens_in":14923,"tokens_out":2086,"would_cite":false,"duration_ms":20990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single dot product between knowledge and skill vectors outperforms all existing knowledge tracing models and reveals a probabilistic skill similarity.","keywords":["Knowledge Tracing","Knowledge Query Network","Neural Networks","Skill Vectors","Probabilistic Skill Similarity","Knowledge State","Interpretability","Learning Analytics"],"falsifier":"Construct a synthetic dataset with two skills that are negatively correlated, so that answering one correctly makes answering the other less likely; because KQN's skill vectors are confined to the positive orthant, any increase in the knowledge state along one skill vector must weakly increase the dot product with every other skill vector, so the model cannot represent a learning-induced drop in the second skill's probability. If, on such data, KQN's predictions for the second skill remain near chance or the model compensates in ways that break the distance-to-odds-ratio identity, the claim of a descriptive knowledge interaction would be falsified.","tokens_in":13830,"feed_emoji":"🧠","tokens_out":8360,"duration_ms":72470,"temperature":0.7,"pith_summary":"The paper introduces Knowledge Query Network (KQN), a knowledge tracing model that places a student's evolving knowledge state and each skill into the same d-dimensional vector space and predicts the probability of a correct answer as the logistic function of their dot product. The central claim is that this simple interaction outperforms all existing knowledge tracing models on four public datasets in prediction accuracy, while also making the knowledge interaction interpretable. KQN learns a fixed unit vector for each skill, and the geometry of these skill vectors yields a 'probabilistic skill similarity': the squared Euclidean distance between two skill vectors equals twice their cosine distance and is directly proportional to the squared log odds ratio of the two skills. The paper further shows that the learned skill relationships are stable when the embedding dimensionality changes, and that clustering the skill vectors preserves predictive information.","feed_headline":"A dot product between knowledge and skill vectors tops prior models","feed_subtitle":"The same geometric interaction lets clusters of skills be read directly from distance matrices.","key_machinery":"The load-bearing object is the knowledge state query: at each time step, a recurrent network encodes the student's response history into a knowledge-state vector $\\mathrm{KS}_t \\in \\mathbb{R}^d$, a multi-layer perceptron embeds the next skill into a vector $s_{t+1}$ on the positive-orthant unit sphere $U_d = \\{v \\in \\mathbb{R}^d : \\|v\\|=1, v_i \\ge 0\\}$, and the logit of a correct response is $y_{t+1} = \\mathrm{KS}_t \\cdot s_{t+1}$. This one inner product does three jobs: it defines the knowledge interaction, it guarantees (via the positive-orthant constraint) that learning one skill does not decrease the predicted probability for any other skill, and because all skill vectors have unit length, it makes the squared Euclidean distance between two skill vectors equal to twice their cosine distance. That identity plus the linearity of the dot product yields the paper's probabilistic skill similarity: the squared difference of log odds for two skills equals $\\bigl(\\mathrm{KS} \\cdot \\Delta_{1,2}\\bigr)^2$ times their squared Euclidean distance, where $\\Delta_{1,2}$ is the normalized difference vector. This is what makes the learned skill vectors interpretable and useful for domain analysis such as clustering.","core_discovery":"KQN's central claim is that a student's knowledge state and the skills being learned can be embedded in the same d-dimensional vector space, with the logit of the probability of a correct answer equal to the dot product of the knowledge-state vector and the skill vector. The skill vectors are constrained to the positive-orthant unit sphere, guaranteeing that learning one skill never decreases the predicted probability for another skill within the same subject area. The paper argues that this simple interaction both improves prediction accuracy over prior neural and non-neural knowledge tracing models on four datasets and yields interpretable skill representations, because the squared Euclidean distance between two skill vectors equals twice their cosine distance and the squared log-odds-ratio of two skills equals a projection-dependent factor times that distance. This makes pairwise skill distances meaningful for domain analysis, such as clustering skills into groups that preserve predictive information.","pith_inferences":["The distance-to-odds-ratio identity suggests a concrete diagnostic: by computing $(\\mathrm{KS} \\cdot \\Delta_{1,2})^2$ for different students, one can check whether the relative order of skill distances depends on the knowledge state; the paper leaves this dependence implicit, but a dataset with diverse students could reveal whether skill geometry is truly fixed or student-dependent.","The positive-orthant constraint is a substantive cognitive assumption: it rules out negative transfer between skills by construction. Applying KQN to a multi-domain dataset (e.g., math and reading) would put that assumption to the test; if the model needs to represent that learning one skill reduces another, the constraint would have to be relaxed.","The paper's stability analysis suggests a practical rule of thumb: the skill distance matrix is roughly invariant over a wide range of $d$, so practitioners can choose a small embedding dimension for computational efficiency without losing much skill-structure information."],"forward_implications":["KQN's prediction accuracy exceeds that of all previously published knowledge tracing models on the four evaluated datasets, with the lowest standard deviations, indicating that the dot-product interaction yields stable predictions.","The learned skill vectors are transferable: feeding them to DKT instead of one-hot encodings improves DKT's accuracy on most datasets, showing that the vectors capture reusable skill structure.","The probabilistic skill similarity makes skill-domain analysis possible: clustering the skill vectors with average linkage and Euclidean distance recovers groups of related skills, and replacing skill IDs with cluster labels in DKT costs little prediction accuracy.","KQN's output is robust to the embedding dimensionality: varying d between half and twice the optimal value changes test AUC by less than 0.05% on all datasets, and the Mantel correlation between the skill distance matrices stays above 0.5, confirming strong positive correlation."],"supporting_citations":[{"why":"Supplies the DKT baseline model and the Synthetic-5 dataset used for evaluation.","marker":"[18]"},{"why":"Supplies the DKVMN baseline, the previous state-of-the-art neural KT model, and the data splits used in the experiments.","marker":"[26]"},{"why":"Supplies the Bayesian Knowledge Tracing baseline with binary knowledge states.","marker":"[1]"},{"why":"Supplies the ASSISTments dataset source used in two of the four evaluation datasets.","marker":"[4]"},{"why":"Supplies the IRT model and the generative process used to create Synthetic-5.","marker":"[8]"},{"why":"Supplies the IRT+ baseline accuracy numbers that KQN is compared against on ASSISTments 2009-2010.","marker":"[21]"},{"why":"Supplies the DKT baseline with dropout, whose accuracy is used in the comparison.","marker":"[22]"},{"why":"Supplies baseline results from BKT variants that KQN is compared against.","marker":"[10]"}],"fun_headline_variants":["Dot product of knowledge and skill vectors boosts KT accuracy","KQN: dot product interactions make skill clusters interpretable","Knowledge Query Network maps skills via vector distances","Skill vectors and knowledge states: a dot product for tracing","KQN embeds skills and knowledge, clustering from distances"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes that each skill can be pinned down by a single fixed nonnegative unit vector and that a student's entire knowledge at each step can be summarized by one vector in that same space, so that the logit of correctness equals their dot product; if skills are not fixed in such a space or knowledge cannot be compressed to one vector, the accuracy and interpretability claims would not survive.","fun_headline_variants_meta":{"raw":{"variants":["Dot product of knowledge and skill vectors boosts KT accuracy","KQN: dot product interactions make skill clusters interpretable","Knowledge Query Network maps skills via vector distances","Skill vectors and knowledge states: a dot product for tracing","KQN embeds skills and knowledge, clustering from distances"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2717,"prompt_tokens":976,"completion_tokens":1741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1662}},"tokens_in":592,"tokens_out":1741,"duration_ms":11331,"temperature":1.0,"reasoning_tokens":1662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:21:29.825903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic dataset with two skills that are negatively correlated, so that answering one correctly makes answering the other less likely; because KQN's skill vectors are confined to the positive orthant, any increase in the knowledge state along one skill vector must weakly increase the dot product with every other skill vector, so the model cannot represent a learning-induced drop in the second skill's probability. If, on such data, KQN's predictions for the second skill remain near chance or the model compensates in ways that break the distance-to-odds-ratio identity, the claim of a descriptive knowledge interaction would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DKT baseline model and the Synthetic-5 dataset used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DKVMN baseline, the previous state-of-the-art neural KT model, and the data splits used in the experiments."},{"cited_title":"Anderson, C","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian Knowledge Tracing baseline with binary knowledge states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ASSISTments dataset source used in two of the four evaluation datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IRT model and the generative process used to create Synthetic-5."},{"cited_title":"Back to the Basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation","cited_arxiv_id":"1604.02336","evidence_quote":"Supplies the IRT+ baseline accuracy numbers that KQN is compared against on ASSISTments 2009-2010."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DKT baseline with dropout, whose accuracy is used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies baseline results from BKT variants that KQN is compared against."}],"review_version":1}