{"id":"8ed36710-5f63-4e92-a759-622beca750f3","arxiv_id":"2501.13943","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LRCD turns response logs into text embeddings and maps them into a cognitive diagnosis space, giving strong accuracy on new subjects and platforms only when target training logs are available at inference.","lead":"This paper introduces LRCD, which lets a cognitive diagnosis model trained on one school subject or platform assess students on a different subject without retraining the model. It converts logs into text embeddings and learns a small mapper, but still needs labeled logs from the new subject at inference time, so 'zero-shot' overstates what the experiments show.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target training labels are embedded in the student and exercise profiles used at inference, so the reported 'zero-shot' results do not test the paper's central claim.","rationale":"The central claim is that LRCD performs zero-shot cognitive diagnosis with no target-domain information. My strongest concern is that this claim is falsified by the evaluation protocol itself: target training labels are used to construct the very profiles that are fed to the frozen mappers at inference. The reader identified this same issue as a second load-bearing premise, but their stated weakest assumption also emphasizes domain invariance of the language-to-cognitive mapper. I regard the target-label protocol violation as the decisive problem because even a perfect domain-invariant mapper would not justify a zero-shot claim if target labels are used as input features. A secondary but related leak is that, during source training, the student representation for an interaction includes the label being predicted for that interaction, so the mapper can shortcut by copying the label token. The reported numbers therefore support a transductive, label-conditioned model, not an unsupervised zero-shot transfer model. I agree with the reader's REJECT verdict: the central claim, as stated, is not supported by the experiments. A resubmission that removes labels from profiles, retrains under a strict no-target-label protocol, and re-runs the ablation would be needed before the claim could be credited.","tokens_in":25191,"tokens_out":3728,"duration_ms":44142,"concrete_test":"Re-run the PB-M subject-level experiment under a strict no-target-label protocol: build target student profiles from Eq. 3 with y_ij removed, and replace each target exercise's ACR_e_j with a fixed constant or a source-domain average, while keeping the target test labels untouched. Also retrain the source model with y_ij excluded from the source student profile. If AUC stays within a small margin of the reported 80.23 (vs. Random's 50.03), the method genuinely transfers; if it collapses toward 50, the target labels in the profiles are doing the work and the zero-shot claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3 defines ZSCD as having no overlapping students, exercises, or concepts and no information about the target domain. Section 4.3 implements inference by \"tackl[ing] the response logs in the train data of target domains into textual cognitive profiles.\" Those profiles are not label-free: Eq. 2 puts ACR_e_j, the average target-train correctness of the exercise, into the exercise profile, and Eq. 3 puts y_ij, the student's actual score on that exercise, into the student profile; Eq. 4 then mean-pools the student's target-train interactions. Thus at zero-shot inference the model receives target-domain labels as input features. This makes the method zero-training/transductive rather than zero-shot, so the high AUCs in Tables 2-3 cannot be read as evidence that a source-trained mapper generalizes to unseen target domains.\n\nThe same label also appears in source training: the student representation used to predict y_ij is built from an interaction text that includes y_ij (Eqs. 3-4). After the student mapper, the model can exploit that token directly, so the mapper may be learning to copy labels rather than a domain-invariant cognitive mapping. The paper's statement that target training labels are unavailable \"during the training phase\" conflates not training on target labels with not using them at all; the problem definition requires the latter. The LRCD-w/o-TCP ablation, which reports AUC 75-80 after replacing textual profiles with random normal vectors, is also implausible and suggests the ablation does not actually remove the information the paper claims to remove, though the target-label protocol violation is decisive on its own.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LRCD, a framework for what it calls zero-shot cross-domain cognitive diagnosis (ZSCD). LRCD converts students, exercises, and concepts into textual profiles derived from response logs, embeds these profiles with a text-embedding model, learns per-entity language-to-cognitive mappers on source domains, and then applies the frozen mappers to a target domain. The paper reports AUC and DOA values showing that LRCD approaches or matches classic CDMs trained on full target-domain data, and it claims that LRCD can transfer across subjects and platforms. The central technical idea—using text embeddings as a shared representation across domains—is plausible and the authors provide a code release and ten-seed experiments. However, the experimental protocol contradicts the paper's own zero-shot definition because target-domain training labels are embedded in the profiles used at inference, and the source-training student representation includes the label of the interaction being predicted. These issues are load-bearing for the paper's headline claim.","tokens_in":25440,"tokens_out":4421,"duration_ms":46503,"significance":"If the reported results were valid, the paper would make a significant practical contribution: a single model trained on source domains could diagnose students in arbitrary new subjects or platforms without target-domain training. The use of textual cognitive profiles and learned language-cognitive mappers is a reasonable way to seek a domain-invariant representation, and integrating the approach with existing CDMs is a useful practical design. The authors also share code and report multiple random seeds, which are good practices. However, the significance is undermined by two flaws: target-domain labels are used at inference despite the stated zero-shot setting, and the source-training objective can exploit the label token embedded in the student profile. The ablation study further reports implausibly high performance when all textual profiles are replaced by random vectors. As a result, the experiments do not establish that a source-trained mapper generalizes to unseen target domains, and the central claim is not supported.","major_comments":[{"comment":"The inference procedure violates the paper's own definition of zero-shot cross-domain cognitive diagnosis. Section 3 defines ZSCD as having 'no overlapping students, exercises and concepts' and 'lacks any information about the target domain.' Section 4.3, however, states that at zero-shot inference the model 'tackle[s] the response logs in the train data of target domains into textual cognitive profiles.' Equation (2) puts ACR_ej, the average target-train correctness, into the exercise profile, and Eq. (3) puts y_ij, the student's actual score, into the student profile; Eq. (4) then mean-pools the student's target-train interactions. Section 5.1 confirms that 'we use the training data in the target domain to infer the students' mastery levels and evaluate the results on the test data.' Thus the reported AUCs in Tables 2 and 3 are obtained with target-domain labels available as input features; they are transductive or target-supervised results, not zero-shot results. This invalidates the abstract's claim that 'training LRCD on most real-world datasets can achieve commendable zero-shot performance.'","section":"Section 3 vs. Section 4.3 / Section 5.1, Eqs. (2)-(4)"},{"comment":"During source training, the student representation used to predict y_ij includes y_ij itself. Equation (3) defines each interaction profile I_ij as the text '[{c_k | Q_j,k = 1}, ACR_ej, y_ij]', and Eq. (4) sets the student's language-space vector h_si to the mean of the text embeddings of all of that student's interactions. When the model predicts y_ij for a training interaction, that same y_ij token is present in the mean-pooled student representation. Consequently, the student mapper can learn to detect and copy the label token rather than to compute a domain-invariant cognitive mapping. The paper provides no ablation or analysis that removes the label token from the student profile to show that the reported cross-domain performance is not driven by this shortcut.","section":"Section 4.1, Eqs. (3)-(4)"},{"comment":"The ablation LRCD-w/o-TCP is internally inconsistent with the claim that textual cognitive profiles are essential. This variant replaces every textual profile with a random vector sampled from a standard normal distribution, yet it still achieves AUC values of 75.74, 80.67, and 80.38 on PB-M, PB-C, and EHG-C, respectively, far above the 50% random baseline and close to the full LRCD. A model with random, information-free student, exercise, and concept representations should not be able to reach such performance unless there is additional label leakage in the training pipeline or the evaluation protocol does not reflect the diagnostic task. The authors should explain what information remains in this variant and why it performs so close to the full model; as reported, the ablation undermines the interpretation of the main results.","section":"Section 5.5, Table 4"}],"minor_comments":[{"comment":"LRCD-w/o-LCM is not a clean ablation of the mapper because it substitutes Bert embeddings for the default OpenAI-3-large to avoid memory limits, and the authors report that the variant still exceeds memory limits in some cases. The reported comparison therefore conflates the effect of the mapper with the effect of changing the text-embedding model.","section":"Section 5.5"},{"comment":"The transferability conclusion that science subjects transfer better than humanities subjects is confounded by differences in dataset size, average correct rate, and number of students across subjects. The paper itself acknowledges that 'we cannot rule out the influence of the data size,' so this finding should be presented as a speculative observation rather than a general insight.","section":"Section 5.2"},{"comment":"The text says 'OR-KaNCD, proposed in ORCDF [36]' but reference [36] is the NeuralCD paper; ORCDF is reference [24]. Please correct this citation.","section":"Section 5.1, Implementation Details"},{"comment":"The notation h_c in Eq. (6) is used without a prior definition; earlier, concept-level vectors are written as h_ck. Please clarify how a single concept representation is selected or aggregated when an exercise is associated with multiple concepts.","section":"Section 4.3, Eq. (6)"},{"comment":"The statement that the time complexity of the mappers is 'approximately O(d_l d)' is imprecise because the mapper is an MLP with two linear layers whose cost also depends on the intermediate hidden dimension. Please state the complexity in terms of the actual architecture used.","section":"Section 4.4"}],"recommendation":"reject","confidential_remarks":"The paper's central contribution is a zero-shot method, but the experimental protocol uses target-domain labels at inference, and the source training signal can directly copy the label token. Both problems concern the paper's main claim and cannot be addressed with local edits; the experiments would need to be redesigned under a different problem definition or with a genuinely label-free profile construction. I recommend rejection, although the underlying idea of text-based shared representations for cognitive diagnosis may be worth revisiting in a correctly formulated setting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful idea — representing students, exercises, and concepts as text built from behavior statistics, embedding them with a frozen encoder, and learning small mappers into an existing CDM. That package is new and it works in the narrow sense that the reported numbers are high. But the headline \"zero-shot\" claim is not supported by the experiments. The protocol uses target training labels to build the very profiles fed to the frozen mappers at inference (Eq. 2 puts ACR, Eq. 3 puts y_ij into the student profile, Eq. 4 mean-pools target-train interactions). So the method is zero-training, not zero-shot, and the results in Tables 2-3 cannot be read as evidence that a source-trained mapper generalizes to unseen domains.\n\nWhat's genuinely good: the formulation is clean, the integration with NCDM-family models is seamless, they release code and processed data, and they run ten seeds and report DOA as well as AUC. The ablation of the mapper (w/o-LCM) and the hyperparameter work on text encoders are sensible. The idea of editing student profiles in embedding space is a nice touch, though it is a demo rather than a rigorous evaluation.\n\nSoft spots, in proportion: the target-label protocol violation is decisive for the central claim. It is not a minor implementation detail; the problem definition in Section 3 explicitly forbids target-domain information, and Section 4.3 then uses target train labels to construct profiles. The same label also appears during source training: the student representation used to predict y_ij is built from an interaction text containing y_ij, so the mapper has a direct path to copy the label. The paper's defense — that target labels are unavailable \"during the training phase\" — conflates not training on target labels with not using them at all.\n\nThe LRCD-w/o-TCP ablation is also implausible: replacing all textual profiles with random normal vectors yields AUC 75-80. That suggests the ablation is not actually removing the textual information, or something else in the pipeline (e.g., the target labels in the profiles) is carrying the signal. This matters because the ablation is used to argue that the textual profiles themselves are what work.\n\nIs there a salvageable paper here? Yes, if reframed. The method is a reasonable transductive or few-shot approach: given some target responses, build profiles and diagnose the rest. That is still useful in practice, and the comparison against NCDM trained on full target data is informative. But as written, it does not deliver the zero-shot result it promises.\n\nWho is this for: people working on cognitive diagnosis and cross-domain educational data. A serious referee should engage with it, mainly to push the authors to fix the protocol and rerun the ablations. I'd send it to review with the expectation of major revision, not desk-reject.","headline":"A genuinely new representation idea and solid engineering, but the zero-shot claim is not tested because target-domain labels are baked into the profiles at inference.","tokens_in":26026,"tokens_out":1885,"would_cite":false,"duration_ms":18610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model that describes students, exercises and concepts in text can diagnose mastery in entirely new subjects and platforms without retraining, sometimes matching models trained on the target data.","keywords":["cognitive diagnosis","zero-shot learning","cross-domain transfer","textual cognitive profiles","language-cognitive mapping","text embeddings","student score prediction","intelligent education systems"],"falsifier":"Embed the target-domain concepts in a language not seen in training (e.g., translate the concept names) and rerun the same frozen mappers; if AUC collapses to near chance while the original target still scores high, the claimed domain-invariant mapping depends on lexical overlap rather than on a transferable cognitive representation.","tokens_in":24913,"feed_emoji":"🎓","tokens_out":6198,"duration_ms":56213,"temperature":0.7,"pith_summary":"Cognitive diagnosis models normally learn ID embeddings that only work in the domain where they were trained, so a new subject or platform means a new model. This paper proposes LRCD, which replaces IDs with textual cognitive profiles: concept names for concepts, concept names plus average accuracy for exercises, and the student's interaction records for students. These profiles are embedded into a unified language space and projected into cognitive-diagnosis space by small learned mappers, then fed to an ordinary CDM. The paper's central claim is that a model trained this way on source domains performs well zero-shot in completely different target domains, in some experiments reaching competitive performance with classic CDMs trained on the full target response data. If true, one trained model could serve new subjects and platforms without retraining.","feed_headline":"One model diagnoses students in subjects it was never trained on","feed_subtitle":"Text profiles plus learned mappers carry cognitive diagnosis across subjects and platforms, rivaling full-data models.","key_machinery":"The load-bearing mechanism is the pair of textual cognitive profiles and language-cognitive mappers. Profiles are constructed by three rules: a concept profile is just the concept name; an exercise profile appends the exercise's average correct rate (ACR) to the names of its related concepts; a student profile is the set of textual profiles of that student's interactions, each interaction being the exercise's concept names, its ACR, and the student's score. These texts are vectorized by a frozen text-embedding module, with each student represented by mean pooling over their interactions, and the mappers $\\mathcal{F}_s,\\mathcal{F}_e,\\mathcal{F}_c$ (two linear layers with ReLU) project the language-space vectors to cognitive-space vectors. Training is supervised by a binary cross-entropy loss over source domains, and zero-shot inference applies the frozen mappers to target-domain profiles. The framework is model-agnostic and is demonstrated with OR-KaNCD, KaNCD, KSCD, and MIRT as the integrated CDM $\\mathcal{M}_{\\text{CD}}$.","core_discovery":"LRCD asserts that the barrier to cross-domain cognitive diagnosis is not the diagnosis task itself but the input representation. By writing each entity as text—a concept's name, an exercise's related concept names plus its average correct rate, a student's list of interaction records—and embedding that text with a pretrained text-embedding module, all domains land in one language space. The proposed language-cognitive mappers (simple MLPs) then project these vectors into a cognitive space where any existing CDM can consume them. Since the mapping must apply to unseen entities, the paper defines zero-shot inference for a target domain by freezing the trained mappers and running them on the target's textual profiles. Experiments on three real datasets report that LRCD reaches at least 97.30% of oracle AUC in subject-level transfer and at least 94.91% in platform-level transfer, and in several settings its AUC is competitive with NCDM trained on target data.","pith_inferences":["If the mapper is the only transferable component, a sharper test is to hold the text-embedding module constant and swap concept-name language between source and target; large degradation would show the transfer rides on semantic similarity of names rather than a domain-general cognitive signal.","The profile construction needs labels in the target domain to compute ACR and student interaction records, so a truly cold target with no prior response logs remains outside the method's stated zero-shot scope.","The same unified-space idea could be applied to interleaving diagnosis with knowledge tracing or to cross-course recommendation, since both tasks share the same student-exercise-concept structure; the paper does not pursue these applications.","Science-to-science and higher-to-lower transfer results suggest a hierarchy of learnability across domains; whether that hierarchy reflects data size, concept granularity, or inherent subject structure is left open."],"forward_implications":["A single LRCD model can diagnose target subjects on the same platform with zero overlap, reaching at least 97.30% of oracle AUC in the reported subject-level experiments.","A model trained on one platform's math logs can diagnose math on another platform, reaching at least 94.91% of oracle AUC in the reported platform-level experiments.","Source domains from science subjects transfer better than humanities sources, and higher-education sources transfer better downward; LRCD's unified space exposes these regularities.","The architecture can absorb different text-embedding modules and different CDMs; the authors recommend OpenAI-3-large and OR-KaNCD, while BERT remains serviceable under resource constraints.","A student's diagnosis can be edited by adding hypothetical interactions to the profile, letting a student preview the effect of practicing a concept without retraining or changing the response data."],"supporting_citations":[{"why":"Supplies the SLP K-12 multi-subject response logs used for the subject-level zero-shot experiments.","marker":"[22]"},{"why":"Supplies the EDM Cup 2023 ASSISTments response data used as a platform-level source or target domain.","marker":"[4]"},{"why":"Supplies the MOOC response data used as the second platform-level domain.","marker":"[39]"},{"why":"Defines the ID-embedding neural cognitive diagnosis paradigm that LRCD replaces and serves as an oracle baseline.","marker":"[35]"},{"why":"Provides the text-embedding module used to vectorize textual cognitive profiles into the unified language space.","marker":"[25]"},{"why":"Provides BERT as an alternative text embedding used in hyperparameter analysis and in implementing Zero-1-3.","marker":"[3]"},{"why":"Is the early-bird-student zero-shot baseline that LRCD outperforms in the overlapping-students setting.","marker":"[6]"},{"why":"Is the knowledge-concept-graph cold-start baseline that LRCD outperforms in cross-domain settings.","marker":"[7]"},{"why":"Supplies OR-KaNCD, the default integrated CDM inside LRCD.","marker":"[24]"}],"fun_headline_variants":["Language profiles enable zero-shot diagnosis across subjects and platforms","Text-based model transfers cognitive diagnosis to unseen domains","Zero-shot cross-domain diagnosis using text embeddings and mappers","Diagnose new domains without retraining via language representation","Text embeddings carry cognitive diagnosis to new domains zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mapping from language embeddings to cognitive vectors learned on source domains remains valid in a completely new domain, and that target-domain response logs are available to build the textual profiles, even though the model is 'zero-shot' in not training on those logs.","fun_headline_variants_meta":{"raw":{"variants":["Language profiles enable zero-shot diagnosis across subjects and platforms","Text-based model transfers cognitive diagnosis to unseen domains","Zero-shot cross-domain diagnosis using text embeddings and mappers","Diagnose new domains without retraining via language representation","Text embeddings carry cognitive diagnosis to new domains zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3867,"prompt_tokens":1006,"completion_tokens":2861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":2785}},"tokens_in":622,"tokens_out":2861,"duration_ms":18232,"temperature":1.0,"reasoning_tokens":2785,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:02:13.513883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Embed the target-domain concepts in a language not seen in training (e.g., translate the concept names) and rerun the same frozen mappers; if AUC collapses to near chance while the original target still scores high, the claimed domain-invariant mapping depends on lexical overlap rather than on a transferable cognitive representation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SLP K-12 multi-subject response logs used for the subject-level zero-shot experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ID-embedding neural cognitive diagnosis paradigm that LRCD replaces and serves as an oracle baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the text-embedding module used to vectorize textual cognitive profiles into the unified language space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT as an alternative text embedding used in hyperparameter analysis and in implementing Zero-1-3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the early-bird-student zero-shot baseline that LRCD outperforms in the overlapping-students setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the knowledge-concept-graph cold-start baseline that LRCD outperforms in cross-domain settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies OR-KaNCD, the default integrated CDM inside LRCD."}],"review_version":1}