{"id":"665a09a1-3f59-46de-a1bc-644d06488e14","arxiv_id":"2501.07674","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A knowledge-component diagnostic pipeline, inspired by cognitive diagnosis theory, generates weakness-targeted synthetic data that improves small LLMs on math, code, and exam benchmarks by up to 13.1 percentage points.","lead":"This paper introduces CDS, a method that diagnoses a small language model's weaknesses at a knowledge-component level and uses an advanced LLM to generate targeted practice data. The method improved Qwen1.5-7B and Llama3-8B by up to 13.1 points on math benchmarks, with gains on code and exam benchmarks too.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5's CDS_score has the wrong sign: it assigns higher scores to high-accuracy, high-frequency KCs, yet §3.3 says low-accuracy/low-frequency KCs should be prioritized; Stage 2 therefore selects against the intended weaknesses.","rationale":"The reader's weakest assumption—the DINA binary mastery assumption—is a real external validity concern, but it is secondary to an internal sign error visible directly in Eq. 5. The CDS_score is a named contribution and the whole of Stage 2; Section 5.2's positive results are the only evidence for it. If the sign is wrong, the table may still be reproducible if the code uses a different sign, or if selecting strong-KC data happens to help, but the paper's causal story—weakness-targeted selection—would be false as written. The synthesis strategies, augmentation, and Stage 1 filtering may still explain Table 1 gains, so this is not grounds for outright rejection; it is grounds for keeping the conditional verdict and adding a concrete requirement: correct Eq. 5 and rerun Table 2. I disagree with the reader on the specific load-bearing concern because the sign error would corrupt the selection score even with a perfect diagnostic profile, whereas the DINA concern requires external validation to matter. The limitations section's acknowledgment of KC annotation subjectivity further motivates verifying the code, but the decisive issue is the score's internal inconsistency.","tokens_in":19336,"tokens_out":8909,"duration_ms":95528,"concrete_test":"Check the released code's Stage 2 scoring function. If it literally computes w1*log(acc+1e-6) + w2*log(freq+1e-6), rerun the Table 2 selection experiments (0.1k/0.2k/0.3k and 0.4k/0.8k/1.6k) with the corrected sign, V = -w1*log(acc+ε) - w2*log(freq+ε), holding everything else fixed. If the corrected score's average advantage over Random falls below or reverses the claimed 2.65% and 3.28% gaps, the paper's stated weakness-targeting selection mechanism is not supported. A minimal sanity check: under the intended semantics, a weak KC with Acc=0.2 must receive a higher V than a strong KC with Acc=0.9; under Eq. 5 it receives a lower V.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines V(kcj) = w1 log(Acc(kcj)+ε) + w2 log(Freqa(kcj)+ε) and keeps samples with CDSscore > μ−σ. Since Acc and Freqa lie in [0,1], log is monotone increasing and negative: weak and rare KCs get the most negative V values and are therefore less likely to be retained. This is exactly the opposite of the stated hypothesis that 'data containing low-frequency and low-accuracy KCs are more effective for targeting weaknesses.' A correct implementation for the stated goal would need V = −w1 log(Acc+ε) − w2 log(Freq+ε) or an equivalent use of error rates. This is an internal inconsistency, not an external assumption: even if the DINA diagnostic profile in §3.2 were perfectly valid, the selection score would invert it. Table 2 and the §5.2 claim that CDS_score yields weakness-targeted selection therefore cannot be interpreted as written. The anonymous repository may contain a corrected sign, but the paper as submitted does not.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cognitive Diagnostic Synthesis (CDS), a pipeline that uses cognitive diagnosis theory to profile a small LLM's mastery of fine-grained knowledge components (KCs) from benchmark evaluation, then guides a stronger LLM to synthesize and select weakness-targeted training data. Two synthesis strategies are presented: a global strategy based on KC accuracy/frequency thresholds and a fine-grained strategy based on per-error diagnostic analysis. The resulting synthetic data are augmented by rewriting and fusion, and selected through a two-stage process that includes an LLM-based correctness filter and a novel CDS score. Experiments on GSM8k, MBPP, and GAOKAO-Bench, plus OOD sets, compare against six synthesis baselines and six selection baselines using Qwen1.5-7B-Chat and Llama3-8B-Instruct, reporting consistent average improvements.","tokens_in":19561,"tokens_out":6442,"duration_ms":60926,"significance":"If the findings hold, CDS would be a valuable addition to synthetic-data pipelines, combining fine-grained diagnosis with targeted generation. The empirical scope is broad: three tasks, two student models, multiple baselines, and ID/OOD evaluation. The paper also ships code and detailed prompts, which aids reproducibility. However, a central internal inconsistency in the CDS score equation (Section 3.3) inverts the stated selection objective, so the headline selection results in Table 2 cannot be interpreted as written. The diagnostic core also relies on a strong binary-mastery assumption that is not validated, and none of the experiments report variance or significance tests. These issues are load-bearing and require correction before the claims can be accepted.","major_comments":[{"comment":"The CDS score defined in Eq. (5) is sign-inverted relative to the stated objective. The text hypothesizes that 'data containing low-frequency and low-accuracy KCs are more effective for targeting weaknesses,' and the retention rule keeps samples with CDSscore > μ−σ. However, since V(kcj) = w1 log(Acc(kcj)+ε) + w2 log(Freqa(kcj)+ε) with positive weights and monotone increasing logarithms, high-accuracy and high-frequency KCs receive the largest (least negative) V values, so the retained samples are precisely those least likely to contain weak KCs. Table 2 therefore cannot be interpreted as evidence for weakness-targeted selection; the formula must be changed to use error rates or negative weights, or the stated hypothesis must be revised.","section":"Section 3.3, Eq. (5)-(6)"},{"comment":"The DINA binary mastery assumption is load-bearing but unvalidated for LLMs. Eq. (3) computes per-KC accuracy under the rule that a correct response implies mastery of all associated KCs and an incorrect response implies mastery of none. For a multi-KC item, a model that masters all but one KC will be scored as mastering none of them, and a lucky guess can be scored as mastering all, systematically misidentifying weak KCs. The paper provides no check of this assumption against an alternative diagnosis (e.g., per-KC accuracy on single-KC items or human error attribution), so the global strategy and the CDS score may be aiming at the wrong skills even after the sign error is fixed.","section":"Section 3.2, Eq. (3)"},{"comment":"The paper reports no variance, standard deviations, or significance tests for any experiment, while using the word 'significant' in the abstract and text. Several reported gains are small or tied (e.g., Qwen1.5-7B on GAOKAO ID: 64.60 vs. LEC 64.80; Llama3-8B on MBPP: 42.80 tied with LEC and LLM2LLM), and without multiple seeds or statistical testing the claimed improvements, particularly the OOD generalization results, may be within run-to-run noise. Please report error bars or at least state that results are from a single run and adjust the significance language accordingly.","section":"Section 5.1 and 5.2, Tables 1 and 2"}],"minor_comments":[{"comment":"The statement that 'CDS consistently produces optimal results in coding and examination tasks' is too strong for Table 1: CDS is not strictly best on GAOKAO ID for Qwen (LEC: 64.80 vs. CDS: 64.60) and is tied on MBPP for Llama (LEC and LLM2LLM: 42.80). I suggest rewording to 'competitive or best on average.'","section":"Section 5.1, first paragraph"},{"comment":"There is a typographical inconsistency: the text defines 'Freqs(kcj)' as the frequency in Da, while the equation uses 'Freqa(kcj)'. Please unify the notation and also insert a missing space in 'onkcj'.","section":"Section 3.3, text below Eq. (5)"},{"comment":"Treating low frequency as a weakness criterion in Algorithm 1 is not self-evident; a rare KC in the training set may be underrepresented rather than poorly mastered, and the paper does not justify this design choice or report how many KCs are selected by each threshold condition.","section":"Algorithm 1, line 3"},{"comment":"The data selection evaluation uses only Qwen1.5-7B-Chat; the claim of broad applicability of CDSscore would be stronger if the selection experiments were repeated on Llama3-8B-Instruct, or the paper should at least acknowledge this single-model evidence.","section":"Section 5.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Eq. (5) is likely a correctable typo in the manuscript, but it is central to the data selection contribution. The authors should verify the actual code, correct the formula or the objective, and re-run the selection experiments. I would also encourage adding a validation of the DINA assumption, as it underpins the entire diagnostic step. The paper is within the journal's scope and the idea is promising, but the current version overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dave,\n\nQuick take: this paper brings cognitive diagnosis theory into weakness-targeted synthetic data for LLM fine-tuning, and that combination is genuinely new. The pipeline is clearly described, the experiments are broad (three tasks, two student models, ID and OOD benchmarks, six baselines, plus a selection ablation), and the reported gains are consistent. If the method works as described, it fills a real gap.\n\nBut there is a load-bearing internal inconsistency in the selection stage. Eq. 5 defines V(kcj) = w1 log(Acc+ε) + w2 log(Freq+ε). Since Acc and Freq are in [0,1], the logs are negative and monotone increasing. The paper says it wants to prioritize 'low-frequency and low-accuracy KCs' and keeps samples with CDSscore > μ−σ. That formula does the opposite: high-accuracy, high-frequency KCs get the least-negative V, so samples containing them are more likely to survive. The stated hypothesis and the implemented score point in opposite directions. This isn't a matter of external assumptions; it's a sign error on the page. The anonymous repo might have a corrected version, but the submitted paper as written cannot be interpreted to support the claim that CDSscore targets weaknesses. Table 2 and Section 5.2 rest on that claim.\n\nThat one flaw is the difference between a solid paper and a conditional one. Everything else is in better shape. The DINA binary-mastery assumption is strong and unvalidated for LLMs, but it's a modeling choice, not an internal contradiction. The lack of error bars or significance tests is a real but minor concern given the breadth and consistency of the gains. The hand-set thresholds (δa, δf, w1, w2) would deserve a sensitivity analysis, but they are standard practice in this literature. The limitations section honestly acknowledges randomness and subjectivity in KC annotation.\n\nBottom line: the core idea is worth taking seriously, and the experiments are extensive enough to justify a full referee. But the CDSscore equation needs to be corrected or explained before the paper's central mechanism can be trusted. I'd send it to review, but with the expectation of heavy revision.\n\nBest,\n[Your name]","headline":"A promising CDT-guided data synthesis pipeline, undermined by a sign error in its selection score that inverts the stated weakness-targeting logic.","tokens_in":20131,"tokens_out":3138,"would_cite":false,"duration_ms":27589,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that diagnostic profiles of model ability at the knowledge-component level can guide synthetic data generation and selection, producing reported gains of up to 13.10% in math reasoning, 6.00% in code, and 5.43% in exams.","keywords":["cognitive diagnosis theory","synthetic data","knowledge components","LLM evaluation","fine-tuning","data selection","mathematical reasoning","code generation"],"falsifier":"Take a few hundred erroneous responses from one of the benchmarks, have human experts or a stronger model label which of the question's tagged knowledge components were actually implicated in the error, and compare that list to the DINA-derived weak-KC list; if the lists diverge on a substantial fraction of cases, the diagnostic profile that drives synthesis is not trustworthy.","tokens_in":1871,"feed_emoji":"🎯","tokens_out":3135,"duration_ms":112370,"temperature":0.7,"pith_summary":"The paper tries to establish that a small language model improves more when its evaluation is broken down into knowledge components, the individual skills a question exercises, and that diagnostic profile, not the raw right/wrong score, drives synthetic data generation and selection. The proposed Cognitive Diagnostic Synthesis (CDS) pipeline annotates benchmark questions with knowledge components, computes each component's accuracy and frequency under a binary mastery assumption, and flags weak components. It then uses a stronger open-source LLM to generate training data aimed at those weak components, augments the data by rewriting and multi-component fusion, and selects data with a novelty score built from the same diagnostic profile. On GSM8k the paper reports gains of 10.54% for Qwen1.5-7B and 11.12% for Llama3-8B over the prompting baseline, with consistent gains across code, math, and exam benchmarks. If true, this matters because it turns the usual coarse evaluation loop into a weakness-targeted one that could make synthetic-data fine-tuning cheaper and more precise.","feed_headline":"Diagnosing weak skills lifts LLM scores by 13 percent","feed_subtitle":"A skill-level diagnosis decides which synthetic problems are generated and which are kept.","key_machinery":"The central object is the knowledge-component diagnostic profile, a vector of per-KC accuracy and frequency values over the tagged benchmark. It is built from the Q-KC matrix that records which knowledge components each question exercises, and from the DINA binary mastery assumption that converts right/wrong answers into mastery or non-mastery of all attached KCs. This profile does double duty in the paper: the global synthesis strategy selects weak KCs directly from its low-accuracy/low-frequency entries, and the CDS score reuses the same accuracy and frequency values, combined as $V(kc_j)=w_1\\log(Acc(kc_j)+\\epsilon)+w_2\\log(Freq(kc_j)+\\epsilon)$, to rank synthesized samples so that data containing many low-accuracy and low-frequency KCs survives the $\\mu-\\sigma$ cutoff. The second diagnostic tool is the question-level diagnostic prompt, where a stronger LLM identifies the weak KCs behind a concrete erroneous response and the resulting reasoning text extends the chain of thought used in synthesis.","core_discovery":"On its own terms, CDS is a claim about where the leverage in synthetic-data fine-tuning sits. The paper argues that the reason error-based methods such as seed-data rewriting or positive/negative example synthesis give limited and sometimes negative gains is that they treat each mistake in isolation; what is missing is a summary of which underlying knowledge components are weak. CDS supplies that summary by applying the DINA cognitive-diagnosis model, a binary mastery assumption that a correct answer means mastery of every knowledge component attached to the question and an incorrect answer means mastery of none, and from those rules it computes per-component accuracy and frequency, calls components weak when accuracy or frequency falls below thresholds, and generates data per weak component. A second, question-level strategy uses a stronger LLM to diagnose each erroneous case and folds that diagnostic reasoning into a long chain-of-thought prompt for synthesis; augmentation then rewrites data under KC constraints and fuses two components into one sample, and a two-stage selector filters with an LLM score and then a CDS score that preferentially keeps low-accuracy, low-frequency components. The experimental payload is the reported margin over six baselines across GSM8k, MBPP, and GAOKAO-Bench plus their out-of-domain counterparts, including the headline GSM8k gains and the CDS score's consistent edge over a random selector.","pith_inferences":["Editorial inference: the load-bearing DINA assumption is the part most likely to be tested next; a model that guesses correctly or slips once on a multi-KC question will be misdiagnosed, so auditing per-KC attributions on a few hundred errors would show how much of the profile is noise.","Editorial inference: CDS's loop is applicable to any task with decomposable rubrics, not just math, code, and exams; multi-step tool use, document understanding, or scientific problem solving could adopt the same diagnose-synthesize-select cycle if questions can be tagged with knowledge components.","Editorial inference: because the paper uses only one open-source advanced LLM as generator, the reported gains are a lower bound on what the method might do with a stronger generator; conversely, the method's dependence on the generator's diagnosis quality is untested.","Editorial inference: a direct ablation that replaces CDS's weak-KC selection with randomly selected KCs at the same data volume would isolate whether the gain comes from diagnosis itself or simply from generating extra targeted data."],"forward_implications":["With the same data budget, weakness-targeted KC data outperforms error-based and concept-based baselines, so refining evaluation into knowledge components changes training outcomes rather than just reporting them.","CDS generalizes out of domain: reported gains on HumanEval and GAOKAO-Bench-Updates suggest that data synthesized from in-domain diagnostic profiles can transfer to related tasks the model never trained on.","Different KC annotation styles, model-generated tags for math and code and exam-syllabus chapter titles for GAOKAO, both produce gains, indicating the method is not tied to a single annotation scheme.","The CDS score selection is cheap and stable, beating random selection by average improvements of 2.65% and 3.28% while taking near-zero selection time.","Combining the global and fine-grained synthesis strategies, and combining rewrite with fusion augmentation, consistently beats each strategy alone, so the two diagnostic perspectives are complementary."],"supporting_citations":[{"why":"Supplies the DINA model and the binary mastery assumption used to compute per-KC accuracy and identify weak knowledge components.","marker":"(De La Torre, 2009)"},{"why":"Provides Q-matrix theory that links test items to underlying knowledge components, the basis of the Q-KC matrix.","marker":"(Tatsuoka, 1983)"},{"why":"LLM2LLM is the error-seed synthesis baseline that CDS must beat, generating new data from incorrect examples.","marker":"(Lee et al., 2024)"},{"why":"LEC is the error-embedding baseline that selects similar positive examples and is a core comparison in the main experiments.","marker":"(Ying et al., 2024)"},{"why":"MUSTARD is the knowledge-concept-based synthesis baseline representing the approach CDS claims to improve on.","marker":"(Huang et al., 2024b)"},{"why":"GSM8k is the in-domain mathematical reasoning benchmark on which the largest reported gains appear.","marker":"(Cobbe et al., 2021)"},{"why":"MBPP is the in-domain code generation benchmark used for training and evaluation.","marker":"(Austin et al., 2021)"},{"why":"GAOKAO-Bench is the academic examination benchmark whose chapter-title KC annotation tests the flexibility of the method.","marker":"(Zhang et al., 2023)"},{"why":"HumanEval is the out-of-domain code benchmark used to test whether CDS transfers beyond in-domain training data.","marker":"(Chen et al., 2021)"}],"fun_headline_variants":["Cognitive diagnosis targets weak skills for better synthetic data","Skill-level diagnosis lifts LLM performance by 13%","Weak-skill detection guides synthetic data for 13% gains","CDS: diagnose weak skills, synthesize targeted data","Targeted data synthesis via cognitive diagnosis boosts LLMs"],"cache_read_input_tokens":22272,"weakest_assumption_plain":"The load-bearing premise is that a correct answer always means the model has mastered every skill attached to that question and a wrong answer always means it has mastered none, so a single careless error or lucky guess mislabels an entire skill profile.","fun_headline_variants_meta":{"raw":{"variants":["Cognitive diagnosis targets weak skills for better synthetic data","Skill-level diagnosis lifts LLM performance by 13%","Weak-skill detection guides synthetic data for 13% gains","CDS: diagnose weak skills, synthesize targeted data","Targeted data synthesis via cognitive diagnosis boosts LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000887,"raw_usage":{"total_tokens":3849,"prompt_tokens":985,"completion_tokens":2864,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2786}},"tokens_in":601,"tokens_out":2864,"duration_ms":19001,"temperature":1.0,"reasoning_tokens":2786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:55.256525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a few hundred erroneous responses from one of the benchmarks, have human experts or a stronger model label which of the question's tagged knowledge components were actually implicated in the error, and compare that list to the DINA-derived weak-KC list; if the lists diverge on a substantial fraction of cases, the diagnostic profile that drives synthesis is not trustworthy.","supporting_citations":[{"cited_title":"Mahoney, Kurt Keutzer, and Amir Gholami","cited_arxiv_id":null,"evidence_quote":"LLM2LLM is the error-seed synthesis baseline that CDS must beat, generating new data from incorrect examples."}],"review_version":1}