Pith. sign in

REVIEW 3 major objections 4 minor 56 references

CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that diagnostic profiles of model ability at the knowledge-component level can guide synthetic data generation and selection, producing reported gains of up to 13.10% in math reasoning, 6.00% in code, and 5.43% in exams.

desk verdict A promising CDT-guided data synthesis pipeline, undermined by a sign error in its selection score that inverts the stated weakness-targeting logic. read the letter →

arxiv 2501.07674 v3 pith:CZQMAOS7 submitted 2025-01-13 cs.AI

Haokun Zhao , Jinyi Han , Jiaqing Liang , Yanghua Xiao , Xiaojun Meng , Jiansheng Wei This is my paper · ORCID
classification cs.AI
keywords cognitivediagnosistheorysyntheticdataknowledgecomponentsLLMevaluationfine-tuningselectionmathematicalreasoningcodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small language model improves more when its evaluation is broken down into knowledge components, the individual skills a question exercises, and that diagnostic profile, not the raw right/wrong score, drives synthetic data generation and selection. The proposed Cognitive Diagnostic Synthesis (CDS) pipeline annotates benchmark questions with knowledge components, computes each component's accuracy and frequency under a binary mastery assumption, and flags weak components. It then uses a stronger open-source LLM to generate training data aimed at those weak components, augments the data by rewriting and multi-component fusion, and selects data with a novelty score built from the same diagnostic profile. On GSM8k the paper reports gains of 10.54% for Qwen1.5-7B and 11.12% for Llama3-8B over the prompting baseline, with consistent gains across code, math, and exam benchmarks. If true, this matters because it turns the usual coarse evaluation loop into a weakness-targeted one that could make synthetic-data fine-tuning cheaper and more precise.

What carries the argument

The central object is the knowledge-component diagnostic profile, a vector of per-KC accuracy and frequency values over the tagged benchmark. It is built from the Q-KC matrix that records which knowledge components each question exercises, and from the DINA binary mastery assumption that converts right/wrong answers into mastery or non-mastery of all attached KCs. This profile does double duty in the paper: the global synthesis strategy selects weak KCs directly from its low-accuracy/low-frequency entries, and the CDS score reuses the same accuracy and frequency values, combined as $V(kc_j)=w_1\log(Acc(kc_j)+\epsilon)+w_2\log(Freq(kc_j)+\epsilon)$, to rank synthesized samples so that data containing many low-accuracy and low-frequency KCs survives the $\mu-\sigma$ cutoff. The second diagnostic tool is the question-level diagnostic prompt, where a stronger LLM identifies the weak KCs behind a concrete erroneous response and the resulting reasoning text extends the chain of thought used in synthesis.

What would settle it

Take a few hundred erroneous responses from one of the benchmarks, have human experts or a stronger model label which of the question's tagged knowledge components were actually implicated in the error, and compare that list to the DINA-derived weak-KC list; if the lists diverge on a substantial fraction of cases, the diagnostic profile that drives synthesis is not trustworthy.

Watch

Extended reading notes

Core claim

On its own terms, CDS is a claim about where the leverage in synthetic-data fine-tuning sits. The paper argues that the reason error-based methods such as seed-data rewriting or positive/negative example synthesis give limited and sometimes negative gains is that they treat each mistake in isolation; what is missing is a summary of which underlying knowledge components are weak. CDS supplies that summary by applying the DINA cognitive-diagnosis model, a binary mastery assumption that a correct answer means mastery of every knowledge component attached to the question and an incorrect answer means mastery of none, and from those rules it computes per-component accuracy and frequency, calls components weak when accuracy or frequency falls below thresholds, and generates data per weak component. A second, question-level strategy uses a stronger LLM to diagnose each erroneous case and folds that diagnostic reasoning into a long chain-of-thought prompt for synthesis; augmentation then rewrites data under KC constraints and fuses two components into one sample, and a two-stage selector filters with an LLM score and then a CDS score that preferentially keeps low-accuracy, low-frequency components. The experimental payload is the reported margin over six baselines across GSM8k, MBPP, and GAOKAO-Bench plus their out-of-domain counterparts, including the headline GSM8k gains and the CDS score's consistent edge over a random selector.

Load-bearing premise

The load-bearing premise is that a correct answer always means the model has mastered every skill attached to that question and a wrong answer always means it has mastered none, so a single careless error or lucky guess mislabels an entire skill profile.

Editorial extensions

If this is right

  • With the same data budget, weakness-targeted KC data outperforms error-based and concept-based baselines, so refining evaluation into knowledge components changes training outcomes rather than just reporting them.
  • CDS generalizes out of domain: reported gains on HumanEval and GAOKAO-Bench-Updates suggest that data synthesized from in-domain diagnostic profiles can transfer to related tasks the model never trained on.
  • Different KC annotation styles, model-generated tags for math and code and exam-syllabus chapter titles for GAOKAO, both produce gains, indicating the method is not tied to a single annotation scheme.
  • The CDS score selection is cheap and stable, beating random selection by average improvements of 2.65% and 3.28% while taking near-zero selection time.
  • Combining the global and fine-grained synthesis strategies, and combining rewrite with fusion augmentation, consistently beats each strategy alone, so the two diagnostic perspectives are complementary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the load-bearing DINA assumption is the part most likely to be tested next; a model that guesses correctly or slips once on a multi-KC question will be misdiagnosed, so auditing per-KC attributions on a few hundred errors would show how much of the profile is noise.
  • Editorial inference: CDS's loop is applicable to any task with decomposable rubrics, not just math, code, and exams; multi-step tool use, document understanding, or scientific problem solving could adopt the same diagnose-synthesize-select cycle if questions can be tagged with knowledge components.
  • Editorial inference: because the paper uses only one open-source advanced LLM as generator, the reported gains are a lower bound on what the method might do with a stronger generator; conversely, the method's dependence on the generator's diagnosis quality is untested.
  • Editorial inference: a direct ablation that replaces CDS's weak-KC selection with randomly selected KCs at the same data volume would isolate whether the gain comes from diagnosis itself or simply from generating extra targeted data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Cognitive Diagnostic Synthesis (CDS), a pipeline that uses cognitive diagnosis theory to profile a small LLM's mastery of fine-grained knowledge components (KCs) from benchmark evaluation, then guides a stronger LLM to synthesize and select weakness-targeted training data. Two synthesis strategies are presented: a global strategy based on KC accuracy/frequency thresholds and a fine-grained strategy based on per-error diagnostic analysis. The resulting synthetic data are augmented by rewriting and fusion, and selected through a two-stage process that includes an LLM-based correctness filter and a novel CDS score. Experiments on GSM8k, MBPP, and GAOKAO-Bench, plus OOD sets, compare against six synthesis baselines and six selection baselines using Qwen1.5-7B-Chat and Llama3-8B-Instruct, reporting consistent average improvements.

Significance. If the findings hold, CDS would be a valuable addition to synthetic-data pipelines, combining fine-grained diagnosis with targeted generation. The empirical scope is broad: three tasks, two student models, multiple baselines, and ID/OOD evaluation. The paper also ships code and detailed prompts, which aids reproducibility. However, a central internal inconsistency in the CDS score equation (Section 3.3) inverts the stated selection objective, so the headline selection results in Table 2 cannot be interpreted as written. The diagnostic core also relies on a strong binary-mastery assumption that is not validated, and none of the experiments report variance or significance tests. These issues are load-bearing and require correction before the claims can be accepted.

major comments (3)
  1. [Section 3.3, Eq. (5)-(6)] The CDS score defined in Eq. (5) is sign-inverted relative to the stated objective. The text hypothesizes that 'data containing low-frequency and low-accuracy KCs are more effective for targeting weaknesses,' and the retention rule keeps samples with CDSscore > μ−σ. However, since V(kcj) = w1 log(Acc(kcj)+ε) + w2 log(Freqa(kcj)+ε) with positive weights and monotone increasing logarithms, high-accuracy and high-frequency KCs receive the largest (least negative) V values, so the retained samples are precisely those least likely to contain weak KCs. Table 2 therefore cannot be interpreted as evidence for weakness-targeted selection; the formula must be changed to use error rates or negative weights, or the stated hypothesis must be revised.
  2. [Section 3.2, Eq. (3)] The DINA binary mastery assumption is load-bearing but unvalidated for LLMs. Eq. (3) computes per-KC accuracy under the rule that a correct response implies mastery of all associated KCs and an incorrect response implies mastery of none. For a multi-KC item, a model that masters all but one KC will be scored as mastering none of them, and a lucky guess can be scored as mastering all, systematically misidentifying weak KCs. The paper provides no check of this assumption against an alternative diagnosis (e.g., per-KC accuracy on single-KC items or human error attribution), so the global strategy and the CDS score may be aiming at the wrong skills even after the sign error is fixed.
  3. [Section 5.1 and 5.2, Tables 1 and 2] The paper reports no variance, standard deviations, or significance tests for any experiment, while using the word 'significant' in the abstract and text. Several reported gains are small or tied (e.g., Qwen1.5-7B on GAOKAO ID: 64.60 vs. LEC 64.80; Llama3-8B on MBPP: 42.80 tied with LEC and LLM2LLM), and without multiple seeds or statistical testing the claimed improvements, particularly the OOD generalization results, may be within run-to-run noise. Please report error bars or at least state that results are from a single run and adjust the significance language accordingly.
minor comments (4)
  1. [Section 5.1, first paragraph] The statement that 'CDS consistently produces optimal results in coding and examination tasks' is too strong for Table 1: CDS is not strictly best on GAOKAO ID for Qwen (LEC: 64.80 vs. CDS: 64.60) and is tied on MBPP for Llama (LEC and LLM2LLM: 42.80). I suggest rewording to 'competitive or best on average.'
  2. [Section 3.3, text below Eq. (5)] There is a typographical inconsistency: the text defines 'Freqs(kcj)' as the frequency in Da, while the equation uses 'Freqa(kcj)'. Please unify the notation and also insert a missing space in 'onkcj'.
  3. [Algorithm 1, line 3] Treating low frequency as a weakness criterion in Algorithm 1 is not self-evident; a rare KC in the training set may be underrepresented rather than poorly mastered, and the paper does not justify this design choice or report how many KCs are selected by each threshold condition.
  4. [Section 5.2, Table 2] The data selection evaluation uses only Qwen1.5-7B-Chat; the claim of broad applicability of CDSscore would be stronger if the selection experiments were repeated on Llama3-8B-Instruct, or the paper should at least acknowledge this single-model evidence.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the diagnostic profile drives a selection heuristic and all gains are measured on held-out or OOD evaluation sets. The only self-citation is minor and non-load-bearing, while the sign issue in Eq. 5 is a correctness risk rather than a circular derivation.

full rationale

CDS's claimed derivation is not circular: the diagnostic profile is an input to data synthesis and selection, not a fitted stand-in for the reported evaluation. Acc and Freq (Eqs. 3-4) are computed on Dtarget from the student model and used to generate and filter synthetic data; final scores in Table 1 come from held-out Deval and OOD sets, so no evaluation metric is recycled as a selection target. The CDS_score (Eqs. 5-6) is a hand-set heuristic with weights w1=0.85, w2=0.15; Table 2 evaluates selection methods on the same synthetic pool, which is an honest empirical comparison rather than a self-fulfilling derivation. The DINA binary-mastery assumption in Section 3.2 is a substantive modeling assumption; if it is wrong the diagnosis is misdirected, but the improvement claim is not equivalent to the assumption. The only self-citation (Zhao et al., 2024, CEM) appears in the introduction as an example of evaluation-guided data workflows and supports no equation or algorithm; it is non-load-bearing. I also note two non-circular concerns: (a) Eq. 5's sign appears inconsistent with the stated priority for low-accuracy/low-frequency KCs, because log is monotone increasing and the retention rule keeps high CDSscore, which would retain high-accuracy/high-frequency KCs; this is a correctness/consistency risk, not circularity; (b) the paper itself acknowledges KC annotation randomness and subjectivity (Section 7), which is a validity caveat. Neither concern makes the derivation reduce to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small number of hand-set parameters (weights and thresholds) and on strong assumptions about the reliability of LLM-generated knowledge component annotations and the DINA mastery model. No new physical or conceptual entities are introduced beyond the CDS score, which is a simple summation heuristic.

free parameters (5)
  • w1, w2 (CDS score weights) = w1=0.85, w2=0.15
    Weights balancing the accuracy and frequency terms in Eq. (5). No sensitivity analysis is reported, and these values directly affect which samples are selected in Stage 2.
  • delta_a, delta_f (weak-KC thresholds) = Not specified in main text
    Used in Algorithm 1 to decide which KCs are weak (Acc<=delta_a or Freq<=delta_f). The values are not given in the paper, so the identification of weak KCs is not exactly reproducible.
  • theta (Stage 1 filtering threshold) = theta=8 (Table 9)
    Score threshold for filtering generated data in Stage 1. Hand-set without sensitivity analysis.
  • epsilon (log smoothing constant) = 1e-6
    Small constant in Eq. (5) to avoid log(0). Minor, but part of the score calculation.
  • augmentation proportions p_rw, p_fusion = p_rw=0.25, p_fusion=0.25
    Proportion of data rewritten and fused in augmentation. The choice of 0.25 is not motivated or varied.
assumptions (4)
  • domain assumption DINA binary mastery: a correct response to a question implies mastery of all associated KCs, and an incorrect response implies no mastery of any associated KCs.
    Stated in Section 3.2 (Global Strategy). This is the basis for computing KC accuracy in Eq. (3), which drives weak-KC identification and the CDS score. The assumption is known to be a simplification in educational measurement and is not validated for LLMs.
  • domain assumption KC annotations produced by the advanced LLM are reliable, mutually exclusive, collectively exhaustive, and appropriately granular.
    Section 3.1 describes LLM-based coarse and constrained annotation with optional expert involvement. The entire diagnostic and synthesis pipeline depends on these annotations, but no human agreement or error analysis is reported. The paper themselves note in Limitations that annotation 'involves significant randomness and subjectivity'.
  • domain assumption Data with more KCs have higher complexity and comprehensiveness, and data containing low-frequency or low-accuracy KCs are more effective for targeting weaknesses.
    Stated as a hypothesis in Section 3.3, leading to the CDS score definition. This heuristic is not derived from first principles, and its validity is only indirectly supported by the selection experiment.
  • domain assumption The advanced LLM Qwen2-72B-Instruct can generate accurate and diverse synthetic data, and its diagnostic analyses are informative.
    The method relies on the advanced model for KC annotation, diagnosis, data generation, augmentation, and Stage 1 scoring. No independent validation of the generated data quality is provided beyond the final benchmark numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory." pith.science (2026). https://pith.science/paper/CZQMAOS7

@misc{pith2026250107674,
  author       = {Pith},
  title        = {Pith review of: CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZQMAOS7}},
  note         = {Machine review of arXiv:2501.07674}
}
read the original abstract

Large Language Models (LLMs) have achieved significant advancements, but the increasing complexity of tasks and higher performance demands highlight the need for continuous improvement. Some approaches utilize synthetic data generated by advanced LLMs based on evaluation results to train models. However, conventional evaluation methods fail to provide detailed, fine-grained profiles of LLMs, limiting their guidance for data synthesis. In this paper, we introduce the Cognitive Diagnostic Synthesis (CDS) method, which incorporates a diagnostic process inspired by Cognitive Diagnosis Theory (CDT) to refine evaluation results and characterize model profiles at the knowledge component level. Based on these diagnostics, we propose two diagnosis-synthesis strategies for weakness-targeted data synthesis. Additionally, we present an enhanced data augmentation and selection pipeline to improve the quality and diversity of synthesized data. Our experiments with several open-source models show significant improvements across multiple benchmarks, achieving up to 6.00% improvement in code generation, 13.10% in mathematical reasoning, and 5.43% in academic exams. Code and data are available on GitHub.

Figures

Figures reproduced from arXiv: 2501.07674 by the authors.

Figure 1
Figure 1. The math problem assesses Geometry, Mixed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of CDS method. the accuracy and frequency of each KC, identifying weakly mastered KCs as those with low accuracy or low frequency, denoted as Kw. To address these weaknesses, we use Ma to generate data targeted at these weaknesses, based on Kw. The process is detailed in Algorithm 1. Algorithm 1 Global Strategy Require: A set of KCs K, an advanced model Ma, accuracy threshold δa, frequency threshold δf … view at source ↗
Figure 3
Figure 3. Prompt for KC Annotation of Math Task (Stage 1). [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Prompt for KC Annotation of Coding Task (Stage 1). [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Prompt for KC Annotation of Math Task (Stage 2). [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Prompt for KC Annotation of Coding Task (Stage 2). [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Prompt for Fine-grained Diagnosis of Math Task. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Prompt for Fine-grained Diagnosis of Coding Task. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Prompt for Fine-grained Synthesis of Math Task. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Prompt for Fine-grained Synthesis of Coding Task. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Prompt for Global Synthesis of Math Task. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Prompt for Global Synthesis of Coding Task. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Fusion Augmentation of Math Task. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Prompt for Fusion Augmentation of Coding Task. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Prompt for Rewriting Augmentation of Math Task. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Prompt for Rewriting Augmentation of Coding Task. [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Prompt for Scoring of Math Task [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Prompt for Scoring of Coding Task [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 17 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenn...

    work page 2023

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  5. [5]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. https://api.semanticscholar.org/CorpusID:264813981 Learning from mistakes makes llm better reasoner . ArXiv, abs/2310.20689

    arXiv 2023

  6. [6]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. https://api.semanticscholar.org/CorpusID:237142385 Program synthesis with large language models . ArXiv, abs/2108.07732

    arXiv 2021

  7. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, ...

    arXiv 2023

  8. [8]

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023 b . https://api.semanticscholar.org/CorpusID:259095491 Benchmarking foundation models with language-model-as-an-examiner . ArXiv, abs/2306.04181

    arXiv 2023

Show all 56 references
  1. [9]

    Ricardo J. G. B. Campello, Davoud Moulavi, and J \"o rg Sander. 2013. https://api.semanticscholar.org/CorpusID:32384865 Density-based clustering based on hierarchical density estimates . In Pacific-Asia Conference on Knowledge Discovery and Data Mining

    2013

  2. [10]

    Haowen Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan YangGong, and Junbo Jake Zhao. 2023 a . https://api.semanticscholar.org/CorpusID:258715090 Maybe only 0.5\ ArXiv, abs/2305.09246

    2023 arXiv

  3. [12]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023 b . Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701

    2023 arXiv

  4. [13]

    Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \'e , Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick R...

    2021 arXiv

  5. [14]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://api.semanticscholar.org/CorpusID:239998651 Training verifiers to solve mat...

    2021 arXiv

  6. [15]

    Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li

    Haixing Dai, Zheng Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, W. Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. https://api.semanticscholar.org/CorpusID:2576319...

    2023

  7. [16]

    Jimmy De La Torre. 2009. Dina model and parameter estimation: A didactic. Journal of educational and behavioral statistics, 34(1):115--130

    2009

  8. [17]

    Jimmy de la Torre. 2011. https://api.semanticscholar.org/CorpusID:120801311 The generalized dina model framework. Psychometrika, 76:179--199

    2011

  9. [18]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  10. [19]

    Yao Fu, Hao-Chun Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. https://api.semanticscholar.org/CorpusID:256390607 Specializing smaller language models towards multi-step reasoning . ArXiv, abs/2301.12726

    2023 arXiv

  11. [20]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. https://api.semanticscholar.org/CorpusID:235458009 Lora: Low-rank adaptation of large language models . ArXiv, abs/2106.09685

    2021 arXiv

  12. [21]

    Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. 2024 a . https://arxiv.org/abs/2403.02333 Key-point-driven data synthesis with its enhancement on mathematical reasoning . Preprint, arXiv:2403.02333

    2024 arXiv

  13. [22]

    Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. 2024 b . https://api.semanticscholar.org/CorpusID:267658021 Mustard: Mastering uniform synthesis of theorem and proof data . ArXiv, abs/2402.08957

    2024 arXiv

  14. [23]

    Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. 2023. https://api.semanticscholar.org/CorpusID:265281298 Camels in a changing climate: Enhancing lm adap...

    2023 arXiv

  15. [24]

    Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. https://api.semanticscholar.org/CorpusID:266902900 The impact of reasoning step length on large language models . ArXiv, abs/2401.04925

    2024 arXiv

  16. [25]

    Junker and Klaas Sijtsma

    Brian W. Junker and Klaas Sijtsma. 2001. https://api.semanticscholar.org/CorpusID:53456887 Cognitive assessment models with few assumptions, and connections with nonparametric item response theory . Applied Psychological Measurement, 25:258 -- 272

    2001

  17. [26]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    2015 arXiv

  18. [27]

    Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. https://api.semanticscholar.org/CorpusID:211987786 Data augmentation using pre-trained transformer models . ArXiv, abs/2003.02245

    2020 arXiv

  19. [28]

    Mahoney, Kurt Keutzer, and Amir Gholami

    Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Krishna Anumanchipalli, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. 2024. https://api.semanticscholar.org/CorpusID:268666980 Llm2llm: Boosting llms with novel iterative data enhance...

    2024

  20. [29]

    Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. 2024 a . https://api.semanticscholar.org/CorpusID:268264074 Common 7b language models already possess strong math capabilities . ArXiv, abs/2403.04706

    2024 arXiv

  21. [30]

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. 2024 b . https://...

    2024 arXiv

  22. [31]

    Haoran Li, Yiran Liu, Xingxing Zhang, Wei Lu, and Furu Wei. 2023. https://api.semanticscholar.org/CorpusID:264405905 Tuna: Instruction tuning using feedback from large language models . In Conference on Empirical Methods in Natural Language Processing

    2023

  23. [32]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024 c . https://api.semanticscholar.org/CorpusID:268063753 Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers . In Annual Meeting of the Association for Co...

    2024

  24. [33]

    Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, and Jun Zhao. 2024. https://api.semanticscholar.org/CorpusID:272770459 Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks . ArXiv, abs/2409.13203

    2024 arXiv

  25. [34]

    Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. 2023. https://api.semanticscholar.org/CorpusID:258832686 Logicot: Logical chain-of-thought instruction-tuning data collection with gpt-4 . In Conference on Empirical Methods in Natural Language Processing

    2023

  26. [35]

    Qi Liu. 2021. Towards a new generation of cognitive diagnosis. In IJCAI, pages 4961--4964

    2021

  27. [36]

    Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. https://arxiv.org/abs/2309.04564 When less is more: Investigating data pruning for pretraining llms at scale . Preprint, arXiv:2309.04564

    2023 arXiv

  28. [37]

    Sein Minn. 2022. https://api.semanticscholar.org/CorpusID:246674363 Ai-assisted knowledge assessment techniques for adaptive learning environments . Comput. Educ. Artif. Intell., 3:100050

    2022

  29. [38]

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023. https://api.semanticscholar.or...

    2023 arXiv

  30. [39]

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. https://api.semanticscholar.org/CorpusID:267897618 Orca-math: Unlocking the potential of slms in grade school math . ArXiv, abs/2402.14830

    2024 arXiv

  31. [40]

    Steven Moore, Robin Schmucker, Tom Mitchell, and John C. Stamper. 2024. https://api.semanticscholar.org/CorpusID:270199338 Automated generation and tagging of knowledge components from multiple-choice questions . ArXiv, abs/2405.20526

    2024 arXiv

  32. [41]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. https://doi.org/10.1109/SC41405.2020.00024 Zero: memory optimizations toward training trillion parameter models . In Proceedings of the International Conference for High Performance Computing, Networking, ...

    2020 arXiv

  33. [42]

    Nils Reimers and Iryna Gurevych. 2019. https://api.semanticscholar.org/CorpusID:201646309 Sentence-bert: Sentence embeddings using siamese bert-networks . In Conference on Empirical Methods in Natural Language Processing

    2019

  34. [43]

    Rupp, Jonathan L Templin, and Robert Henson

    Andre A. Rupp, Jonathan L Templin, and Robert Henson. 2010. https://api.semanticscholar.org/CorpusID:60997926 Diagnostic measurement: Theory, methods, and applications

    2010

  35. [44]

    Ozan Sener and Silvio Savarese. 2017. https://api.semanticscholar.org/CorpusID:3383786 Active learning for convolutional neural networks: A core-set approach . arXiv: Machine Learning

    2017

  36. [45]

    Cox, Yiming Yang, and Chuang Gan

    Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. 2023. https://api.semanticscholar.org/CorpusID:258479665 Principle-driven self-alignment of language models from scratch with minimal human supervision . ArXiv, abs...

    2023 arXiv

  37. [46]

    Tatsuoka

    Kikumi K. Tatsuoka. 1983. https://api.semanticscholar.org/CorpusID:145769079 Rule space: An approach for dealing with misconceptions based on item response theory . Journal of Educational Measurement, 20:345--354

    1983

  38. [47]

    Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. 2019. https://api.semanticscholar.org/CorpusID:211734128 Neural cognitive diagnosis for intelligent education systems . In AAAI Conference on Artificial Intelligence

    2019

  39. [48]

    Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. 2024. https://api.semanticscholar.org/CorpusID:274982380 Drt-o1: Optimized deep reasoning translation via long chain-of-thought . ArXiv, abs/2412.17498

    2024 arXiv

  40. [49]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. https://api.semanticscholar.org/CorpusID:259108263 How far can camels go? exploring the stat...

    2023 arXiv

  41. [50]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. https://api.semanticscholar.org/CorpusID:254877310 Self-instruct: Aligning language models with self-generated instructions . In Annual Meeting of the Associa...

    2022

  42. [51]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. https://api.semanticscholar.org/CorpusID:258298159 Wizardlm: Empowering large language models to follow complex instructions . ArXiv, abs/2304.12244

    2023 arXiv

  43. [52]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  44. [53]

    Jiahao Ying, Mingbao Lin, Yixin Cao, Wei Tang, Bo Wang, Qianru Sun, Xuanjing Huang, and Shuicheng Yan. 2024. https://api.semanticscholar.org/CorpusID:270870583 Llms-as-instructors: Learning from errors toward automating model improvement . ArXiv, abs/2407.00497

    2024 arXiv

  45. [54]

    Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. 2024. https://api.semanticscholar.org/CorpusID:267547744 In-context principle learning from mistakes . ArXiv, abs/2402.05403

    2024 arXiv

  46. [55]

    Xiaotian Zhang, Chun yan Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. https://api.semanticscholar.org/CorpusID:258833562 Evaluating the performance of large language models on gaokao benchmark . ArXiv, abs/2305.12474

    2023 arXiv

  47. [56]

    Haokun Zhao, Haixia Han, Jie Shi, Chengyu Du, Jiaqing Liang, and Yanghua Xiao. 2024. https://api.semanticscholar.org/CorpusID:269148977 Cem: A data-efficient method for large language models to continue evolving from mistakes

    2024

  48. [57]

    Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://api.semanticscholar.org/CorpusID:258822910 Lima: Less is more for alignment . ArX...

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.