Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Dynamic Skill Adaptation for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that automatically generated textbook and exercise data, ordered by a prerequisite skill graph and updated dynamically during tuning, lets 7B and 13B open models beat ChatGPT on pre-calculus.

desk verdict A worthwhile pipeline with one clean ordering result, but the central effectiveness claim is statistically under-powered and the dynamic-training ablation is confounded by data volume. read the letter →

arxiv 2412.19361 v1 pith:LMSTSOG4 submitted 2024-12-26 cs.CL

Jiaao Chen , Diyi Yang This is my paper · ORCID
classification cs.CL
keywords DynamicSkillAdaptationgraphprerequisiteorderingcurriculumlearninginstructiontuningsyntheticdatagenerationmathreasoningsocialstudies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a large language model can be taught a new complex skill more effectively when its fine-tuning data is organized like a human curriculum rather than served as a static random mixture. The authors build a skill graph that decomposes calculus and social studies into prerequisite-ordered sub-skills, have GPT-4 write a textbook-style passage for each node and exercise problems that explicitly cite the skills being used, pretrain on the passages in graph order, and then instruction-tune on the exercises while continuously revising the corpus: easy examples are composed into harder ones, hard examples are used to generate more of the same, and error-laden examples are removed. On the Pre-Calculus subset of MATH, the resulting DSA-Mistral-7b reaches 18.6, above DeepSeekMATH-Inst-7b's 16.8 and ChatGPT's 16.1, while DSA-LLAMA2-70b reaches 22.6 against GPT-4's 29.8; on a collected social-studies exam set, DSA-LLAMA2-70b reaches 87.9 against GPT-4's 95.0. The broader claim is that skill-level curriculum ordering plus dynamic data curation is a general mechanism for adapting LLMs to specialized domains, with the order of skills and the filtering of data mattering beyond what static synthetic-data instruction tuning achieves. A reader should care because this points to a training-data organization principle that can be applied to any domain with a decomposable syllabus, potentially reducing dependence on human-curated expert corpora.

What carries the argument

The load-bearing object is the skill graph, a directed graph whose nodes are sub-skills and whose edges point from prerequisite lower-level skills to higher-level skills; it is constructed by merging a human-curated academic syllabus with GPT-4's recursive decomposition of the target skill, yielding 4,907 math nodes across 14 levels and 990 social-studies nodes across 9 levels. Around the graph sit two further mechanisms. The first is automatic data generation: for every node, GPT-4 writes a textbook-style passage linking the skill to its predecessors and providing homework, and also writes exercise problems whose solutions must explicitly cite the invoked skills, with self-consistency used to improve answer reliability. The second is dynamic training: after pretraining on textbooks in graph order, instruction tuning computes each exercise's average loss and loss variance across epochs, classifies it as error, hard-to-learn, easy-to-learn, or ambiguous, and rebuilds the corpus by discarding errors, generating more examples like the hard ones, and composing easy examples into harder ones. The graph supplies the curriculum order; the categorization loop supplies the adaptation.

What would settle it

Run the same generated textbook and exercise corpus through the same pipeline, but order the pretraining by the chapter sequence of a standard college calculus textbook or by any other plausible syllabus, and compare Pre-Calculus accuracy; if the alternative order matches or beats the merged skill-graph order, the specific graph is not what carries the reported gains. A complementary check is to have independent human instructors rate prerequisite dependencies on a sample of graph edges and see whether low-agreement edges are the ones whose reordering changes performance.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the order in which generated training data is presented, and the way that data changes during training, are as important as the data itself. Concretely, DSA decomposes a target skill into sub-skills and arranges them in a directed graph where edges point from prerequisites to higher-level skills; it then generates textbook-like passages for pre-training and exercise-like problems for instruction tuning, constraining the answers to explicitly invoke learned sub-skills and applying self-consistency with k = 3 to improve answer quality. During instruction tuning, each example's average loss and loss variance across epochs assign it to one of four categories (error, hard-to-learn, easy-to-learn, ambiguous); error examples are dropped, hard examples seed the generation of similar new problems, and easy examples are compositionally augmented into harder problems. The reported result is that this pipeline lifts LLAMA2 and Mistral models far above their raw baselines, above prior synthetic-instruction and human-curated math-specialized baselines, and in the 7B and 13B cases above ChatGPT on the Pre-Calculus evaluation; the models also generalize to MATH, GSM8K, and a novel arithmetic task. What the authors are trying to establish is that mimicking the organization, rehearsal, elaboration, and monitoring strategies of human teaching is a viable route to specialized skill acquisition in LLMs.

Load-bearing premise

The ordering benefit depends on the skill graph's prerequisite edges actually matching how the sub-skills build on each other, a premise the paper does not validate against human experts or an independent taxonomy.

Editorial extensions

If this is right

  • The same generated corpus produces substantially higher Pre-Calculus accuracy when presented lower-to-higher in the skill graph (8.2) than reversed (3.2) or in three random orders (3.0-4.8), so the ordering itself, not the data alone, carries much of the gain.
  • The framework's components are additive in the LLAMA2-7b ablation, moving from 0.8 to 5.2 with textbooks, 8.2 with graph order, 12.4 with exercise tuning, and 16.5 with dynamic updates.
  • A 7B model tuned this way (DSA-Mistral-7b, 18.6) and a 13B model (DSA-LLAMA2-13b, 18.8) exceed ChatGPT's 16.1 on Pre-Calculus, while the 70B model (22.6) approaches GPT-4's 29.8.
  • Calculus-adapted DSA models generalize to other math evaluations: DSA-Mistral-7b scores 43.5 on MATH, 83.8 on GSM8K, and 58.0 on the constructed novel-operation arithmetic task.
  • Dynamic updates add a substantial final increment (from 12.4 to 16.5 for LLAMA2-7b on Pre-Calculus), indicating that filtering, augmenting, and composing data during tuning matters beyond a fixed corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The prerequisite ordering, not the specific GPT-4 output, is likely the active ingredient: the graph-ordered textbook-only model already accounts for most of the improvement over the base, which suggests the framework would transfer to any domain with a decomposable syllabus and any sufficiently capable data generator.
  • Inference: Because the dynamic loop operates on per-example loss and variance, it should also improve existing static synthetic-instruction datasets without regenerating the corpus, as long as the tuning run observes each example for several epochs.
  • Inference: DSA orders skills instead of individual examples, so it is complementary to example-level pacing methods and could be stacked with them for additional gains.
  • Inference: A decisive extension would be to replace the merged human-syllabus graph with other plausible prerequisite structures, for example a college textbook's chapter sequence, on the same generated corpus; if gains persist, the principle is robust to the exact graph.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dynamic Skill Adaptation (DSA), a framework for adapting LLMs to specialized skills such as calculus and social studies. DSA builds a skill graph by merging the IXL human syllabus with GPT-4 recursive decomposition of the target skill into sub-skills with dependency edges, then generates textbook-like descriptions for continued pre-training on each skill and exercise-like problems for instruction tuning, with answers grounded in named skills and refined via self-consistency. During instruction tuning, DSA categorizes examples by their per-example loss mean and variance across epochs into error, hard, easy, and ambiguous groups, filtering error examples, generating additional hard examples, and composition-augmenting easy examples. Experiments on LLAMA2-7/13/70b and Mistral-7b report accuracy on the Pre-Calculus subset of MATH and on a web-assembled social studies set, plus generalization results on MATH, GSM8K, and a constructed arithmetic task. The central reported findings are large gains over the base LLMs and a same-data ordering effect: training on the skill graph from lower to higher levels outperforms reversed and random orders of the same textbook corpus.

Significance. The paper addresses a real and current problem, namely how to structure automatically generated training data for skill adaptation rather than mixing it randomly, and the proposed pipeline is fully automatic. Credit is due for the same-data ordering ablation in Table 3, for evaluating on external benchmarks (MATH Pre-Calculus, GSM8K, MATH) plus a purpose-built arithmetic task rather than fitting to the test sets, and for the explicit data-leakage sanity checks in Section 6. These design choices substantially reduce the risk of circular evaluation. If the results hold, the skill-level curriculum perspective is a useful complement to example-level curriculum learning. However, the headline comparisons against strong baselines rest on single runs without statistical support, the dynamic-training ablation is confounded with added data volume, and the skill graph is not validated, so the contributions are plausible but not yet established at the strength the paper claims.

major comments (4)
  1. [§4.2, Table 2] The caption of Table 2 states that DSA is 'significantly better' than open-sourced baselines and 'even better than ChatGPT', but the paper reports no error bars, no number of random seeds, and no significance tests for any result. The Pre-Calculus subset size is not reported; if it is on the order of a few hundred problems, as is typical for MATH subject splits, a difference of 1.8-2.5 points (DSA-Mistral-7b 18.6 versus DeepSeekMATH-Inst-7b 16.8 and ChatGPT 16.1) is roughly one standard error and within sampling noise. The large gains over the untuned LLAMA2 and Mistral bases are credible, but the abstract-level and caption-level superiority claims over strong baselines need multiple seeds with error bars and a significance test, or a more cautious formulation.
  2. [§3.3, Table 4] The last row of Table 4 (12.4 to 16.5 on Pre-Calculus) is attributed to dynamic training, but applying the dynamic mechanism also adds newly generated data (expanded hard examples and composition-augmented easy examples) to the corpus, so the gain is consistent with a token-count effect rather than with adaptivity. A control that matches the total number of training tokens without loss/variance-based selection is required to support the dynamic-training claim. In addition, Section 3.3 says the baseline loss Lb and variance sigma_b are computed 'after fine-tuning with constructed error examples for three epochs', but the construction of those error examples is never described, and the update schedule is inconsistent between Section 3.3 ('after every three epochs') and Section 4.1 ('after every epoch'). These details must be clarified before the mechanism is reproducible.
  3. [§3.1, Table 3] The skill graph is the load-bearing artifact of the method, but its correctness is never validated: the merge of the IXL syllabus with GPT-4 decompositions is treated as ground truth without expert annotation or comparison against an independent taxonomy. Table 3 shows that lower-to-higher ordering beats reversed and random orderings of the same graph, which is evidence that the ordering encodes some useful structure, but it does not establish that the specific dependency edges are correct, and the random-order conditions show considerable spread (e.g., 5.8-9.5 for LLAMA2-70b) while each main ordering condition is run only once. The claim should be supported by multiple seeds and by at least one external check of the graph structure, or reframed as 'an inductively constructed ordering helps' rather than 'the graph correctly represents prerequisite dependencies'.
  4. [§4.1] The social studies evaluation set is assembled from two web quiz collections with no documented inclusion criteria, grade-level distribution, or filtering procedure, and all social studies headline results (e.g., DSA-LLAMA2-70b 87.9 versus ChatGPT 83.5) rest on this nonstandard set. The authors should either report a rigorous construction protocol for the set, characterize its composition, or corroborate the social studies claim on an established benchmark before the claim that DSA improves social study skills is accepted.
minor comments (6)
  1. [§4.1, Table 5] WizardMATH-7b's Pre-Calculus accuracy is 2.5 in Table 2 but 2.8 in Table 5 for what appears to be the same setting; this discrepancy should be reconciled.
  2. [Abstract and title page] 'Human syllables' in the abstract and Section 1 should be 'human syllabi', and the title page contains the typo 'Georgia Institutr of Technology'.
  3. [§4.1] The number of test examples in the Pre-Calculus subset of MATH and in the constructed arithmetic task (described only as '200 new mathematical operations') is not reported; this information is needed to interpret the point estimates.
  4. [§3.3, Algorithm 1] The convergence criterion of the 'while No convergence' loop in Algorithm 1 is not defined, and the number of dynamic update iterations used in the experiments is not stated; please give the exact schedule.
  5. [§3.3] The four-way categorization in Section 3.3 would benefit from a cleaner partition of the (loss, variance) plane; as written, the easy-to-learn condition is stated redundantly ('Lb >= loss and Laverage >= loss') and the boundaries of the four categories are hard to check.
  6. [§6] The leakage argument that GPT-4's 29.8% on Pre-Calculus 'indicates that the data is less likely to be contaminated' is suggestive but not conclusive, since low accuracy can also reflect task difficulty; the reported exact-match check is the stronger evidence and could be described more prominently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DSA's skill graph, data generation, and dynamic training are built from external sources and internal training signals, while evaluation uses held-out benchmarks; no prediction reduces to a fitted input by construction.

full rationale

The paper's derivation chain is: (1) construct a skill graph by merging the IXL human syllabus with GPT4 recursive decomposition of the target skill; (2) generate textbook and exercise data with GPT4, regularizing outputs to reference skills; (3) pre-train on textbooks in lower-to-higher skill order; (4) instruction-tune on exercises while dynamically filtering or augmenting based on per-example training loss and variance; (5) evaluate on external held-out sets (MATH Pre-Calculus, collected social-studies exams, GSM8K, and a synthetic arithmetic task). Each component's input is external or an internal training signal, not the evaluation quantity. The skill-graph ordering is a design choice validated through ablations that shuffle or reverse the order; those ablations do not define the evaluation outcome. The several self-citations (Chen et al. 2023, Ouyang et al. 2023, Qin et al. 2023, Zhu et al. 2023, Ziems et al. 2024) appear as related-work context or as references for techniques already described in the paper, and they are not load-bearing for the central effectiveness claim. The paper also reports a contamination check—GPT4 only reaches 29.8% on Pre-Calculus and no exact match was found between test and generated training data—which weakens any concern that the GPT4-generated training data encodes the test answers. Remaining issues such as missing error bars, single-run point estimates, possible token-count confounds in the dynamic-training ablation, and unvalidated skill-graph correctness are statistical or validity risks rather than circular reasoning. No step reduces by construction to its own output.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on several domain assumptions: the skill graph's correctness, the quality of GPT4-generated content, the validity of the evaluation sets, and the meaningfulness of the dynamic data categories. No free parameters are fitted to the test set; the listed parameters are hand-chosen design settings in the generation and dynamic-training pipeline.

free parameters (3)
  • Number of epochs for baseline loss/variance computation = 3
    Section 3.3 computes Lb and sigma_b after 'fine-tuning with constructed error examples for three epochs'. This choice is arbitrary and determines the thresholds for categorizing data as error, hard, or easy.
  • GPT4 sampling temperatures for data generation = T=0.5 (textbook), T=0.1 (exercise), T=1.0 (dynamic augmentation)
    Section 4.1. The temperatures are chosen by hand and affect data quality and diversity.
  • Self-consistency sample count k = 3
    Section 4.1 applies self-consistency with k=3 to generate answers; the choice is not justified.
assumptions (4)
  • domain assumption IXL human syllabus and GPT4 recursive decomposition produce a valid prerequisite graph for calculus and social studies.
    Section 3.1 constructs the skill graph by merging the IXL syllabus and GPT4 decomposition; no validation against external skill taxonomies is provided.
  • domain assumption GPT4-generated textbook and exercise data are sufficiently accurate and free of errors to serve as training data (errors are only partially filtered during dynamic training).
    Section 3.2 relies on GPT4 for all content generation; the only quality control is self-consistency (k=3) for answers and dynamic filtering of high-loss examples.
  • domain assumption The Pre-Calculus subset of MATH and the web-collected social studies questions are valid measures of the adapted skills.
    Section 4.1 defines evaluation sets; no evidence of alignment with the skill graph or grade levels is given.
  • ad hoc to paper The loss/variance categorization into error, hard, easy, and ambiguous data isolates meaningful training dynamics.
    Section 3.3 defines arbitrary thresholds (average loss, average variance, baselines) to split data; the categories are not validated against downstream task performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Skill Adaptation for Large Language Models." pith.science (2026). https://pith.science/paper/LMSTSOG4

@misc{pith2026241219361,
  author       = {Pith},
  title        = {Pith review of: Dynamic Skill Adaptation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMSTSOG4}},
  note         = {Machine review of arXiv:2412.19361}
}
read the original abstract

We present Dynamic Skill Adaptation (DSA), an adaptive and dynamic framework to adapt novel and complex skills to Large Language Models (LLMs). Compared with previous work which learns from human-curated and static data in random orders, we propose to first automatically generate and organize the training data by mimicking the learning pathways of human and then dynamically tailor the training data based on the training dynamics. Specifically, inspired by the learning structures and teaching strategies in the human education system, we first construct a skill graph by decomposing complex skills into sub-skills and arranging them based on their dependencies in human syllables. For every skill, we utilize LLMs to generate both textbook-like data which contains detailed descriptions of skills for pre-training and exercise-like data which targets at explicitly utilizing the skills to solve problems for instruction-tuning. Furthermore, during the instruction-tuning, we dynamically update the training data which down-weight easy-to-learn examples, generate more complex examples, and filter out data with errors. Experiments on large language models such as LLAMA and Mistral demonstrate the effectiveness of our proposed methods in adapting math reasoning skills and social study skills.

Figures

Figures reproduced from arXiv: 2412.19361 by the authors.

Figure 1
Figure 1. The overall process of our Dynamic Skill Adaptation framework. For a given complex [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The accuracy on Pre-Calculus evaluation set of every intermediate step when learning [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. A sub-skill graph in our constructed Calculus skill graph. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A sub-skill graph in our constructed Social Study skill graph. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms

    Simone Balloccu, Patr \' cia Schmidtov \'a , Mateusz Lango, and Ond r ej Du s ek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. arXiv preprint arXiv:2402.03927, 2024

    arXiv 2024

  3. [3]

    Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets

    Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy nlp datasets. arXiv preprint arXiv:2302.13959, 2023

    work page Pith review arXiv 2023

  4. [4]

    Curriculum learning

    Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp.\ 41--48, 2009

    work page 2009

  5. [5]

    On education

    Harry Brighouse. On education. Routledge, 2006

    work page 2006

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  7. [7]

    Skills-in-context prompting: Unlocking compositionality in large language models

    Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. Skills-in-context prompting: Unlocking compositionality in large language models. arXiv preprint arXiv:2308.00304, 2023

    arXiv 2023

  8. [8]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher R \'e . Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 66 references
  1. [9]

    Adapting large language models via reading comprehension

    Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530, 2023

    2023 arXiv

  2. [10]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022

    2022 arXiv

  3. [11]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022

    arXiv

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [14]

    Description of learning to learn in human subjects

    Carl P Duncan. Description of learning to learn in human subjects. The American Journal of Psychology, 73 0 (1): 0 108--114, 1960

    1960

  7. [15]

    Faith and fate: Limits of transformers on compositionality

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654, 2023

    2023 arXiv

  8. [16]

    Irreducible curriculum for language model pretraining

    Simin Fan and Martin Jaggi. Irreducible curriculum for language model pretraining. arXiv preprint arXiv:2310.15389, 2023

    2023 arXiv

  9. [17]

    Citing: Large language models create curriculum for instruction tuning

    Tao Feng, Zifeng Wang, and Jimeng Sun. Citing: Large language models create curriculum for instruction tuning. arXiv preprint arXiv:2310.02527, 2023

    2023 arXiv

  10. [18]

    Mathematical capabilities of chatgpt

    Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Data contamination quiz: A tool to detect and estimate contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Data contamination quiz: A tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.06233, 2023

    2023 arXiv

  12. [20]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  13. [21]

    Towards a comprehensive theory of human learning

    Peter Jarvis. Towards a comprehensive theory of human learning. Routledge, 2012

    2012

  14. [22]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  15. [23]

    Self-paced curriculum learning

    Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  16. [24]

    Decomposed prompting: A modular approach for solving complex tasks

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022

    2022 arXiv

  17. [25]

    Charles Kivunja. Do you want your students to be job-ready with 21st century skills? change pedagogies: A pedagogical paradigm shift from vygotskyian social constructivism to critical thinking, problem solving and siemens' digital connectivism. International journal of higher ...

    2014

  18. [26]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022

    2022 arXiv

  19. [27]

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064, 2024

    2024 arXiv

  20. [28]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  21. [29]

    Yoda: Teacher-student progressive learning for language models

    Jianqiao Lu, Wanjun Zhong, Yufei Wang, Zhijiang Guo, Qi Zhu, Wenyong Huang, Yanlin Wang, Fei Mi, Baojun Wang, Yasheng Wang, et al. Yoda: Teacher-student progressive learning for language models. arXiv preprint arXiv:2401.15670, 2024

    2024 arXiv

  22. [30]

    Wizardcoder: Empowering code large language models with evol-instruct

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  23. [31]

    Teacher--student curriculum learning

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher--student curriculum learning. IEEE transactions on neural networks and learning systems, 31 0 (9): 0 3732--3740, 2019

    2019

  24. [32]

    Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version)

    S \"o ren Mindermann, Muhammed Razzak, Winnie Xu, Andreas Kirsch, Mrinank Sharma, Adrien Morisot, Aidan N Gomez, Sebastian Farquhar, Jan Brauner, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version). arXiv pre...

    2021 arXiv

  25. [33]

    Cross-task generalization via natural language crowdsourcing instructions

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3470--348...

    2022 doi

  26. [34]

    Rethinking the social studies curriculum in the context of globalization: Education for global citizenship in the us

    John P Myers. Rethinking the social studies curriculum in the context of globalization: Education for global citizenship in the us. Theory & Research in Social Education, 34 0 (3): 0 370--394, 2006

    2006

  27. [35]

    Codegen: An open large language model for code with multi-turn program synthesis

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022

    2022 arXiv

  28. [36]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  29. [37]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 2...

    2022

  30. [38]

    Compositional data augmentation for abstractive conversation summarization

    Siru Ouyang, Jiaao Chen, Jiawei Han, and Diyi Yang. Compositional data augmentation for abstractive conversation summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1471--1488, 2023

    2023

  31. [39]

    Automatic curriculum learning for deep rl: A short survey

    R \'e my Portelas, C \'e dric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020

    2003 arXiv

  32. [40]

    Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023

    2023 arXiv

  33. [41]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, pp.\ 9, 2019

    2019

  34. [42]

    Conformal nucleus sampling

    Shauli Ravfogel, Yoav Goldberg, and Jacob Goldberger. Conformal nucleus sampling. arXiv preprint arXiv:2305.02633, 2023

    2023 arXiv

  35. [43]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  36. [44]

    Data parameters: A new family of parameters for learning a differentiable curriculum

    Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. Advances in Neural Information Processing Systems, 32, 2019

    2019

  37. [45]

    When flue meets flang: Benchmarks and large pre-trained language model for financial domain

    Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. When flue meets flang: Benchmarks and large pre-trained language model for financial domain. arXiv preprint arXiv:2211.00083, 2022

    2022 arXiv

  38. [46]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  39. [47]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. ...

    2022 arXiv

  40. [48]

    Curriculum learning: A survey

    Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 130 0 (6): 0 1526--1565, 2022

    2022

  41. [49]

    Dataset cartography: Mapping and diagnosing datasets with training dynamics

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020

    2009 arXiv

  42. [50]

    Openmathinstruct-1: A 1.8 million math instruction tuning dataset

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402.10176, 2024

    2024 arXiv

  43. [51]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  44. [52]

    A survey on curriculum learning

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 4555--4576, 2021

    2021

  45. [53]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  46. [54]

    Generative ai for math: Part i--mathpile: A billion-token-scale pretraining corpus for math

    Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i--mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120, 2023

    2023 arXiv

  47. [55]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  48. [56]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022

    2022

  49. [57]

    The teaching of learning strategies

    Claire E Weinstein and Richard E Mayer. The teaching of learning strategies. In Innovation abstracts, volume 5, pp.\ n32. ERIC, 1983

    1983

  50. [58]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024

    2024

  51. [59]

    Curriculum learning for natural language understanding

    Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 6095--6104, 2020

    2020

  52. [60]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  53. [61]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  54. [62]

    Dyval: Graph-informed dynamic evaluation of large language models

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Graph-informed dynamic evaluation of large language models. arXiv preprint arXiv:2309.17167, 2023

    2023 arXiv

  55. [63]

    Can large language models transform computational social science? Computational Linguistics, pp.\ 1--55, 2024

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, pp.\ 1--55, 2024

    2024

  56. [64]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  57. [65]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  58. [66]

    q>D)o6-_ Ű -1 R!UR& e6q1,PRp ^< 5T!v B NA 5 A,g7ʹaM 'c

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.