How Well Do LLMs Predict Prerequisite Skills? Zero-Shot Comparison to Expert-Defined Concepts

Marie-H\'el\`ene Abel; Ngoc Luyen Le

REVIEW 3 major objections 5 minor 29 references

How Well Do LLMs Predict Prerequisite Skills? Zero-Shot Comparison to Expert-Defined Concepts

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can infer prerequisite skill relationships from skill names and natural-language descriptions alone, with the best zero-shot model reaching $F1_{\mathrm{BERT}} = 0.8347$ against expert-defined ESCO prerequisites.

desk verdict ESCO-PrereqSkill could be a useful testbed, but the ground-truth construction is unvalidated and may not measure prerequisites at all. read the letter →

arxiv 2507.18479 v1 pith:UJBEETCF submitted 2025-07-24 cs.IR

Ngoc Luyen Le , Marie-Hélène Abel This is my paper · ORCID

classification cs.IR

keywords prerequisiteskillpredictionzero-shotlearninglargelanguagemodelsESCOtaxonomysemanticsimilarityBERTScorebenchmarkdatasetgapanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that modern large language models can infer which skills a learner must master before attempting a more advanced skill, using only the skill's name and its natural-language description, with no task-specific fine-tuning. To test this, the authors build ESCO-PrereqSkill, a benchmark of 3,196 skills from the ESCO taxonomy paired with expert-defined prerequisite lists, and evaluate 13 LLMs with a standardized zero-shot prompt. The best model, LLaMA4-Maverick, reaches $F1_{\mathrm{BERT}} = 0.8347$, closely followed by Qwen2-72B, LLaMA4-Scout, and Gemini-2.0-Flash. The authors interpret this as evidence that general-purpose LLMs contain curriculum-like structural knowledge that can be tapped for scalable prerequisite modeling in personalized learning, tutoring, and skill-gap analysis.

What carries the argument

The load-bearing object is ESCO-PrereqSkill, a benchmark dataset built from the skills/competences pillar of the ESCO taxonomy, where each of 3,196 target skills carries a name, a textual description, and a ground-truth prerequisite list extracted through the inverse of ESCO's "isEssentialSkillFor" relation. Predictions are elicited by a standardized prompt that asks the model to act as an education expert and return a comma-separated list, then normalized and scored by Sentence-BERT cosine similarity and BERTScore. The machine that carries the argument is the combination of this expert-grounded reference set with the zero-shot protocol: because no model is fine-tuned, any agreement between predicted and reference lists is attributed to knowledge already encoded in pretraining.

What would settle it

Take a random sample of 100 pairs from ESCO-PrereqSkill and have independent education experts judge, without seeing the model predictions, whether each listed "prerequisite" is genuinely a skill a learner must master first; if expert agreement with the benchmark labels is low, the ground truth—and the F1 scores built on it—cannot support the paper's conclusion.

Watch

Extended reading notes

Core claim

The central discovery is that zero-shot LLM predictions align substantially with expert-defined prerequisite skills, despite the models never being trained on this task. The paper reports $F1_{\mathrm{BERT}}$ values above 0.81 for all thirteen tested models, with LLaMA4-Maverick posting the highest score at 0.8347. The authors also find that some models combine high $F1_{\mathrm{BERT}}$ with lower cosine semantic similarity, which they read as surface-level lexical overlap rather than deep conceptual alignment; models such as Claude-3-7-Sonnet and LLaMA4-Scout show a better balance between lexical and semantic agreement. The conclusion is that LLMs can, to a significant extent, recover expert-defined prerequisite structure from skill names and descriptions alone.

Load-bearing premise

The evaluation stands on the assumption that the ESCO-derived ground truth really consists of expert-defined skill-to-skill prerequisite links; if the inverse of "isEssentialSkillFor" does not actually produce prerequisite pairs, or if the ESCO taxonomy was largely present in the models' pretraining data, the benchmark measures the wrong target.

Editorial extensions

If this is right

Prerequisite modeling for curricula and skill-gap analysis can be performed without per-domain fine-tuning, using a model API plus the skill's name and description.
Several high-scoring models, including LLaMA4-Maverick, also respond in under two seconds on average, making real-time tutoring and adaptive learning systems feasible.
ESCO-PrereqSkill provides a reusable 3,196-skill test set for comparing future LLMs on prerequisite inference.
Because some models achieve high BERTScore but lower semantic similarity, evaluation of prerequisite prediction should report both lexical and semantic metrics rather than a single number.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the claim holds, expert-curated taxonomies could be refreshed semi-automatically, with LLM predictions flagging new or missing prerequisite links for human review and cutting the maintenance cost of frameworks like ESCO.
Beyond the paper: the metric discrepancy the paper reports suggests a testable extension, namely re-scoring predictions with human judgments of pedagogical necessity, since BERTScore alone may reward lexical proximity that is not actual prerequisite structure.
Beyond the paper: a clean contamination check would compare performance on skills whose descriptions were published after the models' training cutoffs; if accuracy drops sharply, the zero-shot result may partly reflect memorized ESCO content rather than reasoning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: useful idea, broken load-bearing part. The new asset is ESCO-PrereqSkill, a 3,196-skill benchmark with supposed expert-defined prerequisite links, plus a clean zero-shot comparison of 13 LLMs using a standardized prompt and BERTScore/semantic similarity. The evaluation protocol is well described, and the latency comparison is a nice practical touch. Novelty is moderate—prerequisite extraction and LLM link prediction are both established—but the specific dataset and model sweep are new.

The problem is the ground truth. Section IV-A says prerequisite links were extracted using "relational properties such as the inverse of isEssentialSkillFor." In ESCO, isEssentialSkillFor relates skills to occupations, not skills to skills. Its inverse gives occupation–skill membership, not prerequisite ordering. The paper gives no SPARQL query, no schema excerpt, no manual validation. That is not a reproducibility footnote; it is the target variable. The example in Table I makes the worry concrete: the four "prerequisites" for "Supervise correctional procedures" are nearly verbatim noun phrases from its description. If the labels are description fragments, then BERTScore F1 around 0.83 mainly measures description-to-description lexical overlap, not pedagogical reasoning.

Two more gaps. First, there are no baselines—no lexical baseline, no embedding-retrieval baseline, no random or co-occurrence baseline—so the LLMs' added value is unclear. Second, no contamination analysis; ESCO is public and very likely in pretraining corpora, so "zero-shot" might mean memorization. The authors also report no error bars or significance tests, and all models cluster in a narrow F1 band (0.81–0.83), so specific ranking claims are unsupported.

The good news: all of this is fixable. Clarify the ESCO extraction and validate a sample. Add a TF-IDF or Sentence-BERT retrieval baseline. Run a contamination probe, for example by asking models for ESCO IDs or descriptions. Show examples where predicted prerequisites are not lexically present in the description. The related-work coverage is adequate—PREREQ, GKROM, and recent LLM link-prediction papers are cited—so the literature placement is honest. If the benchmark is repaired, it could genuinely serve the tutoring-systems community.

My call: send to peer review, but expect major revision. The reader's REJECT is defensible; I would frame it as "reject in current form." I would not cite the dataset as-is until the ground truth is cleaned. This is a good reading-group case study in why benchmark construction is the whole game.

Referee Report

3 major / 5 minor

Summary. The manuscript investigates whether large language models (LLMs) can predict prerequisite skills in a zero-shot setting. The authors construct a benchmark, ESCO-PrereqSkill, from 3,196 ESCO skills whose prerequisite lists are extracted, per Section IV-A, through relational properties such as the inverse of isEssentialSkillFor. They query 13 LLMs with a standardized prompt containing each skill's name and description, then compare the generated comma-separated lists against the ESCO-derived ground truth using Sentence-BERT semantic similarity and BERTScore, and also report inference latency. The central empirical claim is that several models, notably LLaMA4-Maverick with F1_BERT = 0.8347, produce predictions that closely align with expert-defined prerequisite structures. The paper further argues that high-accuracy, low-latency models are promising for personalized learning and skill-gap analysis.

Significance. If the benchmark were a valid operationalization of expert-defined skill prerequisites, this would be a useful empirical contribution: the manuscript releases a public dataset, evaluates 13 current models under a fixed prompt protocol, and provides latency data relevant to deployment. The strengths of the manuscript are its scale (3,196 skills), the breadth of models, and the clear reporting of prompt and parsing conventions. However, the contribution depends entirely on the validity of the ground-truth extraction, and the manuscript provides no evidence that the extracted lists are genuine skill-to-skill prerequisite relations rather than occupation-skill co-occurrence or description-derived noun phrases. Because the evaluation target is unvalidated and no baselines or significance tests are reported, the stated conclusion about LLMs' prerequisite reasoning is not currently supported.

major comments (3)

[IV-A, Table I] The central construction of ESCO-PrereqSkill is not validated. In ESCO, isEssentialSkillFor relates a skill to an occupation (or an occupation to a skill); its inverse is an occupation-to-skill membership relation, not a skill-to-skill prerequisite ordering. The manuscript gives no SPARQL query, schema excerpt, or human validation showing that the extracted lists are prerequisites. The example in Table I reinforces this worry: the four listed prerequisites (Correctional procedures, Legal regulations in corrections, Staff management, Safety protocols) are near-verbatim noun phrases from the skill's own description, which is also part of the LLM prompt. If the ground truth is description-derived, the high BERTScore values reported in Table II largely reflect description-to-description lexical overlap, and the central claim about prerequisite reasoning is unsupported. This is a load-bearing validity issue, not a reproducibility gap.
[V-A, Table II] No baselines, variance estimates, or significance tests are reported. Table II lists point estimates of Sim_sem and BERTScore for each model, but the manuscript never compares against a trivial extractive baseline (e.g., selecting noun phrases from the skill description) or a random-list baseline. Given that Table I suggests the ground-truth lists are extractable from the description, such a baseline is essential to interpret F1_BERT = 0.8347. Moreover, without repeated runs or confidence intervals, the small differences among models used to rank them (e.g., 0.8347 vs. 0.8262) cannot be distinguished from noise.
[III-B, IV-C] The zero-shot interpretation is not protected from training-data contamination. ESCO is a public taxonomy and its text is a plausible component of web-scale pretraining corpora for all 13 evaluated models. The manuscript does not report any contamination check, such as time-controlled evaluations, membership tests, or paraphrase-based prompt variants. Without such a check, the high scores may reflect memorization of ESCO descriptions rather than independent reasoning, which is a premise of the paper's title and conclusion.

minor comments (5)

[III-B, IV-D] Section III-B states that evaluation uses exact string match and semantic similarity metrics, but Section IV-D defines only semantic similarity and BERTScore, and no exact-match results appear in Table II; the text should be aligned.
[III-B, Figure 3] Figure 3's prompt block labels the components as Context, Instruction, Input Data, and Formatting Guidelines, whereas Section III-B describes the four components as context, instruction, input text, and formatting indicator; make the naming consistent.
[I, IV-B] Model names are inconsistent: LLama, Llama, and LLaMA 4 are used interchangeably; standardize to a single convention, and similarly use Qwen2-72B consistently.
[IV-A] The GitHub repository link appears inline as 2https://github.com/... due to footnote formatting; fix the citation.
[V-A, Table II] Table II would be more useful with the exact API model version, temperature, and decoding parameters used for each model, since several model names include 'preview' or release dates.

Circularity Check

1 steps flagged · score 5.0 of 10

The benchmark's 'expert-defined prerequisite' ground truth is built by renaming ESCO's occupation-to-skill relation (the inverse of isEssentialSkillFor), and the displayed example's ground-truth labels are lexical chunks of the skill description that is then fed verbatim into the LLM prompt; the central alignment claim therefore partially reduces to description-to-description overlap.

renaming known result [Section IV-A (Benchmark Dataset: ESCO-PrereqSkill), Table I and Figure 3]
"Target skills were selected based on three criteria: ... (iii) the existence of linked prerequisite skills, identified through relational properties such as the inverse of isEssentialSkillFor. ... Table I ground truth: 'Correctional procedures, Legal regulations in corrections, Staff management, Safety protocols.'"

In ESCO, isEssentialSkillFor relates a skill to an occupation (or an occupation to a skill); its inverse is occupation-to-skill membership, not a skill-to-skill prerequisite ordering. The paper nevertheless labels these pairs 'expert-defined prerequisite links', so the evaluation target is defined by renaming a different relation. The published example makes the reduction concrete: the four ground-truth entries are near-verbatim noun phrases from the skill description ('correctional procedures', 'legal regulations', 'staff', 'safety'), and Figure 3's prompt supplies that same description verbatim to the LLM.

full rationale

The paper contains no self-citation chain, no imported uniqueness theorem, and no fitted parameter relabeled as a prediction; the model outputs are genuinely generated from prompts. The circularity is confined to benchmark construction: ground truth is asserted to be expert-defined prerequisite links, but it is operationalized as 'the inverse of isEssentialSkillFor' — a relation that at face value connects skills to occupations rather than skills to skills — and the only displayed example has ground-truth labels that are essentially chunks of the skill description that is later placed in the prompt. That makes the evaluation partially self-referential: agreement with 'expert prerequisites' can reflect lexical overlap with the input description rather than prerequisite reasoning. Public availability of ESCO in pretraining corpora is a contamination risk rather than a formal circular step, so it is not counted in the score; however, it compounds the difficulty of interpreting the absolute F1 values. No limitation section addresses either issue, so the central result should be treated as not yet independently validated.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim relies on the validity of the ESCO-derived ground truth, the absence of training contamination, and the appropriateness of the similarity metrics. No numerical free parameters are fit, but the dataset selection criteria and prompt design are hand-chosen and not varied.

assumptions (3)

domain assumption ESCO's relational properties encode true prerequisite relations between skills
Section IV-A states ground truth is 'identified through relational properties such as the inverse of isEssentialSkillFor', but no validation of this relation as a prerequisite is given.
domain assumption The public ESCO taxonomy does not appear in LLM pretraining corpora (or its presence does not affect outputs)
The zero-shot claim assumes predictions are novel inferences, but LLMs are trained on web-scale data that likely includes ESCO; no contamination analysis is provided.
domain assumption BERTScore and Sentence-BERT cosine similarity are valid measures of prerequisite-list agreement
Section IV-D uses these metrics without validating that higher scores correspond to pedagogically better predictions; concatenating a variable-length set into one string may obscure item-level errors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Well Do LLMs Predict Prerequisite Skills? Zero-Shot Comparison to Expert-Defined Concepts." pith.science (2026). https://pith.science/paper/UJBEETCF

@misc{pith2026250718479,
  author       = {Pith},
  title        = {Pith review of: How Well Do LLMs Predict Prerequisite Skills? Zero-Shot Comparison to Expert-Defined Concepts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJBEETCF}},
  note         = {Machine review of arXiv:2507.18479}
}

read the original abstract

Prerequisite skills - foundational competencies required before mastering more advanced concepts - are important for supporting effective learning, assessment, and skill-gap analysis. Traditionally curated by domain experts, these relationships are costly to maintain and difficult to scale. This paper investigates whether large language models (LLMs) can predict prerequisite skills in a zero-shot setting, using only natural language descriptions and without task-specific fine-tuning. We introduce ESCO-PrereqSkill, a benchmark dataset constructed from the ESCO taxonomy, comprising 3,196 skills and their expert-defined prerequisite links. Using a standardized prompting strategy, we evaluate 13 state-of-the-art LLMs, including GPT-4, Claude 3, Gemini, LLaMA 4, Qwen2, and DeepSeek, across semantic similarity, BERTScore, and inference latency. Our results show that models such as LLaMA4-Maverick, Claude-3-7-Sonnet, and Qwen2-72B generate predictions that closely align with expert ground truth, demonstrating strong semantic reasoning without supervision. These findings highlight the potential of LLMs to support scalable prerequisite skill modeling for applications in personalized learning, intelligent tutoring, and skill-based recommender systems.

Figures

Figures reproduced from arXiv: 2507.18479 by the authors.

**Figure 2.** Zero-shot prerequisite skill prediction workflow. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Zero-shot prompt template. The prompt was carefully designed to elicit concise, structured, and comparable outputs across models, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison of minimum, mean, and maximum inference latency (in seconds, log scale) for different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

[1]

Learning by dis- covering concept hierarchies,

B. Zupan, M. Bohanec, J. Dem ˇsar, and I. Bratko, “Learning by dis- covering concept hierarchies,” Artificial Intelligence, vol. 109, no. 1-2, pp. 211–242, 1999

work page 1999
[2]

Deep knowledge tracing,

C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein, “Deep knowledge tracing,” Advances in neural information processing systems , vol. 28, 2015

work page 2015
[3]

Prerequisite-driven deep knowledge tracing,

P. Chen, Y . Lu, V . W. Zheng, and Y . Pian, “Prerequisite-driven deep knowledge tracing,” in 2018 IEEE international conference on data mining (ICDM), pp. 39–48, IEEE, 2018

work page 2018
[4]

B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill, et al. , Taxon- omy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain . Longman New York, 1956

work page 1956
[5]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv:2303.08774, 2023

arXiv 2023
[6]

The claude 3 model family: Opus, sonnet, haiku

Anthropic, “The claude 3 model family: Opus, sonnet, haiku.” https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model Card Claude 3.pdf, Mar. 2024. Accessed April 15, 2025

work page 2024
[7]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, Rozi `ere, et al. , “Llama: Open and efficient foundation language models,” arXiv:2302.13971, 2023

arXiv 2023
[8]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. , “Gemini: a family of highly capable multimodal models,” arXiv:2312.11805, 2023

arXiv 2023

Show all 29 references

[9]

Emergent abilities of large language models,

J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler,et al., “Emergent abilities of large language models,” arXiv:2206.07682, 2022

2022 arXiv
[10]

J. T. F. on Computing Curricula, Computer Science Curricula 2013 . ACM/Association for Computing Machinery, 2013

2013
[11]

Personal learning environment toward lifelong learning: an ontology-driven conceptual model,

B. A. Nan Cenka, H. B. Santoso, and K. Junus, “Personal learning environment toward lifelong learning: an ontology-driven conceptual model,” Interactive Learning Environments, pp. 6445–6461, 2023

2023
[12]

Inferring concept prerequisite relations from online educational resources,

S. Roy, M. Madhyastha, S. Lawrence, and V . Rajan, “Inferring concept prerequisite relations from online educational resources,” in Proceedings of the AAAI conference on artificial intelligence , pp. 9589–9594, 2019

2019
[13]

Prerequisite relation learning for concepts in moocs,

L. Pan, C. Li, J. Li, and J. Tang, “Prerequisite relation learning for concepts in moocs,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1447–1456, 2017

2017
[14]

Learning concept prerequisite relation via global knowledge relation optimization,

M. Zhang, J. Wang, K. Xiao, S. Wang, Y . Zhang, H. Chen, and Z. Li, “Learning concept prerequisite relation via global knowledge relation optimization,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 1638–1646, 2025

2025
[15]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

2019
[16]

Retrieval-augmented recommender system: Enhancing recommender systems with large language models,

D. Di Palma, “Retrieval-augmented recommender system: Enhancing recommender systems with large language models,” in Proceedings of the 17th ACM Conference on Recommender Systems , 2023

2023
[17]

Empowering education with llms-the next-gen interface and content generation,

S. Moore, R. Tong, A. Singh, Z. Liu, X. Hu, Y . Lu, J. Liang, C. Cao, H. Khosravi, P. Denny, et al. , “Empowering education with llms-the next-gen interface and content generation,” in International Conference on Artificial Intelligence in Education , pp. 32–37, Springer, 2023

2023
[18]

Exploring large language models for knowledge graph completion,

L. Yao, J. Peng, C. Mao, and Y . Luo, “Exploring large language models for knowledge graph completion,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2025

2025
[19]

Zero-shot link prediction in knowledge graphs with large language models,

M. Li, C. Ling, R. Zhang, and L. Zhao, “Zero-shot link prediction in knowledge graphs with large language models,” in 2024 IEEE International Conference on Data Mining (ICDM) , pp. 753–760, 2024

2024
[20]

Skill-llm: Repurposing general-purpose llms for skill extraction,

A. Herandi, Y . Li, Z. Liu, X. Hu, and X. Cai, “Skill-llm: Repurposing general-purpose llms for skill extraction,” arXiv:2410.12052, 2024

2024 arXiv
[21]

Taxonomy- guided zero-shot recommendations with llms,

Y . Liang, L. Yang, C. Wang, X. Xu, P. S. Yu, and K. Shu, “Taxonomy- guided zero-shot recommendations with llms,” arXiv:2406.14043, 2024

2024 arXiv
[22]

Zero-shot learners for natural language understanding via a unified multiple-choice perspective,

J. Wang, P. Yang, R. Gan, Y . Zhang, J. Zhang, and T. Sakai, “Zero-shot learners for natural language understanding via a unified multiple-choice perspective,” IEEE Access, vol. 11, pp. 142829–142845, 2023

2023
[23]

Artificial intelligence in education: Aied for personalised learning pathways.,

O. Tapalova and N. Zhiyenbayeva, “Artificial intelligence in education: Aied for personalised learning pathways.,” Electronic Journal of e- Learning, vol. 20, no. 5, pp. 639–653, 2022

2022
[24]

Human resources for big data professions: A systematic classification of job roles and required skill sets,

A. De Mauro, M. Greco, M. Grimaldi, and P. Ritala, “Human resources for big data professions: A systematic classification of job roles and required skill sets,” Information Processing & Management , vol. 54, no. 5, pp. 807–817, 2018

2018
[25]

Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,

Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” IEEE transactions on pattern analysis and machine intelligence , 2018

2018
[26]

Training language models to follow instructions with human feedback,

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27730–27744, 2022

2022
[27]

Chain of preference optimization: Improving chain-of-thought reasoning in llms,

X. Zhang, C. Du, T. Pang, Q. Liu, W. Gao, and M. Lin, “Chain of preference optimization: Improving chain-of-thought reasoning in llms,” Advances in Neural Information Processing Systems, pp. 333–356, 2024

2024
[28]

Sentence-bert: Sentence embeddings using siamese bert-networks,

N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, 11 2019

2019
[29]

Bertscore: Evaluating text generation with bert,

T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv:1904.09675, 2019

1904 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Learning by dis- covering concept hierarchies,

B. Zupan, M. Bohanec, J. Dem ˇsar, and I. Bratko, “Learning by dis- covering concept hierarchies,” Artificial Intelligence, vol. 109, no. 1-2, pp. 211–242, 1999

work page 1999

[2] [2]

Deep knowledge tracing,

C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein, “Deep knowledge tracing,” Advances in neural information processing systems , vol. 28, 2015

work page 2015

[3] [3]

Prerequisite-driven deep knowledge tracing,

P. Chen, Y . Lu, V . W. Zheng, and Y . Pian, “Prerequisite-driven deep knowledge tracing,” in 2018 IEEE international conference on data mining (ICDM), pp. 39–48, IEEE, 2018

work page 2018

[4] [4]

B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill, et al. , Taxon- omy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain . Longman New York, 1956

work page 1956

[5] [5]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv:2303.08774, 2023

arXiv 2023

[6] [6]

The claude 3 model family: Opus, sonnet, haiku

Anthropic, “The claude 3 model family: Opus, sonnet, haiku.” https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model Card Claude 3.pdf, Mar. 2024. Accessed April 15, 2025

work page 2024

[7] [7]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, Rozi `ere, et al. , “Llama: Open and efficient foundation language models,” arXiv:2302.13971, 2023

arXiv 2023

[8] [8]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. , “Gemini: a family of highly capable multimodal models,” arXiv:2312.11805, 2023

arXiv 2023