Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4 can add structural markup to course questions, but not reliable concept links.

desk verdict A useful, honest negative result about LLM question generation, but the expert evaluation needs more methodological rigor before the numbers can be trusted. read the letter →

arxiv 2412.04185 v1 pith:MTZTUZTZ submitted 2024-12-05 cs.AI

Dominic Lohr , Marc Berges , Abhishek Chugh , Michael Kohlhase , Dennis Müller This is my paper · ORCID
classification cs.AI
keywords automatedquestiongenerationlargelanguagemodelsretrieval-augmentedsemanticannotationsTeXcomputerscienceeducationadaptivelearningBloom'staxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether a large language model can generate quiz questions for a specific university course that are both pedagogically sound and annotated with the semantic information an adaptive learning system needs. Using retrieval-augmented generation, the authors feed GPT-4 Turbo relevant sections of a symbolic AI lecture and ask it to produce sTeX-annotated multiple-choice or fill-in-the-blank questions targeting Bloom's 'understand' level. The central result is mixed: structural annotations (question environments, correct-answer markers, objectives) are produced correctly in almost all cases, but relational annotations linking text to ontology symbols rarely work, and 11 of 30 expert-rated questions contain content errors. The authors conclude that LLMs can contribute to a pool of course-specific learning materials, but current performance requires significant human filtering and validation.

What carries the argument

The load-bearing mechanism is the sTeX/OMDoc annotation framework paired with retrieval-augmented generation. sTeX marks up LaTeX documents with two kinds of annotations: structural annotations (the sproblem environment, mcb/scb choice blocks, fillinsol blanks, and objective declarations) that are schematic and independent of specific content, and relational annotations (symbol and module references that link text to concepts in an ontology). The pipeline injects the course's own sTeX source into the prompt, asks GPT-4 Turbo to produce annotated questions targeting the 'understand' level of Bloom's taxonomy, and then has domain experts rate the output on fit, solvability, clarity, relevance, feedback quality, and format alignment. The distinction between structural and relational annotations is what carries the central evaluation: the first works, the second does not.

What would settle it

Give the same 30 questions to a second panel of course experts, or to students who have completed the course, and compare solvability and content-error judgments; if agreement is low, the reported quality counts do not generalize.

Watch

Extended reading notes

Core claim

The paper's central claim is that a state-of-the-art LLM, GPT-4 Turbo, can generate course-specific quiz questions with correct structural sTeX semantic annotations, but fails at relational semantic annotations that connect question text to concepts in an ontology, and the generated questions frequently miss educational quality standards. In an expert evaluation of 30 questions across six topics from an 'Artificial Intelligence I' course, structural annotations (question environments, answer markers, objectives) were almost always correct, while relational annotations were rarely usable despite retrieval-augmented generation and function-calling attempts. Content errors appeared in 11 of 30 questions, often in answer options and feedback, including at least one question that actively reinforced a logical fallacy (denying the antecedent). The authors conclude that such LLM output can feed a pool of learning materials only under significant human review, and that the human-in-the-loop remains essential.

Load-bearing premise

The results stand on the assumption that the experts' ratings of the 30 questions are consistent and representative; the paper reports no inter-rater agreement, no student evaluation, and no control items, so the counts (28/30, 27/30, 11/30) may be specific to this course and this model.

Editorial extensions

If this is right

  • Course-specific question generation is feasible with retrieval-augmented generation: experts rated 28 of 30 questions as fitting the teaching material and 27 of 30 as solvable from it.
  • Because relational annotation fails, any practical adaptive-learning pipeline must treat concept linking as a separate step from question writing rather than expecting the LLM to do both.
  • The generated question pool still needs expert review: 11 of 30 questions contained content errors, some of which actively reinforce common misconceptions.
  • LLMs strongly prefer multiple-choice and single-choice formats and rarely produce fill-in-the-blank questions when targeting the understand dimension.
  • Feedback is a persistent weak point: it is often missing or merely rephrases the wrong answer, limiting the learning value of otherwise usable questions.
  • The model's observed difficulty with 'apply'-level questions suggests that asking for deeper cognitive dimensions will require more than prompt tuning.
  • A hybrid design that uses the LLM only for question text and a deterministic ontology lookup for relational annotations may sidestep the main failure mode the paper identifies.
  • The expert-survey evidence would be strengthened by inter-rater agreement and student testing, both of which the paper notes are absent, and both are needed before the quality counts can be treated as stable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether GPT-4 Turbo, combined with retrieval-augmented generation and the sTeX semantic markup framework, can generate course-specific, semantically annotated quiz questions for an adaptive learning assistant. The authors describe a prompt-based pipeline, generate 30 questions across six topics of an 'Artificial Intelligence I' course, and evaluate them through an expert survey. They report that structural semantic annotations are generated reliably, while relational annotations are not; that 28/30 questions fit the teaching material, 27/30 are solvable, and 11/30 contain content errors; and that question quality often falls below educational standards, requiring human intervention.

Significance. If the empirical findings are reliable, the paper makes a useful contribution to automated question generation in higher education by showing both a feasible direction (structural semantic annotation with RAG) and a clear limitation (relational concept linking and autonomous question quality). The authors are honest about several threats to validity, including the absence of student validation and control items, and they provide a detailed account of prompt iteration and the sTeX infrastructure. The main quantitative claims, however, rest on an expert evaluation whose rater design and reliability are not reported, so the strength of the evidence is currently indeterminate.

major comments (3)
  1. [§4.5.1 and §5.1] The paper's central quantitative results (28/30 fit, 27/30 solvable, 11/30 with content errors) rest entirely on the expert survey, but the survey design is under-specified: the manuscript does not state how many experts participated, whether each generated question was rated by more than one expert, or any inter-rater agreement statistic. Since the ratings are subjective Likert judgments plus a free-text error field, the reported counts cannot be interpreted without reliability information. The limitation statement in §5.4 explicitly notes the absence of control items and student validation, leaving no anchor for rating severity. Please report the number of raters and per-item agreement, or reframe the counts explicitly as exploratory single-rater observations that do not support generalization.
  2. [§5.3 and RQ2] The conclusion that relational annotations 'exhibited poor integration' is supported only by qualitative observation and a failed function-calling/RAG augmentation; no quantitative measure of relational annotation correctness is provided, such as the fraction of generated symbol references that resolve to existing sTeX symbols, the number of incorrect module imports, or precision/recall of concept links. Because RQ2 asks 'to what extent' LLMs can annotate questions semantically, the headline claim about relational annotations needs a metric to be assessed. Even a coarse count (e.g., correct vs. incorrect |symref or |sr usages) would make the finding reproducible.
  3. [§5.2] The 11/30 content-error count is not accompanied by a coding scheme or error taxonomy, nor by a per-topic breakdown (only arc consistency and propositional logic are named qualitatively). Without a predefined definition of what counts as a 'content error' and evidence of rater agreement on that definition, the count is not independently verifiable. This matters because the paper's educational-standards conclusion leans heavily on this number; please provide the coding criteria and, if feasible, the distribution of errors across the six topics.
minor comments (5)
  1. [§4.2] There is a typo: 'GTP-4-Turbo' should be 'GPT-4-Turbo'.
  2. [§4.5.2] The phrase 'our main objective in AGQ' uses the wrong acronym; it should be 'AQG' (automated question generation) as used elsewhere.
  3. [§5.2] The illustrative natural-deduction question is described as 'generated in an earlier experiment'; please clarify whether it is part of the 30-question evaluation set or an additional qualitative example, since it is used to support a quantitative claim.
  4. [§4.2 and §4.3] The full prompt and the 30 generated questions are not included in the manuscript or a supplement, and §4.2 says no public pipeline instance is available; making these available in supplementary material would substantially improve reproducibility.
  5. [Throughout] The manuscript contains many spacing artifacts (e.g., 'T o', 'AL EA', 'g e ne rat e') that appear to originate from text extraction; a careful copyedit is needed before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation whose conclusions rest on expert ratings and observed outputs, not on fitted parameters or self-citation chains.

full rationale

The paper contains no mathematical derivation chain and no fitted parameter that is later relabeled as a prediction. Its central claims—that structural semantic annotations are generated reliably while relational annotations are not, and that many generated questions fail educational quality standards—are summaries of expert ratings and qualitative observations of the generated artifacts. The authors' own sTeX, ALeA, and Y-Model infrastructure is the object under test, not a premise that forces the outcome; citations to that infrastructure document the tooling and annotation schema rather than supplying evidence for the empirical conclusion. The iterative prompt design is transparently reported and does not hide a fit to the evaluation criteria. The study's acknowledged limitations, including the absence of student evaluation, control items, and any reported inter-rater reliability, are threats to the validity of the measurements, which belongs under correctness or evidence quality, not circularity. No quote from the paper exhibits a specific reduction of a result to its own inputs, so no circular step can be identified. The honest finding is therefore a score of 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper contains no mathematical derivation, so there are no fitted model parameters in the usual sense. The experimental design does fix several hand-chosen values (understand dimension, medium difficulty, five questions per prompt, top-ten retrieval) that bound the claims. The central results also rest on domain assumptions: Bloom's taxonomy as a valid cognitive-level model, sTeX as representative of semantic annotation schemes, expert evaluation as a sufficient quality proxy, the sample of six topics from one course as representative, and GPT-4-turbo as a representative current LLM. These are stated or implied in Sections 3, 4.1, 4.2, 4.5, and 4.5.2.

free parameters (4)
  • cognitive_dimension = understand
    The experiment only targets Bloom's 'understand' level; the authors assume other levels are harder, so conclusions are specific to this dimension.
  • difficulty_level = medium
    Difficulty was fixed to 'medium' because earlier experiments produced too-easy questions; this choice affects the quality and discriminative power of generated items.
  • questions_per_prompt = 5
    The number of questions generated per prompt was fixed to five per topic; the paper notes a trade-off between diversity and the model's willingness to produce many questions.
  • retrieval_top_k = 10
    The function-calling search returns the top ten definitions for each query; the authors conjecture this long appended context degraded question quality, so it is load-bearing for the relational annotation failure.
assumptions (5)
  • domain assumption Bloom's revised taxonomy is a valid model for cognitive levels, and the 'understand' dimension is a meaningful target for university CS questions.
    Used in Sections 3 and 4.4 to define the generation target and in the expert survey; a different taxonomy or level could change results.
  • domain assumption The sTeX/OMDoc annotation schema is representative enough for conclusions to generalize to other semantic annotation systems.
    Section 4.1 argues sTeX subsumes most annotation systems; the paper's conclusions about relational annotation failure are meant to generalize.
  • domain assumption Expert ratings from the course environment are a sufficient proxy for question quality, without student testing or control items.
    Section 4.5 selects expert-based evaluation; Section 5.4 admits students and controls were absent. The central quality counts depend on this proxy.
  • domain assumption The six topics from one AI course are representative of university-level CS content for assessing LLM performance.
    Section 4.5.2 picks six topics from a single course; the paper generalizes to CS education broadly.
  • domain assumption GPT-4-turbo is representative of the current generation of capable LLMs; single-model results support general statements about LLMs.
    Section 4.2 justifies the choice by leaderboard comparisons; the conclusions about 'LLMs' are drawn from one proprietary model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects." pith.science (2026). https://pith.science/paper/MTZTUZTZ

@misc{pith2026241204185,
  author       = {Pith},
  title        = {Pith review of: Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTZTUZTZ}},
  note         = {Machine review of arXiv:2412.04185}
}
read the original abstract

Background: Over the past few decades, the process and methodology of automated question generation (AQG) have undergone significant transformations. Recent progress in generative natural language models has opened up new potential in the generation of educational content. Objectives: This paper explores the potential of large language models (LLMs) for generating computer science questions that are sufficiently annotated for automatic learner model updates, are fully situated in the context of a particular course, and address the cognitive dimension understand. Methods: Unlike previous attempts that might use basic methods like ChatGPT, our approach involves more targeted strategies such as retrieval-augmented generation (RAG) to produce contextually relevant and pedagogically meaningful learning objects. Results and Conclusions: Our results show that generating structural, semantic annotations works well. However, this success was not reflected in the case of relational annotations. The quality of the generated questions often did not meet educational standards, highlighting that although LLMs can contribute to the pool of learning materials, their current level of performance requires significant human intervention to refine and validate the generated content.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 40 canonical work pages

  1. [1]

    Visible learning: a synthesis of over 800 meta-analyses relating to achievement

    Hattie JAC. Visible learning: a synthesis of over 800 meta-analyses relating to achievement. New Y ork: Routledge; 2009

    work page 2009

  2. [2]

    Enhancing the Quality of Learning: Dispositions, Instruction, and Learning Processes

    Kirby JR, Lawson MJ, editors. Enhancing the Quality of Learning: Dispositions, Instruction, and Learning Processes. Cambridge: Cambridge University Press; 2012

    work page 2012

  3. [3]

    Cognitive Load During Problem Solving: Effects on Learning

    Sweller J. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 1988;12(2):257–285

    work page 1988

  4. [4]

    Cognitive Architecture and Instructional Design

    Sweller J, van Merrienboer JG, Paas FWC. Cognitive Architecture and Instructional Design. Educational Psychology Review 1998;10(3):251–296

    work page 1998

  5. [5]

    Learning of mastery

    Bloom BS. Learning of mastery. Evaluation Comment 1968;1(2):1–12

    work page 1968

  6. [6]

    Good-bye, teacher

    Keller FS. "Good-bye, teacher...". Journal of applied behavior analysis 1968;1(1):79–89

    work page 1968

  7. [7]

    Learning with ALeA: T ailored Experiences through Annotated Course Material

    Kruse T, Berges M, Betzendahl J, Kohlhase M, Lohr D, Müller D. Learning with ALeA: T ailored Experiences through Annotated Course Material. [object Object]; 2023

    work page 2023

  8. [8]

    Automatic Question Generation from T ext - an Aid to Independent Study

    Wolfe JH. Automatic Question Generation from T ext - an Aid to Independent Study. In: Proceedings of the ACM SIGCSE-SIGCUE T echnical Symposium on Computer Science and Education - Not Known: ACM Press; 1976. p. 104– 112

    work page 1976

Show all 42 references
  1. [9]

    Can We T rust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources

    Denny P , Khosravi H, Hellas A, Leinonen J, Sarsa S. Can We T rust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources. eprint 2023

    2023

  2. [10]

    Automatic Generation of Multiple-Choice Questions for CS0 and CS1 Curricula Using Large Language Models

    Song T, Tian Q, Xiao Y, Liu S. Automatic Generation of Multiple-Choice Questions for CS0 and CS1 Curricula Using Large Language Models. In: Hong W, Kanaparan G, editors. Computer Science and Education. Computer Science and T echnology, vol. 2023 Singapore: Springer Nature Sing...

    2023

  3. [11]

    Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models

    Sarsa S, Denny P , Hellas A, Leinonen J. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 Lugano and Virtual Event Switzerland: A...

    2022

  4. [12]

    A Systematic Review of Automatic Question Generation for Educational Purposes

    Kurdi G, Leo J, Parsia B, Sattler U, Al-Emari S. A Systematic Review of Automatic Question Generation for Educational Purposes. International Journal of Artificial Intelligence in Education 2020 Mar;30(1):121–204

    2020

  5. [13]

    The Robots Are Here: Navigating the Generative AI Revolution in Computing Education

    Prather J, Denny P , Leinonen J, Becker BA, Albluwi I, Craig M, et al. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In: Proceedings of the 2023 Working Group Reports on Innovation and T echnology in Computer Science Education T urku Finl...

    2023

  6. [14]

    T owards Automated Generation and Evaluation of Questions in Educational Domains

    Bhat S, Nguyen H, Moore S, Stamper J, Sakr M, Nyberg E. T owards Automated Generation and Evaluation of Questions in Educational Domains. In: Proceedings of the 15th International Conference on Educational Data Mining International Educational Data Mining Society; 2022. p. 701–704

    2022

  7. [15]

    T owards Human-Like Educational Question Generation with Large Language Models

    Wang Z, Valdez J, Basu Mallick D, Baraniuk RG. T owards Human-Like Educational Question Generation with Large Language Models. In: Rodrigo MM, Matsuda N, Cristea AI, Dimitrova V, editors. Artificial Intelligence in Education, vol. 13355 Cham: Springer International Publishing; ...

    2022

  8. [16]

    From Hype to Insight: Exploring ChatGPT ’s Early Footprint in Education via Altmetrics and Bibliometrics

    Wong LH, Park H, Looi CK. From Hype to Insight: Exploring ChatGPT ’s Early Footprint in Education via Altmetrics and Bibliometrics. Journal of Computer Assisted Learning 2024 Feb;p. jcal.12962

    2024

  9. [17]

    Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

    Y an L, Sha L, Zhao L, Li Y, Martinez-Maldonado R, Chen G, et al. Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review. British Journal of Educational T echnology 2024 Jan;55(1):90–112. Lohr et al. 19

    2024

  10. [18]

    Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

    McNichols H, Feng W, Lee J, Scarlatos A, Smith D, Woodhead S, et al. Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning. eprint 2023

    2023

  11. [19]

    Reading Comprehension Quiz Generation Using Generative Pre-trained T ransform- ers

    Dijkstra R, Genç Z, Kayal S, Kamps J. Reading Comprehension Quiz Generation Using Generative Pre-trained T ransform- ers. In: Proceedings of the Fourth International Workshop on Intelligent T extbooks 2022, vol. 4 Durham, UK: Aachen: CEUR-WS; 2022. p. 4–17

    2022

  12. [20]

    Generating Multiple Choice Questions for Computing Courses Using Large Language Models

    T ran A, Angelikas K, Rama E, Okechukwu C, Smith DH, MacNeil S. Generating Multiple Choice Questions for Computing Courses Using Large Language Models. In: 2023 IEEE Frontiers in Education Conference (FIE) College Station, TX, USA: IEEE; 2023. p. 1–8

    2023

  13. [21]

    A T axonomy for Learning, T eaching, and Assessing: A Revision of Bloom’s T axon- omy of Educational Objectives

    Anderson LW, Krathwohl DR, editors. A T axonomy for Learning, T eaching, and Assessing: A Revision of Bloom’s T axon- omy of Educational Objectives. Complete ed ed. New Y ork: Longman; 2001

    2001

  14. [22]

    The Y-Model - Formalization of Computer Science T asks in the Context of Adaptive Learning Systems

    Lohr D, Berges M, Kohlhase M, Müller D, Rapp M. The Y-Model - Formalization of Computer Science T asks in the Context of Adaptive Learning Systems. In: 2023 IEEE 2nd German Education Conference (GECon) Berlin, Germany: IEEE; 2023. p. 1–6

    2023

  15. [23]

    Automatic Multiple Choice Question Generation From T ext: A Survey

    Ch DR, Saha SK. Automatic Multiple Choice Question Generation From T ext: A Survey. IEEE T ransactions on Learning T echnologies 2020 Jan;13(1):14–25

    2020

  16. [24]

    Automatic Question Generation: A Review of Methodologies, Datasets, Evaluation Metrics, and Applications

    Mulla N, Gharpure P . Automatic Question Generation: A Review of Methodologies, Datasets, Evaluation Metrics, and Applications. Progress in Artificial Intelligence 2023 Mar;12(1):1–32

    2023

  17. [25]

    The Power of Feedback

    Hattie J, Timperley H. The Power of Feedback. Review of Educational Research 2007 Mar;77(1):81–112

    2007

  18. [26]

    Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

    Kiesler N, Lohr D, Keuning H. Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. In: 2023 IEEE Frontiers in Education Conference (FIE) College Station, TX, USA: IEEE; 2023. p. 1–5

    2023

  19. [27]

    Let Them T ry to Figure It Out First

    Lohr D, Kiesler N, Keuning H, Jeuring J. “Let Them T ry to Figure It Out First” - Reasons Why Experts (Do Not) Pro- vide Feedback to Novice Programmers. In: Proceedings of the 2024 Innovation and T echnology in Computer Science Education (ITiCSE 2024), vol. 1 Milan, Italy: ACM...

    2024

  20. [28]

    Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests

    Hellas A, Leinonen J, Sarsa S, Koutcheme C, Kujanpää L, Sorva J. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In: Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 Chicago IL USA: ACM; 2023. p. 93–105

    2023

  21. [29]

    Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments

    Balse R, Valaboju B, Singhal S, Warriem JM, Prasad P . Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In: Proceedings of the 2023 Conference on Innovation and T echnology in Computer Science Education V. 1 T urku Finland: ACM; 2023. p. 292–298

    2023

  22. [30]

    A Large Language Model-Assisted Education T ool to Provide Feed- back on Open-Ended Responses

    Matelsky JK, Parodi F, Liu T, Lange RD, Kording KP . A Large Language Model-Assisted Education T ool to Provide Feed- back on Open-Ended Responses. eprint 2023

    2023

  23. [31]

    System Description: s T eX3 – A LATEX-based Ecosystem for Semantic/Active Mathematical Docu- ments

    Kohlhase M, Müller D. System Description: s T eX3 – A LATEX-based Ecosystem for Semantic/Active Mathematical Docu- ments. In: Buzzard K, Kutsia T, editors. Intelligent Computer Mathematics (CICM) 2022, vol. 13467 of LNAI Springer

    2022

  24. [32]

    OMDoc – An open markup format for mathematical documents [Version 1.2]

    Kohlhase M. OMDoc – An open markup format for mathematical documents [Version 1.2]. No. 4180 in LNAI, Springer Verlag; 2006. http://omdoc.org/pubs/omdoc1.2.pdf

    2006

  25. [33]

    https://github.com/slatex/sTeX/blob/main/doc/stex-manual.pdf

    Kohlhase M, Müller D, The s T eX3 Manual;. https://github.com/slatex/sTeX/blob/main/doc/stex-manual.pdf

  26. [34]

    An HTML/CSS schema for T EX primitives – generating high-quality responsive HTML from generic T EX

    Müller D. An HTML/CSS schema for T EX primitives – generating high-quality responsive HTML from generic T EX. In: TUGboat; TUG 2023 Conference Proceedings, vol. 44; 2023. p. 275–286.https://kwarc.info/people/dmueller/pubs/ tug23.pdf. 20 Lohr et al

    2023

  27. [35]

    arXiv; 2023

    OpenAI, GPT-4 T echnical Report. arXiv; 2023. https://arxiv.org/abs/2303.08774

    2023 arXiv

  28. [36]

    T ouvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al., LLaMA: Open and Efficient Foundation Language Models; 2023

    2023

  29. [37]

    Why Johnny Can’t Prompt: How Non-AI Experts T ry (and Fail) to Design LLM Prompts

    Zamfirescu-Pereira JD, Wong RY, Hartmann B, Y ang Q. Why Johnny Can’t Prompt: How Non-AI Experts T ry (and Fail) to Design LLM Prompts. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems CHI ’23, New Y ork, NY, USA: Association for Computing Machin...

    2023

  30. [38]

    https://platform.openai.com/docs/guides/prompt- engineering

    OpenAI, OpenAI API Documentation: Prompt Engineering; 2024. https://platform.openai.com/docs/guides/prompt- engineering

    2024

  31. [39]

    A Novel Framework for the Generation of Multiple Choice Question Stems Using Semantic and Machine-Learning T echniques

    Kumar AP , Nayak A, K MS, Chaitanya, Ghosh K. A Novel Framework for the Generation of Multiple Choice Question Stems Using Semantic and Machine-Learning T echniques. International Journal of Artificial Intelligence in Education 2023 Mar

    2023

  32. [40]

    Y ou’re (Not) My Type – Can LLMs Generate Feedback of Specific Types for Introductory Programming T asks? Journal of Computer Assisted Learning (JCAL) 2024;Accepted

    Lohr D, Keuning H, Kiesler N. Y ou’re (Not) My Type – Can LLMs Generate Feedback of Specific Types for Introductory Programming T asks? Journal of Computer Assisted Learning (JCAL) 2024;Accepted

    2024

  33. [41]

    Playing Games with Ais: The Limits of GPT-3 and Similar Large Language Models

    Sobieszek A, Price T. Playing Games with Ais: The Limits of GPT-3 and Similar Large Language Models. Minds and Machines 2022 Jun;32(2):341–364

    2022

  34. [2022]

    p. 184–188. https://kwarc.info/people/dmueller/pubs/cicm22stexsd.pdf

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.