Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A carefully engineered prompt can make GPT-4 produce a requested feedback type for novice programming submissions in 63 of 66 cases.

desk verdict A useful, mostly honest study of prompting GPT-4 for taxonomy-defined feedback, but the headline 63/66 match rate is inflated by including the prompt-development submissions in the evaluation set. read the letter →

arxiv 2412.03516 v1 pith:3YOS6EYC submitted 2024-12-04 cs.AI

Dominic Lohr , Hieke Keuning , Natalie Kiesler This is my paper · ORCID
classification cs.AI
keywords largelanguagemodelsGPT-4feedbacktypesintroductoryprogrammingpromptengineeringformativeclassificationcomputingeducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can be made to produce one specific kind of feedback for introductory programming exercises, instead of an uncontrolled mixture of corrections, hints, and explanations. Using six feedback categories from an established taxonomy and a prompt refined over five iterations, the authors report that GPT-4 delivers the requested type in 63 of 66 generated responses. The finding matters because programming learning tools have mostly offered simple correct/incorrect or test-based feedback, while richer formative feedback, such as explanations of concepts, descriptions of mistakes, and hints for next steps, has been hard to automate. The paper also shows that such controlled output is not error-free: misleading statements still appear, especially when the submitted program is already correct.

What carries the argument

The load-bearing mechanism is the final prompt (Listing 1), built through five iterations. It bundles the task description, class skeleton, student solution, and a model solution, then tells the model which of six feedback types to emit, with per-type instructions (KR should contain only correct/incorrect; KM should describe errors without fixes; KH should give concrete next-step hints; KP should divide the task into subgoals and give a percentage; KTC and KC should raise constraints or concepts only where the student's code actually has an issue). Global rules ask for brief wording, no mention of the model solution, and no additional information beyond the requested type. The taxonomy supplies the shared labels used both for generation and for the subsequent coding of the output.

What would settle it

Have a new rater team, blind to the requested feedback type, classify the same 66 raw messages; if the requested type is coded as present in fewer than about 50 of the 66, the paper's claim of controllable feedback types is not robust.

Watch

Extended reading notes

Core claim

The central claim is that feedback content can be steered by prompt design: given the task description, the starter code, the student's submission, a model solution, and a label naming one of six feedback types, GPT-4 produces feedback whose actual type matches the requested type in 63 of 66 cases. The six types are knowledge of result (correct/incorrect), knowledge about task constraints, knowledge about concepts, knowledge about mistakes, knowledge on how to proceed, and knowledge of performance. In 19 cases the output contained an additional type as well, most often knowledge about mistakes appearing alongside how-to-proceed or task-constraint feedback. The authors read this as evidence that LLMs can be controlled well enough to support richer automated feedback, while acknowledging that the match is not exact and that some statements are misleading.

Load-bearing premise

The load-bearing premise is that the two authors' consensus coding of the generated feedback types is a valid, unbiased measure of what GPT-4 actually produced; if other coders would not place the messages in the same categories, the 63-out-of-66 match rate could be a measurement artifact.

Editorial extensions

If this is right

  • Learners could be given a menu of feedback types, such as 'what is wrong' versus 'what do I do next', that sends a preset prompt so they do not need to phrase their own request to the model.
  • A hybrid tool that first runs test cases and inserts their results into the prompt should make simple feedback types like KR and KP more reliable, because the model would not have to judge correctness on its own.
  • Educators should treat generated feedback as needing a guardrail: the model still claims problems that are not present in the student's code, so learners need guidance in evaluating such feedback critically.
  • The prompt and the six-type classification give researchers a way to isolate feedback type as a variable in studies of feedback effects and learners' informational needs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 63-of-66 figure is evidence about GPT-4 at the time of the study, not about LLMs in general; re-running the same prompt on newer models would be needed to see whether the control improves or degrades over time.
  • The frequent leakage of KM into KH and KTC feedback may reflect a genuine conceptual boundary: a useful next-step hint usually has to name the mistake it addresses, so strict single-type generation may be fighting the nature of helpful feedback.
  • The model's habit of labelling style advice as 'mistakes' suggests it follows the everyday English label more than the taxonomy's intended definitions, so renaming the categories inside the prompt could reduce a source of misleading feedback.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper investigates whether GPT-4 can be prompted to generate feedback for introductory programming submissions in six predefined types from Keuning et al.'s feedback taxonomy. The authors iteratively refined a single prompt over five iterations using three student submissions, then applied the final prompt to 11 submissions (including the same three) across six exercises and four programming languages, producing 66 feedback messages. Two expert coders coded the feedback types and 13 characteristics, resolving disagreements by consensus. The paper reports that in 63 of 66 cases the desired feedback type matched the actual feedback type, notes that 19 messages contained additional types, and presents a qualitative analysis of characteristics such as misleading feedback, performance percentages, and concept explanations. The authors conclude that controllable generation of specific feedback types is promising, while acknowledging single-model and single-domain limitations.

Significance. If the core result were robust, this paper would be a useful contribution: it would show that a single prompt template can steer an LLM toward qualitatively distinct feedback types, and it provides a concrete prompt (Listing 1), a transparent iterative design process, and a taxonomy-based analysis that could inform follow-up work on feedback effects and educational tool design. The study covers authentic student submissions in multiple languages, and the authors are commendably transparent about misleading outputs and about cases where additional feedback types appear. However, the central quantitative claim is currently not established at the level the title and abstract suggest: the evaluation includes the prompt-development submissions, the coding reliability is not quantified, and the reported match rate is inclusive rather than strict. These issues need to be resolved before the claim of type-controllability can be accepted as a generalizable result.

major comments (4)
  1. [Sections 3.3.1, 3.4, and Tables 4-5] The three submissions labeled as the test set in Table 4 (10_TEOS, 13_NEGF, 01_TTBSA) are exactly the submissions used for prompt development in Section 3.3.1, and Section 3.4 states that the prompt was refined by generating all six feedback types for those three submissions across five iterations. Therefore, 18 of the 66 messages in Table 5 are not independent evaluations of the final prompt; they reflect prompt fitting rather than generalization. Please report the DFT-AFT match rate for the eight held-out submissions separately from the three development submissions, and state the match rate excluding those 18 development messages. The headline '63 out of 66' currently conflates prompt development with evaluation.
  2. [Section 3.5 and Table 5] The feedback analysis relies on two expert coders coding independently and resolving disagreements by consensus, but no measure of inter-rater reliability is reported for the feedback-type coding or for the 13 characteristic categories. The central 63/66 match count is produced entirely by this coding, so coding reliability is load-bearing for the paper's main claim. Please report initial agreement, both raw and chance-corrected (e.g., Cohen's kappa), for the AFT codes, and clarify whether the reported AFT values reflect the initial independent codes or the consensus outcome. Without this information, the main result cannot be distinguished from coder interpretation.
  3. [Section 4.1 and Table 5] The paper reports 63/66 matches and, in the same paragraph, notes that 19 messages contained additional feedback types beyond the requested one. The reported 'match' rate therefore appears to mean 'the desired type is among the actual types,' not 'the generated feedback consists only of the desired type.' The final prompt (Listing 1, line 23) explicitly instructs the model not to provide additional information, so the strict-match rate (DFT equals AFT, with no other types) is the more relevant measure of type control. Please report the strict-match rate separately and discuss the 19 mixed-type messages, such as KM appearing with KH and KTC, as partial deviations from the prompt instruction.
  4. [Sections 3.4 and 3.5] The study reports a single generation per task-type pair (66 total) using the ChatGPT interface, but it does not state the exact GPT-4 model version, sampling parameters, or number of runs. Because GPT-4 is stochastic and the paper's abstract itself emphasizes randomness as a relevant property of generated feedback, the 63/66 rate is a point estimate from one set of outputs. Please specify the model and interface settings, including temperature if available, and either report the stability of the match rate across repeated generations or state clearly that the result comes from a single run and is subject to sampling variation.
minor comments (4)
  1. [Listing 1, line 19] The word 'usefull' in the prompt text should be corrected to 'useful.'
  2. [Section 4.2.4] The sentence stating that the 'COMP' category was obsolete because no complete code examples were generated appears to refer to the wrong category: Table 2 defines COMPL as 'it matches the task description,' while examples are coded under EXA and code under CODE. Please correct the category label or the explanation.
  3. [Table 5] The legend uses nonstandard symbols (CIRCLE, LEFTCIRCLE) that are difficult to read in the PDF; using conventional check/cross marks or explicit Y/N values would improve readability.
  4. [Section 3.3.1 and Table 3] The exact task description used for the modified Triangles exercise is not provided in the paper. Since the prompt includes the task text, please include the full prompt inputs (task descriptions, class skeletons, and model solutions) in an appendix or repository to support replication.

Circularity Check

1 steps flagged · score 5.0 of 10

The 63/66 DFT-AFT match rate is partly a report of prompt-fitting: the three submissions used for iterative prompt development are included in the 66-message evaluation.

  1. fitted input called prediction [Section 3.4 (Prompt Design Process), Table 4, Section 4.1 (Table 5)]
    "Using the existing version of the prompt, we generated feedback for the three student submissions within the test set across all six types of feedback, resulting in a total of 18 feedback messages for each iteration. ... To determine whether the prompt should be further refined, we checked whether the expected feedback type was present ... . ... In almost all cases (63 out of 66), the desired feedback type (DFT) matches the actual feedback type (AFT) generated by the LLM."

    The three submissions used for prompt development (10_TEOS, 13_NEGF, 01_TTBSA) are the same three labeled 'Test set' in Table 4 and are included in the Table 5 evaluation of all 66 messages. The prompt was iteratively revised until the expected feedback type was present on exactly those 18 messages, so their DFT-AFT matches are a record of prompt fitting, not independent evaluation. The headline '63 out of 66' therefore mixes development-set performance with held-out performance; the match rate on the 48 messages from the other eight submissions is never reported. The central RQ1 answer is thus partially forced by the fit rather than by generalization to unseen submissions.

full rationale

The paper's main empirical claim is about prompt controllability, and the authors are transparent about their iterative prompt design. The central circularity is evaluative: the same three student submissions used to refine the prompt across five iterations are included in the 66-message test set, and the refinement criterion ('checked whether the expected feedback type was present') is exactly the evaluation criterion. This means 18 of the 66 reported cases are not an independent test of the prompt. The paper does not report the match rate on the 48 held-out messages alone, so the '63 out of 66' headline overstates the generality of the result. This is not a definitional or self-citation circularity: the feedback taxonomy and coding categories are reused transparently, and the prompt's explicit instructions to match a feedback type do not guarantee that the LLM will comply. However, the development/evaluation overlap is a genuine fitted-input-called-prediction issue, warranting a moderate circularity score rather than a high one.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No numerical model parameters are fitted; the central claim depends on the hand-tuned prompt and on the validity of the expert coding procedure. The inclusion of development-set submissions in the evaluation creates a moderate circularity burden.

free parameters (1)
  • Final prompt wording and six feedback-type instructions = Listing 1
    The prompt was modified across five iterations based on observed output quality; the final version is a hand-tuned artifact central to the reported 63/66 match rate.
assumptions (3)
  • domain assumption The Keuning et al. feedback taxonomy can be applied to free-form LLM-generated feedback text.
    The coding procedure treats each feedback message as one or more of the six types; if the taxonomy is not suited to generative model output, the match rates are not meaningful. Used throughout Section 3.5 and Section 4.1.
  • domain assumption Expert consensus coding provides a valid ground truth for identifying the actual feedback type.
    Two authors independently coded and then resolved conflicts via consensus, but no inter-rater reliability statistic is reported. The validity of the 63/66 count depends on this assumption. See Section 3.5.
  • domain assumption GPT-4 outputs with the final prompt are representative of the model's behavior for introductory programming tasks.
    The study uses a single model, a single prompt version, and no temperature or seed control; stochastic variation across runs is not captured. This limits generalization. See Sections 3.4 and 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?." pith.science (2026). https://pith.science/paper/3YOS6EYC

@misc{pith2026241203516,
  author       = {Pith},
  title        = {Pith review of: You're (Not) My Type -- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3YOS6EYC}},
  note         = {Machine review of arXiv:2412.03516}
}
read the original abstract

Background: Feedback as one of the most influential factors for learning has been subject to a great body of research. It plays a key role in the development of educational technology systems and is traditionally rooted in deterministic feedback defined by experts and their experience. However, with the rise of generative AI and especially Large Language Models (LLMs), we expect feedback as part of learning systems to transform, especially for the context of programming. In the past, it was challenging to automate feedback for learners of programming. LLMs may create new possibilities to provide richer, and more individual feedback than ever before. Objectives: This paper aims to generate specific types of feedback for introductory programming tasks using LLMs. We revisit existing feedback taxonomies to capture the specifics of the generated feedback, such as randomness, uncertainty, and degrees of variation. Methods: We iteratively designed prompts for the generation of specific feedback types (as part of existing feedback taxonomies) in response to authentic student programs. We then evaluated the generated output and determined to what extent it reflected certain feedback types. Results and Conclusion: The present work provides a better understanding of different feedback dimensions and characteristics. The results have implications for future feedback research with regard to, for example, feedback effects and learners' informational needs. It further provides a basis for the development of new tools and learning systems for novice programmers including feedback generated by AI.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools

    cs.CY 2024-12 conditional novelty 5.0 of 10

    Computing educators are adopting GenAI faster than they are formalizing policies, and both educators and developers see code reading, evaluation, and problem decomposition as rising in importance over syntax recall.

Reference graph

Works this paper leans on

66 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Students’ Misconceptions and Other Diffi culties in Introductory Programming: A Literature Review

    Qian Y, Lehman J. Students’ Misconceptions and Other Diffi culties in Introductory Programming: A Literature Review. ACM T rans Comput Educ 2017 oct;18(1). https://doi.org/10.1145/3077618

    doi:10.1145/3077618 2017

  2. [2]

    A systematic literature review on teaching and learning introductory programming in higher education

    Medeiros RP , Ramalho GL, Falcão TP . A systematic literature review on teaching and learning introductory programming in higher education. IEEE T ransactions on Education 2018;6 2(2):77–90. https://doi.org/10.1109/TE.2018.2864133

    arXiv 2018

  3. [3]

    Introductory programming: a systematic literature review

    Luxton-Reilly A, Simon, Albluwi I, Becker BA, Giannakos M, Kumar AN, et al. Introductory programming: a systematic literature review. In: Proceedings Companion of the 23rd An nual ACM Conference on Innovation and T echnology in Computer Science Education ITiCSE 2018 Companion, New Y ork, NY, USA: Association for Computing Machinery; 2018. p. 55–106. https...

    arXiv 2018

  4. [4]

    Modeling Programming Competency: A Qualitat ive Analysis

    Kiesler N. Modeling Programming Competency: A Qualitat ive Analysis. Cham: Springer; 2024

    work page 2024

  5. [5]

    Visible Learning: A Synthesis of over 800 Meta- Analyses Relating to Achievement

    Hattie J. Visible Learning: A Synthesis of over 800 Meta- Analyses Relating to Achievement. London ; New Y ork: Routledge; 2009. https://doi.org/10.4324/9780203887332

    doi:10.4324/9780203887332 2009

  6. [6]

    T owards Giving Timely Formative Feedback and Hints to Novice Programmers

    Jeuring J, Keuning H, Marwan S, Bouvier D, Izu C, Kiesler N , et al. T owards Giving Timely Formative Feedback and Hints to Novice Programmers. In: Proceedings of the 2022 Wor king Group Reports on Innovation and T echnology in Computer Science Education ITiCSE-WGR ’22, New Y ork, NY, US A: Association for Computing Machinery; 2022. p. 95–115. https://doi...

    arXiv 2022

  7. [7]

    T owards Criteria for Valuable Automati c Feedback in Large Programming Classes

    Lohr D, Berges M. T owards Criteria for Valuable Automati c Feedback in Large Programming Classes. In: Hochschul- didaktik Informatik HDI 2021 Dortmund: Jörg Desel, Simone O pel; 2021. p. 181–186

    work page 2021

  8. [8]

    Feedback-Generation f or Programming Exercises With GPT-4

    Azaiz I, Kiesler N, Strickroth S. Feedback-Generation f or Programming Exercises With GPT-4. In: Proceedings of the 2024 on Innovation and T echnology in Computer Science Education V. 1 ITiCSE 2024, New Y ork, NY, USA: Association for Computing Machinery; 2024. p. 31–37. https://doi.org/10.1145/3649217.3653594

    arXiv 2024

Show all 66 references
  1. [10]

    Automatic Genera tion of Programming Exercises and Code Explanations Using Large Language Models

    Sarsa S, Denny P , Hellas A, Leinonen J. Automatic Genera tion of Programming Exercises and Code Explanations Using Large Language Models. In: Proc. of ICER ACM; 2022. https://doi.org/10.1145/3501385.3543957

    2022

  2. [11]

    Experiences from Using Code Explanations Gener- ated by Large Language Models in a Web Software Development E -Book

    MacNeil S, T ran A, Hellas A, Kim J, Sarsa S, Denny P , et al. Experiences from Using Code Explanations Gener- ated by Large Language Models in a Web Software Development E -Book. In: Proc. SIGCSE TS; 2023. p. 931–937. https://doi.org/10.1145/3545945.3569785

    2023

  3. [12]

    AI-enhanced Auto-Cor rection of Programming Exercises: How Effective is GPT-3.5? International Journal of Engineering Pedagogy (iJEP) 2023 Dec;13(8):67–83

    Azaiz I, Deckarm O, Strickroth S. AI-enhanced Auto-Cor rection of Programming Exercises: How Effective is GPT-3.5? International Journal of Engineering Pedagogy (iJEP) 2023 Dec;13(8):67–83

    2023

  4. [13]

    https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva- 331000

    Bengtsson D, Kaliff A, Assessment Accuracy of a Large Lan guage Model on Programming Assignments; 2023. https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva- 331000

    2023

  5. [14]

    Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

    Kiesler N, Lohr D, Keuning H. Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. In: 2023 IEEE Frontiers in Education Conference ( FIE); 2023. p. 1–5

    2023

  6. [15]

    A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

    Keuning H, Jeuring J, Heeren B. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. ACM T rans Comput Educ 2018 sep;19(1). https://doi.org/10.1145/3231711

    2018 doi

  7. [16]

    Feedback strategies for interactive learni ng tasks

    Narciss S. Feedback strategies for interactive learni ng tasks. Handbook of research on educational communications and technology 2008;3:125–144

    2008

  8. [17]

    The Power of Feedback

    Hattie J, Timperley H. The Power of Feedback. Review of E ducational Research 2007;77(1):81–112. https://doi.org/10.3102/003465430298487. Lohr et al. 25

    2007 doi

  9. [18]

    Focus on formative feedback

    Shute VJ. Focus on formative feedback. Review of Educat ional Research 2008;78(1)

    2008

  10. [19]

    Informatives T utorielles Feedback: Entwic klungs- und Evaluationsprinzipien auf der Basis instrukti onspsy- chologischer Erkenntnisse

    Narciss S. Informatives T utorielles Feedback: Entwic klungs- und Evaluationsprinzipien auf der Basis instrukti onspsy- chologischer Erkenntnisse. Münster: Waxmann Verlag; 2006

    2006

  11. [20]

    Do students need detailed feedbac k on programming exercises and can automated assessment systems provide it? Journal of Computing Sciences in Colleg es 2016;31(4):115–121

    Kyrilov A, Noelle DC. Do students need detailed feedbac k on programming exercises and can automated assessment systems provide it? Journal of Computing Sciences in Colleg es 2016;31(4):115–121

    2016

  12. [21]

    Investigating Feedback Types i n JupyterLab for Programming Novices 2023

    Brocker A, Schroeder U. Investigating Feedback Types i n JupyterLab for Programming Novices 2023

    2023

  13. [22]

    Let Them T ry to F igure It Out First

    Lohr D, Kiesler N, Keuning H, Jeuring J. “Let Them T ry to F igure It Out First” - Reasons Why Experts (Do Not) Pro- vide Feedback to Novice Programmers. In: Proceedings of the 2024 Innovation and T echnology in Computer Science Education (ITiCSE 2024), vol. 1 Milan, Italy: AC...

    2024

  14. [23]

    Investigating the Use and Effects of Feedback in CodingBat Exercises: An Exploratory Thinking Aloud Stud y

    Kiesler N. Investigating the Use and Effects of Feedback in CodingBat Exercises: An Exploratory Thinking Aloud Stud y. In: 2023 Future of Educational Innovation-Workshop Series Data in Action; 2023. p. 1–12

    2023

  15. [24]

    Using conceptual models to teach BAS IC computer programming

    Bayman P , Mayer RE. Using conceptual models to teach BAS IC computer programming. Journal of Educational Psychol- ogy 1988;80(3):291

    1988

  16. [25]

    Compiler Error Messages Considered Unhelp- ful: The Landscape of T ext-Based Programming Error MessageResearch

    Becker BA, Denny P , Pettit R, Bouchard D, Bouvier DJ, Harrington B, et al. Compiler Error Messages Considered Unhelp- ful: The Landscape of T ext-Based Programming Error MessageResearch. In: Proceedings of the Working Group Reports on Innovation and T echnology in Computer Sc...

    2019

  17. [26]

    Mental Models of Recursion: A Secondary Anal ysis of Novice Learners’ Steps and Errors in Java Exercises

    Kiesler N. Mental Models of Recursion: A Secondary Anal ysis of Novice Learners’ Steps and Errors in Java Exercises. In: PPIG; 2022. p. 226–240

    2022

  18. [27]

    Automated Grading and Feedback T ools for Programming Education: A Systematic Review

    Messer M, Brown NCC, Kölling M, Shi M. Automated Grading and Feedback T ools for Programming Education: A Systematic Review. ACM T rans Comput Educ 2024 feb;24(1). https://doi.org/10.1145/3636515

    2024 doi

  19. [28]

    Automated Assessment in C omputer Science Education: A State-of-the-Art Review

    Paiva JC, Leal JP , Figueira A. Automated Assessment in C omputer Science Education: A State-of-the-Art Review. ACM T rans Comput Educ 2022 jan;Just Accepted

    2022

  20. [29]

    The Robots Are Here: Navigating the Generative AI Revolution in Computing Education

    Prather J, Denny P , Leinonen J, Becker BA, Albluwi I, Cra ig M, et al. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In: Proce edings of the 2023 Working Group Reports on In- novation and T echnology in Computer Science Education ITiC SE...

    2023

  21. [30]

    Beyond the Hype: A Comprehensive Review of Current T rends in Generative AI Research, T eaching Practices, and T ools

    Prather J, Leinonen J, Kiesler N, Benario JG, Lau S, MacN eil S, et al. Beyond the Hype: A Comprehensive Review of Current T rends in Generative AI Research, T eaching Practices, and T ools. In: Proceedings of the 2024 Conference on Innovation and T echnology in Computer Scien...

    2024

  22. [31]

    Generative AI in Introductory Programming

    Becker BA, Craig M, Denny P , Keuning H, Kiesler N, Leinon en J, et al. Generative AI in Introductory Programming. In: Kumar AN, Raj RK, Aly SG, Anderson MD, Becker BA, Blumenth al RL, et al., editors. Computer Science Curric- ula 2023 New Y ork, NY, USA: Association for Compu...

    2023

  23. [32]

    The Robots Are Coming: Explor- ing the Implications of OpenAI Codex on Introductory Progra mming

    Finnie-Ansley J, Denny P , Becker BA, Luxton-Reilly A, P rather J. The Robots Are Coming: Explor- ing the Implications of OpenAI Codex on Introductory Progra mming. In: Proc. ACE; 2022. p. 10–19. https://doi.org/10.1145/3511861.3511863

    2022

  24. [34]

    Using GitHub Copilot to Solve Simple Pro gramming Problems

    Wermelinger M. Using GitHub Copilot to Solve Simple Pro gramming Problems. In: Proceedings of the 54th ACM T echnical Symposium on Computer Science Education V. 1 SIGCSE 2023, New Y ork, NY, USA: Association for Computing Machinery; 2023. p. 172–178. https://doi.org/10.1145/35...

    2023

  25. [36]

    Large Language Models in Introdu ctory Programming Education: ChatGPT’s Per- formance and Implications for Assessments

    Kiesler N, Schiffner D. Large Language Models in Introdu ctory Programming Education: ChatGPT’s Per- formance and Implications for Assessments. In: CoRR abs/23 08.08572. arXiv: 2308.08572.; 2023. https://doi.org/10.48550/arXiv.2308.08572

    2023 doi arXiv

  26. [37]

    Analyzing Chat Protoco ls of Novice Programmers Solving Introductory Programming T asks with ChatGPT

    Scholl A, Schiffner D, Kiesler N. Analyzing Chat Protoco ls of Novice Programmers Solving Introductory Programming T asks with ChatGPT. In: Schulz S, Kiesler N, editors. Procee dings of DELFI 2024; 2024. p. 63–79

    2024

  27. [38]

    https://arxiv.org/abs/2407.20792, accepted at 2024 IEEE ASEE Frontiers in Education Conference

    Scholl A, Kiesler N, How Novice Programmers Use and Expe rience ChatGPT when Solving Programming Exercises in an Introductory Course; 2024. https://arxiv.org/abs/2407.20792, accepted at 2024 IEEE ASEE Frontiers in Education Conference

    2024 arXiv

  28. [39]

    How Instructors Incorporate Generative AI into T eaching Computing

    Prather J, Leinonen J, Kiesler N, Benario JG, Lau S, MacN eil S, et al. How Instructors Incorporate Generative AI into T eaching Computing. In: Proceedings of the 2024 Conference on Innovation and T echnology in Computer Science Education V. 2; 2024. p. 771–772. https://doi.or...

    2024

  29. [40]

    https://doi.org/10.48550/arXiv.2308.15276

    Wu Y, Li Z, Zhang JM, Papadakis M, Harman M, Liu Y, Large La nguage Models in Fault Localisation; 2023. https://doi.org/10.48550/arXiv.2308.15276

    2023 doi arXiv

  30. [41]

    Large Language Mo dels (GPT) Struggle to Answer Multiple-Choice Questions About Code:

    Savelka J, Agarwal A, Bogart C, Sakr M. Large Language Mo dels (GPT) Struggle to Answer Multiple-Choice Questions About Code:. In: Proceedings of the 15th International Conf erence on Computer Supported Education Prague, Czech Republic: SCITEPRESS - Science and T echnology Pub...

    2023

  31. [44]

    Comparing Code Explanations Created by Students and Large Language Models

    Leinonen J, Denny P , MacNeil S, Sarsa S, Bernstein S, Kim J, et al. Comparing Code Explanations Created by Students and Large Language Models. In: Proc. ITiCSE; 2023. p. 124–13 0. https://doi.org/10.1145/3587102.3588785

    2023

  32. [45]

    Inv estigating the Potential of GPT-3 in Providing Feedback for Programming Assessments

    Balse R, Valaboju B, Singhal S, Warriem JM, Prasad P . Inv estigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In: Proceedings of the 2023 Co nference on Innovation and T echnology in Computer Science Education V. 1 ITiCSE 2023, New Y ork, NY, ...

    2023

  33. [46]

    Large Language Models (GPT) for a utomating feedback on programming assignments

    Pankiewicz M, Baker R. Large Language Models (GPT) for a utomating feedback on programming assignments. In: Proceedings of the 31st International Conference on Comput ers in Education Matsue, Shimane, JAPAN; 2023

    2023

  34. [47]

    Next-Step Hint Generatio n for Introductory Programming Using Large Language Models

    Roest L, Keuning H, Jeuring J. Next-Step Hint Generatio n for Introductory Programming Using Large Language Models . In: Proceedings of the 26th Australasian Computing Educati on Conference ACE ’24, New Y ork, NY, USA: Association for Computing Machinery; 2024. p. 144–153. htt...

    2024

  35. [48]

    Exploring How Multiple Levels o f GPT-Generated Programming Hints Support or Disappoint Novices

    Xiao R, Hou X, Stamper J. Exploring How Multiple Levels o f GPT-Generated Programming Hints Support or Disappoint Novices. In: Extended Abstracts of the 2024 CHI Conference o n Human Factors in Computing Systems CHI EA ’24, New Y ork, NY, USA: Association for Computing Machine...

    2024

  36. [49]

    Using GPT-4 to Provide Tiered, Format ive Code Feedback

    Nguyen H, Allan V. Using GPT-4 to Provide Tiered, Format ive Code Feedback. In: Proceedings of the 55th ACM T echnical Symposium on Computer Science Education V. 1 S IGCSE 2024, New Y ork: ACM; 2024. p. 958–964. https://doi.org/10.1145/3626252.3630960

    2024

  37. [50]

    Automating Human T utor-Style Programming Feedback: Leveraging GPT-4 T utor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

    Phung T, Pădurean VA, Singh A, Brooks C, Cambronero J, Gu lwani S, et al. Automating Human T utor-Style Programming Feedback: Leveraging GPT-4 T utor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In: Proceedings of the 14th Learning Analytics and Kno...

    2024

  38. [51]

    Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A -Judge

    Koutcheme C, Dainese N, Sarsa S, Hellas A, Leinonen J, De nny P . Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A -Judge. In: Proceedings of the 2024 on Innovation and T echnology in Computer Science Education V. 1 I...

    2024

  39. [52]

    A Large Scale RCT on Effective Error Messages in CS1

    Wang S, Mitchell J, Piech C. A Large Scale RCT on Effective Error Messages in CS1. In: Proceedings of the 55th ACM T echnical Symposium on Computer Science Education V. 1 SIGCSE 2024, New Y ork, NY, USA: Association for Computing Machinery; 2024. p. 1395–1401. https://doi.org/1...

    2024

  40. [53]

    dcc –help: T ransforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language Models

    T aylor A, Vassar A, Renzella J, Pearce H. dcc –help: T ransforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language Models. In: Proceed ings of the 55th ACM T echnical Symposium on Computer Science Education V. 1 SIGCSE 2024, New Y o...

    2024

  41. [54]

    Enhancing Programming Error Messages in Real Time with Generative AI

    Kimmel B, Geisert AL, Y aro L, Gipson B, Hotchkiss RT, Osa e-Asante SK, et al. Enhancing Programming Error Messages in Real Time with Generative AI. In: Extended Abstr acts of the 2024 CHI Conference on Human Factors in Computing Systems CHI EA ’24, New Y ork, NY, USA: As soci...

    2024

  42. [55]

    https://github.com/Programming-Steps-Working-Group- 2022/public-datasets/tree/main/FITec

    Lehtinen T, FIT ech dataset; 2022. https://github.com/Programming-Steps-Working-Group- 2022/public-datasets/tree/main/FITec

    2022

  43. [56]

    https://doi.org/10.21249/DZHW:studentsteps:1.0.0, datenerhebung: 2017

    Kiesler N, Dataset: Recursive problem solving in the on line learning environment CodingBat by computer science st u- dents; 2022. https://doi.org/10.21249/DZHW:studentsteps:1.0.0, datenerhebung: 2017. Version: 1.0.0. Datenpaket- zugangsweg: Download-SUF . Hannover: FDZ-DZHW. ...

    2022 doi

  44. [57]

    https://metadata.fdz.dzhw.eu/public/files/data-pack ages/stu-studentsteps$/attachments/studentst

    Kiesler N, Daten- und Methodenbericht Rekursive Probl emlösung in der Online Lernumgebung CodingBat durch Informatik-Studierende; 2022. https://metadata.fdz.dzhw.eu/public/files/data-pack ages/stu-studentsteps$/attachments/studentst

    2022

  45. [58]

    T askT rack er-T ool: A T oolkit for T racking of Code Snapshots and Activ- ity Data During Solution of Programming T asks

    L yulina E, Birillo A, Kovalenko V, Bryksin T. T askT rack er-T ool: A T oolkit for T racking of Code Snapshots and Activ- ity Data During Solution of Programming T asks. In: Proceedi ngs of the 52nd ACM T echnical Symposium on Com- puter Science Education SIGCSE ’21, New Y or...

    2021

  46. [59]

    On Designing Programming Error Mes- sages for Novices: Readability and its Constituent Factors

    Denny P , Prather J, Becker BA, Mooney C, Homer J, Albrech t ZC, et al. On Designing Programming Error Mes- sages for Novices: Readability and its Constituent Factors . In: Proceedings of the 2021 CHI Conference on Hu- man Factors in Computing Systems CHI ’21, New Y ork, NY, U...

    2021

  47. [60]

    A Guide to Conducting Consensual Qualitative Research

    Hill CE, Thompson BJ, Williams EN. A Guide to Conducting Consensual Qualitative Research. The Counseling Psychol- ogist 1997;25(4):517–572. https://doi.org/10.1177/0011000097254001

    1997 doi

  48. [61]

    Consensual qualitative research: An update

    Hill CE, Knox S, Thompson BJ, Williams EN, Hess SA, Ladan y N. Consensual qualitative research: An update. Journal of counseling psychology 2005;52(2):196

    2005

  49. [62]

    arXiv; 2023

    OpenAI, GPT-4 T echnical Report. arXiv; 2023. https://arxiv.org/abs/2303.08774. 28 Lohr et al

    2023 arXiv

  50. [63]

    Why J ohnny Can’t Prompt: How Non-AI Experts T ry (and Fail) to Design LLM Prompts

    Zamfirescu-Pereira JD, Wong RY, Hartmann B, Y ang Q. Why J ohnny Can’t Prompt: How Non-AI Experts T ry (and Fail) to Design LLM Prompts. In: Proceedings of the 2023 CHI Confer ence on Human Factors in Computing Systems CHI ’23, New Y ork, NY, USA: Association for Computing Mach...

    2023

  51. [64]

    https://platform.openai.com/docs/guides/prompt-engi neering

    OpenAI, OpenAI API Documentation: Prompt Engineering ; 2024. https://platform.openai.com/docs/guides/prompt-engi neering

    2024

  52. [65]

    Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests

    Hellas A, Leinonen J, Sarsa S, Koutcheme C, Kujanpää L, S orva J. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In: Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 Chicago IL USA: ACM; 2023. p. 93–105

    2023

  53. [66]

    Employing subg oals in computer programming education

    Margulieux LE, Catrambone R, Guzdial M. Employing subg oals in computer programming education. Computer Science Education 2016;26(1):44–67. https://doi.org/10.1080/08993408.2016.1144429

    2016

  54. [67]

    Selection-Inferenc e: Exploiting Large Language Models for Interpretable Logi cal Reasoning

    Creswell A, Shanahan M, Higgins I. Selection-Inferenc e: Exploiting Large Language Models for Interpretable Logi cal Reasoning. In: International Conference on Learning Repre sentations; 2023

    2023

  55. [68]

    It’s Weird That it Knows What I Want

    Prather J, Reeves BN, Denny P , Becker BA, Leinonen J, Lux ton-Reilly A, et al. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Program mers. ACM T rans Comput-Hum Interact 2023 nov;31(1). https://doi.org/10.1145/3617367

    2023 doi

  56. [69]

    What Skills Do Y ou Need Whe n Developing Software Using ChatGPT? (Discussion Paper)

    Jeuring J, Groot R, Keuning H. What Skills Do Y ou Need Whe n Developing Software Using ChatGPT? (Discussion Paper). In: Proceedings of the 23rd Koli Calling Internatio nal Conference on Computing Education Research Koli Calling ’23, New Y ork, NY, USA: Association for Computi...

    2024

  57. [70]

    Adaptive Learning Systems in Programming Education: A Prototype for Enhanced Formative Feedback

    Lohr D, Berges M, Chugh A, Striewe M. Adaptive Learning Systems in Programming Education: A Prototype for Enhanced Formative Feedback. In: Proceedings of DELFI 2024. Fulda: G esellschaft für Informatik e.V.; 2024. p. 549–554

    2024

  58. [2024]

    p. 52–58. https://doi.org/10.1145/3649217.3653612

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.