pith. sign in

arxiv: 2605.16933 · v1 · pith:Y4ON2ORUnew · submitted 2026-05-16 · 💻 cs.HC

The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

Tsvetomila Mihaylova , Evanfiya Logacheva , Arto Hellas , Jing Fan , Francisco Castro , Bita Akram , Narges Norouzi , Peter Brusilovsky
show 1 more author
Juho Leinonen
This is my paper

Pith reviewed 2026-05-19 20:36 UTC · model grok-4.3

classification 💻 cs.HC
keywords LLM-generated feedbackprogramming assignmentsautomated feedbacktime to solutionfeedback guidancestudent performanceonline programming coursecompiler error messages
0
0 comments X

The pith

LLM-generated feedback is associated with faster time to solution for programming assignments than compiler messages alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to determine whether feedback produced by large language models with different levels of guidance can change how students solve programming problems. The authors ran the test in an online course by giving some students one of three feedback versions and others only the usual compiler messages. The results linked the presence of LLM feedback to quicker arrival at correct code, with the less guided type showing a modest advantage. A sympathetic reader would care because scaling good feedback has always been difficult in programming classes, and this suggests a workable alternative that does not require extra human effort for each student. The study also examines whether students with more or less experience benefit differently.

Core claim

The paper claims that students who received any of the three structured forms of LLM-generated feedback on their programming errors completed their assignments in less time than those limited to compiler error messages. The version with less guidance performed slightly better than the more guided alternatives. The authors state that feedback structure matters for how students advance toward correct solutions and that this supports continued work on designs that adapt to individual needs and on measuring longer-term learning gains.

What carries the argument

Varying levels of guidance in LLM-generated feedback, compared to a baseline of compiler error messages only, with the key measures being time to solution and number of attempts made by students.

If this is right

  • Students solve programming problems faster when LLM feedback supplements compiler messages.
  • Less guided feedback may yield slightly better results than more detailed guidance.
  • Prior programming experience can moderate how much benefit students receive from the feedback.
  • The structure of feedback affects the path students take to correct their code.
  • These findings encourage development of adaptive feedback systems for improved outcomes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Less guided feedback might help students develop stronger independent debugging skills by requiring more self-directed thinking.
  • Integrating such LLM feedback into standard development environments could provide real-time support without disrupting the coding flow.
  • The short-term speed gains might lead to improved retention of programming concepts if tracked over multiple assignments.
  • Similar structured feedback approaches could be applied in other technical fields involving error correction, such as data analysis or hardware design.

Load-bearing premise

The three feedback types must have delivered genuinely different degrees of guidance and students must have read and applied the feedback to their code rather than ignoring the messages.

What would settle it

Running the same experiment again and finding that students in all feedback conditions took the same amount of time or more to reach solutions as those in the compiler-only condition would disprove the observed association.

Figures

Figures reproduced from arXiv: 2605.16933 by Arto Hellas, Bita Akram, Evanfiya Logacheva, Francisco Castro, Jing Fan, Juho Leinonen, Narges Norouzi, Peter Brusilovsky, Tsvetomila Mihaylova.

Figure 1
Figure 1. Figure 1: Effects of hint type on Time to Success, compared to the No Feedback baseline [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Interaction plot of hint type and student level, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: The three prompt variants used in the study for the different types of feedback. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples for each of the three variants of feedback used in the study. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Student random slopes for Time to Success. C.2 Experiment 2 Additional Results: Effect of the feedback for students with different expertise. The fixed-effect estimates for log time to success with hint type × student level interaction are re￾ported in [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
read the original abstract

When programming students encounter errors in their code, compiler messages or static analysis output often provide limited guidance, particularly for novice programmers. Personalized feedback from instructors can be effective but does not scale well. Recent advances in large language models (LLMs) enable automated feedback generation at scale. This study examines whether LLM-generated feedback with different levels of guidance is associated with differences in students' problem-solving behavior. We analyze effects on time to solution and number of attempts, and examine whether these effects differ by programming experience. We design three feedback types and compare them to a baseline in which students receive only compiler error messages. Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects. Overall, the findings suggest that feedback structure plays an important role in how students progress toward correct solutions and motivate further work on adaptive feedback designs and longer-term learning outcomes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript reports an empirical study in an online programming course comparing three types of LLM-generated feedback (varying in guidance level) against a compiler-error-only baseline. It claims an association between LLM feedback and faster time to solution, with less-guided variants showing slightly stronger effects; secondary outcomes include number of attempts and moderation by prior programming experience.

Significance. If the reported associations prove robust, the work would contribute to HCI and CS education research by providing evidence on how feedback structure influences problem-solving efficiency at scale. The authentic course setting offers ecological validity, and the focus on guidance levels could inform design of LLM-based educational tools. However, the current lack of statistical reporting and engagement measures limits the strength of any conclusions about feedback effectiveness.

major comments (2)
  1. [Abstract / Results] Abstract and Results section: the central claim of an association with faster time to solution is presented without sample sizes per condition, statistical tests, effect sizes, confidence intervals, or details on randomization and controls. This prevents evaluation of whether the observed differences are reliable or practically meaningful.
  2. [Methods] Methods / Experimental Design: the interpretation that less-guided feedback produces stronger effects assumes students read and differentially responded to the three feedback types. No engagement metrics (view logs, time spent, clicks, or self-report) are reported, leaving open the possibility that time differences reflect unmeasured confounders rather than feedback structure.
minor comments (2)
  1. [Abstract] Clarify in the abstract or introduction how the three feedback types were operationalized to ensure they differed meaningfully in guidance level.
  2. [Discussion] Consider adding a limitations subsection discussing potential self-selection or non-compliance with feedback.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and indicate the revisions we will make to strengthen the statistical transparency and discussion of study limitations.

read point-by-point responses

  1. Referee: [Abstract / Results] Abstract and Results section: the central claim of an association with faster time to solution is presented without sample sizes per condition, statistical tests, effect sizes, confidence intervals, or details on randomization and controls. This prevents evaluation of whether the observed differences are reliable or practically meaningful.

    Authors: We agree that the abstract and Results section require more complete statistical reporting to allow proper evaluation. In the revised manuscript we will add the per-condition sample sizes, the exact statistical tests performed (with any assumption checks), effect sizes, 95% confidence intervals, and an explicit description of the randomization procedure and controls. These details exist in our analysis but were not fully summarized in the submitted version. revision: yes

  2. Referee: [Methods] Methods / Experimental Design: the interpretation that less-guided feedback produces stronger effects assumes students read and differentially responded to the three feedback types. No engagement metrics (view logs, time spent, clicks, or self-report) are reported, leaving open the possibility that time differences reflect unmeasured confounders rather than feedback structure.

    Authors: This is a fair observation. Our study did not collect direct engagement metrics such as feedback view logs, reading time, or self-reports. Random assignment to conditions within the live course platform provides some protection against group-level confounders, but we cannot rule out differential engagement. We will add an explicit discussion of this limitation in the revised Limitations section and note it as an important direction for future work that incorporates behavioral logging. revision: partial

Circularity Check

0 steps flagged

Empirical study with no derivation chain or self-referential claims

full rationale

The paper reports results from an experimental comparison of LLM-generated feedback conditions versus a compiler-only baseline in an online programming course, measuring associations with time-to-solution and attempts. No equations, fitted parameters, predictions derived from models, or self-citations appear in the abstract or described design. The central claims rest on observed differences across conditions rather than any reduction to self-defined inputs or imported uniqueness theorems. This is a standard empirical HCI study whose findings are externally falsifiable via replication and do not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard assumptions of educational experiments rather than new mathematical axioms or invented entities.

axioms (2)
  • domain assumption Time to solution and number of attempts are valid proxies for problem-solving behavior and learning progress.
    Invoked when interpreting faster solutions as a positive outcome of feedback.
  • domain assumption Students were randomly assigned or balanced across feedback conditions.
    Required for causal interpretation of the association reported in the abstract.

pith-pipeline@v0.9.0 · 5726 in / 1256 out tokens · 22586 ms · 2026-05-19T20:36:28.362267+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 69 canonical work pages

  1. [1]

    Moraes, Fernanda Oliveira, and Carla A

    Juliana Barros, Laura O. Moraes, Fernanda Oliveira, and Carla A. D. M. Delgado. 2025. https://doi.org/10.1007/978-3-031-98417-4_30 Large language models generating feedback for students of introductory programming courses . In 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 2, volume 15877 of Lecture Notes in Compute...

    work page doi:10.1007/978-3-031-98417-4_30 2025

  2. [3]

    Encarnacion

    Craig Boyle and Antonio O. Encarnacion. 1994. Metadoc: an adaptive hypertext reading system. User Modeling and User-Adapted Interaction, 4(1):1--19

    work page 1994

  3. [4]

    Breslow and David G

    Norman E. Breslow and David G. Clayton. 1993. https://doi.org/10.2307/2290687 Approximate inference in generalized linear mixed models . Journal of the American Statistical Association, 88(421):9--25

    work page doi:10.2307/2290687 1993

  4. [5]

    Neil CC Brown, Pierre Weill-Tessier, Juho Leinonen, Paul Denny, and Michael K \"o lling. 2025. Howzat? appealing to expert judgement for evaluating human and ai next-step hints for novice programmers. ACM Transactions on Computing Education, 25(3):1--43

    work page 2025

  5. [6]

    Marc Brysbaert and Micha \"e l Stevens. 2018. Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1):9

    work page 2018

  6. [7]

    Tom \'a s Capretto, Camen Piho, Ravin Kumar, Jacob Westfall, Tal Yarkoni, and Osvaldo A Martin. 2022. Bambi: A simple interface for fitting bayesian linear models in python. Journal of Statistical Software, 103:1--29

    work page 2022

  7. [8]

    Yung-Ting Chuang and Hsin-Yu Chang. 2024. Analyzing novice and competent programmers' problem-solving behaviors using an automated evaluation system. Science of Computer Programming, 237:103138

    work page 2024

  8. [10]

    Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanp \"a \"a , and Juha Sorva. 2023. Exploring the responses of large language models to beginner programmers’ help requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, pages 93--105

    work page 2023

  9. [11]

    Joseph M Hilbe. 2011. Negative binomial regression. Cambridge University Press

    work page 2011

  10. [12]

    Slava Kalyuga. 2007. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review, 19(4):509--539

    work page 2007

  11. [13]

    Slava Kalyuga. 2009. The expertise reversal effect. In Managing cognitive load in adaptive multimedia learning, pages 58--80. IGI Global Scientific Publishing

    work page 2009

  12. [15]

    Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, Syed Ashraf, and Paul Denny. 2025. Evaluating language models for generating and judging programming feedback. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 624--630

    work page 2025

  13. [16]

    Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024. Open source language models can provide feedback: Evaluating llms' ability to help students using gpt-4-as-a-judge. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, pages 52--58. Association for Computing Machinery, N...

    work page 2024

  14. [17]

    Oka Kurniawan, Christopher M Poskitt, Ismam Al Hoque, Norman Tiong Seng Lee, Cyrille J \'e gourel, and Nachamma Sockalingam. 2023. How helpful do novice programmers find the feedback of an automated repair tool? In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pages 1--6. IEEE

    work page 2023

  15. [19]

    Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pages 563--569

    work page 2023

  16. [20]

    Mary J Lindstrom and Douglas M Bates. 1988. Newton—raphson and em algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83(404):1014--1022

    work page 1988

  17. [23]

    Maciej Pankiewicz and Ryan Baker. 2023. https://arxiv.org/abs/2307.00150 Large language models (gpt) for automating feedback on programming assignments . In 31st International Conference on Computers in Education (ICCE’2023), pages 68--77

    work page arXiv 2023

  18. [24]

    Tung Phung, Heeryung Choi, Mengyan Wu, Adish Singla, and Christopher Brooks. 2025. https://doi.org/10.1007/978-3-031-98414-3_1 Plan more, debug less: Applying metacognitive theory to ai-assisted programming education . In 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 1, volume 15877 of Lecture Notes in Computer Sci...

    work page doi:10.1007/978-3-031-98414-3_1 2025

  19. [25]

    Skipper Seabold and Josef Perktold. 2010. https://doi.org/10.25080/Majora-92bf1922-011 statsmodels: Econometric and statistical modeling with python . In Proceedings of the 9th Python in Science Conference, pages 92--96

    work page doi:10.25080/majora-92bf1922-011 2010

  20. [26]

    Raj Shrestha, Juho Leinonen, Albina Zavgorodniaia, Arto Hellas, and John Edwards. 2022. Pausing while programming: Insights from keystroke analysis. in 2022 ieee/acm 44th international conference on software engineering: Software engineering education and training (icse-seet)

    work page 2022

  21. [28]

    Olga Viberg, Jacqueline Wong, Yael Feldman-Maggor, Nora Dunder, and Carrie Demmans Epp. 2025. https://doi.org/10.1007/978-3-031-98465-5_57 Chatting with code: Exploring llms as learning partners in programming education . In 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 6, volume 15882 of Lecture Notes in Computer ...

    work page doi:10.1007/978-3-031-98465-5_57 2025

  22. [29]

    Ruiwei Xiao, Xinying Hou, and John Stamper. 2024. Exploring how multiple levels of gpt-generated programming hints support or disappoint novices. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1--10

    work page 2024

  23. [30]

    Messer, Marcus and Brown, Neil C. C. and K\". Automated Grading and Feedback Tools for Programming Education: A Systematic Review , year =. doi:10.1145/3636515 , journal =

    work page doi:10.1145/3636515

  24. [31]

    , title =

    Becker, Brett A. , title =. 2016 , isbn =. doi:10.1145/2839509.2844584 , booktitle =

    work page doi:10.1145/2839509.2844584 2016

  25. [32]

    2024 , isbn =

    Vassar, Alexandra and Renzella, Jake and Ross, Emily and Taylor, Andrew , title =. 2024 , isbn =. doi:10.1145/3699538.3699581 , booktitle =

    work page doi:10.1145/3699538.3699581 2024

  26. [33]

    2021 , isbn =

    Denny, Paul and Whalley, Jacqueline and Leinonen, Juho , title =. 2021 , isbn =. doi:10.1145/3441636.3442309 , booktitle =

    work page doi:10.1145/3441636.3442309 2021

  27. [34]

    2022 , isbn =

    Leinonen, Juho and Denny, Paul and Whalley, Jacqueline , title =. 2022 , isbn =. doi:10.1145/3478431.3499372 , booktitle =

    work page doi:10.1145/3478431.3499372 2022

  28. [35]

    2025 , isbn =

    Mueller, Moritz and List, Corinna and Kipp, Michael , title =. 2025 , isbn =. doi:10.1145/3723010.3723034 , booktitle =

    work page doi:10.1145/3723010.3723034 2025

  29. [36]

    2023 , isbn =

    Jell, Lea and List, Corinna and Kipp, Michael , title =. 2023 , isbn =. doi:10.1145/3593663.3593692 , booktitle =

    work page doi:10.1145/3593663.3593692 2023

  30. [37]

    2024 , isbn =

    Taylor, Andrew and Vassar, Alexandra and Renzella, Jake and Pearce, Hammond , title =. 2024 , isbn =. doi:10.1145/3626252.3630822 , booktitle =

    work page doi:10.1145/3626252.3630822 2024

  31. [38]

    Journal of Computer Assisted Learning , volume=

    You're (Not) My Type-Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks? , author=. Journal of Computer Assisted Learning , volume=. 2025 , publisher=

    work page 2025

  32. [39]

    2025 , isbn =

    Koutcheme, Charles and Dainese, Nicola and Sarsa, Sami and Hellas, Arto and Leinonen, Juho and Ashraf, Syed and Denny, Paul , title =. 2025 , isbn =. doi:10.1145/3641554.3701791 , booktitle =

    work page doi:10.1145/3641554.3701791 2025

  33. [40]

    2024 , isbn =

    Gabbay, Hagit and Cohen, Anat , title =. 2024 , isbn =. doi:10.1145/3657604.3662040 , booktitle =

    work page doi:10.1145/3657604.3662040 2024

  34. [41]

    2018 , issue_date =

    Keuning, Hieke and Jeuring, Johan and Heeren, Bastiaan , title =. 2018 , issue_date =. doi:10.1145/3231711 , journal =

    work page doi:10.1145/3231711 2018

  35. [42]

    Managing cognitive load in adaptive multimedia learning , pages=

    The expertise reversal effect , author=. Managing cognitive load in adaptive multimedia learning , pages=. 2009 , publisher=

    work page 2009

  36. [43]

    Educational psychology review , volume=

    Expertise reversal effect and its implications for learner-tailored instruction , author=. Educational psychology review , volume=. 2007 , publisher=

    work page 2007

  37. [44]

    2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) , pages=

    How Helpful do Novice Programmers Find the Feedback of an Automated Repair Tool? , author=. 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) , pages=. 2023 , organization=

    work page 2023

  38. [45]

    Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , pages=

    Exploring how multiple levels of GPT-generated programming hints support or disappoint novices , author=. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , pages=

    work page

  39. [46]

    Science of Computer Programming , volume=

    Analyzing novice and competent programmers' problem-solving behaviors using an automated evaluation system , author=. Science of Computer Programming , volume=. 2024 , publisher=

    work page 2024

  40. [47]

    In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) , author=

    Pausing While Programming: Insights From Keystroke Analysis. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) , author=. 2022 , publisher=

    work page 2022

  41. [48]

    Journal of the royal statistical society , volume=

    On the interpretation of 2 from contingency tables, and the calculation of P , author=. Journal of the royal statistical society , volume=. 1922 , publisher=

    work page 1922

  42. [49]

    Statistics surveys , volume=

    Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules , author=. Statistics surveys , volume=

    work page

  43. [50]

    The annals of mathematical statistics , pages=

    On a test of whether one of two random variables is stochastically larger than the other , author=. The annals of mathematical statistics , pages=. 1947 , publisher=

    work page 1947

  44. [51]

    Journal of the American statistical Association , volume=

    Use of ranks in one-criterion variance analysis , author=. Journal of the American statistical Association , volume=. 1952 , publisher=

    work page 1952

  45. [52]

    Corder, Dale I

    Nonparametric statistics for non-statisticians: A step-by-step approach by Gregory W. Corder, Dale I. Foreman , author=. 2010 , publisher=

    work page 2010

  46. [53]

    The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , volume =

    Karl Pearson , title =. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , volume =. 1900 , publisher =. doi:10.1080/14786440009463897 , URL =

    work page doi:10.1080/14786440009463897 1900

  47. [54]

    Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Using large language models to enhance programming error messages , author=. Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 , pages=

    work page

  48. [55]

    31st International Conference on Computers in Education (ICCE’2023) , pages =

    Pankiewicz, Maciej and Baker, Ryan , title =. 31st International Conference on Computers in Education (ICCE’2023) , pages =. 2023 , abstract =

    work page 2023

  49. [56]

    2025 , booktitle =

    Viberg, Olga and Wong, Jacqueline and Feldman-Maggor, Yael and Dunder, Nora and Demmans Epp, Carrie , title =. 2025 , booktitle =

    work page 2025

  50. [57]

    26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 1 , series =

    Phung, Tung and Choi, Heeryung and Wu, Mengyan and Singla, Adish and Brooks, Christopher , title =. 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 1 , series =. 2025 , pages =

    work page 2025

  51. [58]

    and Oliveira, Fernanda and Delgado, Carla A

    Barros, Juliana and Moraes, Laura O. and Oliveira, Fernanda and Delgado, Carla A. D. M. , title =. 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 2 , series =. 2025 , abstract =

    work page 2025

  52. [59]

    , title =

    Boyle, Craig and Encarnacion, Antonio O. , title =. User Modeling and User-Adapted Interaction , volume =. 1994 , type =

    work page 1994

  53. [60]

    Proceedings of the 9th Python in Science Conference , year =

    statsmodels: Econometric and Statistical Modeling with Python , author =. Proceedings of the 9th Python in Science Conference , year =

    work page

  54. [61]

    Journal of the American Statistical Association , volume=

    Newton—Raphson and EM algorithms for linear mixed-effects models for repeated-measures data , author=. Journal of the American Statistical Association , volume=. 1988 , publisher=

    work page 1988

  55. [62]

    Journal of cognition , volume=

    Power analysis and effect size in mixed effects models: A tutorial , author=. Journal of cognition , volume=

    work page

  56. [63]

    2011 , publisher=

    Negative binomial regression , author=. 2011 , publisher=

    work page 2011

  57. [64]

    Journal of Statistical Software , volume=

    Bambi: A simple interface for fitting Bayesian linear models in Python , author=. Journal of Statistical Software , volume=

    work page

  58. [65]

    Journal of the American Statistical Association , volume =

    Approximate Inference in Generalized Linear Mixed Models , author =. Journal of the American Statistical Association , volume =. 1993 , doi =

    work page 1993

  59. [66]

    Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 , pages=

    Exploring the responses of large language models to beginner programmers’ help requests , author=. Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 , pages=

    work page 2023

  60. [67]

    Proceedings of the 56th ACM Technical Symposium on Computer Science Education V

    Evaluating language models for generating and judging programming feedback , author=. Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 , pages=

    work page

  61. [68]

    Proceedings of the 2024 on Innovation and Technology in Computer Science Education V

    Open source language models can provide feedback: Evaluating llms' ability to help students using gpt-4-as-a-judge , author=. Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 , pages=. 2024 , publisher=

    work page 2024

  62. [69]

    ACM Transactions on Computing Education , volume=

    Howzat? Appealing to expert judgement for evaluating human and AI next-step hints for novice programmers , author=. ACM Transactions on Computing Education , volume=. 2025 , publisher=

    work page 2025

  63. [70]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  64. [71]

    Publications Manual , year = "1983", publisher =

    work page 1983

  65. [72]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  66. [73]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

    work page

  67. [74]

    Dan Gusfield , title =. 1997

    work page 1997

  68. [75]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    work page 2015

  69. [76]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

    work page