BugSpotter: Automated Generation of Code Debugging Exercises

Adish Singla; Paul Denny; Victor-Alexandru P\u{a}durean

REVIEW 3 major objections 5 minor 2 cited by

BugSpotter: Automated Generation of Code Debugging Exercises

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BugSpotter generates debugging exercises with an LLM, and a 741-student classroom study finds students perform on them as well as on instructor-written exercises.

desk verdict A genuinely useful debugging-exercise generator with a solid classroom study, but the comparability claim rests on non-significant chi-square tests and a curated exercise set. read the letter →

arxiv 2411.14303 v2 pith:QPQKZP2R submitted 2024-11-21 cs.SE cs.AI

Victor-Alexandru Pădurean , Paul Denny , Adish Singla This is my paper · ORCID

classification cs.SEcs.AI

keywords debuggingprogrammingeducationexercisegenerationlargelanguagemodelstestcasedesignBugSpottercomputingautomatedvalidation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BugSpotter is a tool that turns a plain programming-problem specification into a debugging exercise: a large language model generates a buggy version of the intended function and a corrected version, and an automated stage checks both against the problem's test suite before showing the exercise to a student. The student's task is not to fix the code but to design a failing test case (input, buggy output, correct output), which requires reading the specification carefully and localizing the bug. The paper's central claim is that such LLM-generated exercises are usable in real classrooms: expert review rated them as well-matched and diverse, and in a deployment with 741 students, success rates on generated exercises were statistically indistinguishable from success rates on exercises hand-crafted by an instructor. If this holds, educators could generate debugging practice material automatically, easing a part of teaching that is often inconsistent and labor-intensive.

What carries the argument

The load-bearing mechanism is a two-stage generation-and-validation pipeline. In the generation stage, the LLM is prompted to reason about likely student bugs and output ten JSON tuples, each containing a buggy code, its fixed version, and an explanation. In the validation stage, a candidate is kept only if the fixed version compiles and passes the problem's entire test suite, the buggy version compiles and fails at least one test (by wrong output or runtime error, excluding hangs), and execution stays within a time limit. The student-facing exercise asks for a failing test case satisfying three machine-checkable criteria: the input makes buggy and fixed outputs differ, the reported correct output matches the fixed code, and the reported buggy output matches the buggy code. This design makes exercise quality depend on executable checks rather than on human curation.

What would settle it

Run BugSpotter in real time without pre-selection in the same course and compare student success rates against instructor-created exercises on the same problems; if the chi-square test shows a significant difference (p < 0.05) or the uncurated pipeline frequently fails validation, the comparability claim would not transfer to live use.

Watch

Extended reading notes

Core claim

On the paper's own terms, BugSpotter establishes that LLM-generated debugging exercises can be substituted for instructor-authored ones without measurable loss in student performance. Across three single-function C problems, 741 introductory students each attempted exercises from a random assignment of five LLM-generated and five instructor-created variants per problem, and chi-square tests found no significant difference in success rates by source; the smallest p-value was 0.065 on one problem, with an aggregate p-value of 0.140. Expert ratings of difficulty also aligned with observed student success: exercises classed easy, medium, and hard produced descending success rates. The paper further reports that a cheaper model generated exercises comparable in quality and diversity to a more expensive one, and passed validation more often, making the pipeline cost-effective.

Load-bearing premise

The classroom comparison assumes that five pre-selected exercises per problem, hand-picked for quality and diversity, represent what BugSpotter would generate in regular use; the study did not test real-time generation.

Editorial extensions

If this is right

Instructors can produce debugging exercises for single-function C problems without hand-crafting bugs, with the validation stage replacing human quality control.
Because the cheaper model performed comparably to the more expensive one, cost need not block adoption of the pipeline.
Students get practice reading problem specifications and designing discriminating test cases, not just fixing code.
Exercises can be sorted or recommended by difficulty, since expert-ranked difficulty predicted student success rates.
The same pipeline can be extended to other languages and to multi-function or class-based problems, as the demo already supports Python.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the tool were used without pre-selection, average exercise difficulty could shift from what the study measured, since the authors chose five high-quality exercises per problem rather than sampling all validated output.
The p-value of 0.065 on one problem leaves room for a real difficulty gap on certain problems; more problems and larger samples would tell whether aggregate comparability holds everywhere.
Since a correct attempt requires matching both the buggy and correct outputs exactly, the exercise format doubles as a test of specification comprehension, which the paper does not directly measure.
Fine-tuning the generator on actual student bug-fixing attempts could make the exercises resemble natural novice errors more closely than the current prompt asking for bugs students may make.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

BugSpotter is a real, useful addition to the LLM-for-education toolbox. The exercise format is the best part: a student sees a C problem specification and a buggy implementation, then has to supply an input, the buggy output, and the correct output, with all three checked by actually running both programs. That turns debugging practice into something auto-gradable and forces students to engage with the specification rather than just poking at code. The generation pipeline is also sensible: prompt an LLM for buggy/fixed pairs with explanations, filter out non-compiling or unrelated code, and require that the fixed version passes the problem's test suite while the buggy version fails at least one test. There is no circularity here; the evaluation compares against instructor-created exercises, not against the pipeline's own outputs.

The classroom study is a serious effort: 741 students, random assignment across five LLM-generated and five instructor-written exercises per problem. The expert rubric, with two annotators and kappa above 0.7, is reasonable. The authors are also candid in Section 5.5 about pre-selection, the lack of long-term correlation with course performance, and the absence of a direct educator-workload measurement.

The soft spot is the one the stress-test flags, and it is load-bearing. Section 5.3 and the abstract claim "comparable difficulty" based solely on chi-square tests where every p-value exceeds 0.05, with the lowest at 0.065. Non-significance is not evidence of equivalence. The observed success rates are consistently lower for LLM-generated exercises, and the sample sizes give limited power to detect educationally meaningful differences. To support the substitution claim, the paper needs confidence intervals, a pre-specified equivalence margin, or a power analysis. This is a statistical-inference gap, not a sign of sloppy execution; the authors simply reached one step past what the data support.

The pre-selection issue is real but secondary. The study evaluates a curated sample of five exercises per problem, not the live generation pipeline, so the deployment overstates what an uncurated tool would deliver. The authors acknowledge this. A minor mismatch: the public demo supports Python while the evaluation is in C.

Who is this for: computing education researchers and instructors considering LLM-based exercise generation. It deserves a serious referee and likely acceptance with a requested revision that reanalyzes the RQ3 data using interval estimates and softens the comparability language. I would cite it and bring it to a reading group.

Referee Report

3 major / 5 minor

Summary. The paper introduces BugSpotter, a tool that uses an LLM to generate buggy C code from problem specifications, validates the code against a test suite, and presents students with debugging exercises in which they must design failing test cases. The authors report an expert-based quality/diversity evaluation (RQ1), an analysis of difficulty alignment between expert labels and student success (RQ2), and a classroom study comparing student success on LLM-generated versus instructor-created debugging exercises (RQ3). The classroom study involved 741 students and three problems, with five pre-selected LLM-generated exercises and five instructor-created exercises per problem. The paper claims that LLM-generated exercises are comparable in difficulty to instructor-created ones and suggests that BugSpotter can reduce educator workload.

Significance. If the central claims are properly supported, this is a useful contribution to computing education: it addresses debugging practice, an underserved area, and provides an automated pipeline with a concrete validation step. Strengths include the well-specified exercise-generation and validation methodology, the use of an external baseline of instructor-created exercises, expert annotation with reported inter-rater agreement, and a deployed classroom evaluation. The paper also ships a public demo, which aids reproducibility. However, the statistical support for the central 'comparable difficulty' claim is currently weak, and the classroom study evaluates a curated subset rather than the live generation pipeline, so the significance of the practical recommendation is not yet established.

major comments (3)

[§5.3, Figure 6] The claim that LLM-generated exercises are 'comparable in difficulty' to instructor-created exercises rests on chi-square tests with p-values 0.675, 0.580, 0.065, and 0.140. A non-significant p-value only indicates failure to reject the null hypothesis of equal success rates; it does not provide positive evidence for equivalence, particularly with limited power and with Problem 3 close to conventional thresholds. The paper should report confidence intervals for the success-rate differences, a pre-specified equivalence margin, or an equivalence test (e.g., TOST), and ideally a power analysis; without this, the central practical recommendation in the abstract and Section 5.3 is not statistically supported.
[§4.2 and §5.5] The classroom evaluation did not use BugSpotter's real-time generation pipeline: the authors pre-selected 5 high-quality, diverse exercises per problem, as stated in Section 4.2, and Section 5.5 admits this. Consequently, student success rates measure the curated exercises, not the uncurated output that the tool would produce in live use; this weakens the claim that BugSpotter 'produced' exercises comparable to instructor-created ones. The Limitations section should be extended to state that the comparability result applies to the curated subset and does not yet establish the effectiveness of unmoderated generation.
[§5.2, Figure 5] The difficulty-diversity analysis in Section 5.2 labels the two easiest exercises as 'easy', the next two as 'medium', and the last as 'hard' after expert ranking, and then shows that student success rates decrease with these labels. With only five exercises per problem and no error bars or inferential test, this is a weak basis for the claim of 'strong alignment' between expert-assessed difficulty and student performance; the paper should provide per-exercise success counts or confidence intervals and temper the wording accordingly.

minor comments (5)

[§5.3] The text says '741 data points per problem' but the study assigns 741 students across ten exercises; please clarify whether the contingency table is per problem with 741 students or whether 741 is the total across all conditions, and report the per-arm counts.
[Figure 8] The heading 'Print A verage Rainfall' contains a stray space; it should be 'Print Average Rainfall'.
[§4.2] The phrase '5 instructed-created' should be '5 instructor-created'.
[§4.1] The phrase 'greater than0.7' is missing a space; it should read 'greater than 0.7'.
[§5.4] The web application supports Python exercises while the classroom study uses C; this discrepancy is acknowledged only implicitly and deserves a sentence of discussion in the limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BugSpotter's evaluation is empirical and benchmarked against instructor-created exercises.

full rationale

The paper does not derive a mathematical prediction from fitted inputs; it builds a generation tool and evaluates it empirically against an external benchmark (instructor-created exercises). The generation pipeline in Section 3.2 validates LLM output using the problem's own test suite (fixed code CF must pass all tests, buggy code CB must fail at least one), which is an external correctness criterion independent of the measured student outcomes. RQ2 compares expert-assigned difficulty labels against later student success rates, and RQ3 compares student success on LLM-generated versus instructor-created exercises; in both cases the student data are collected after exercise construction, not used to fit or define the exercises. The pre-selection of five high-quality exercises per problem, acknowledged in Sections 4.2 and 5.5, is a selection-bias concern about generalization to real-time generation, but it is not circularity because the selection does not use the outcome variable being measured (student success). Self-citations appear in related work and rubric grounding, but none is load-bearing as a uniqueness theorem or as a justification that reduces the central claim to the authors' prior work. The statistical concern that non-significant chi-square p-values do not establish equivalence is a correctness-risk issue, not a circularity issue: the p-values are computed from independent student-outcome data rather than from the construction of the exercises. No derivation step is equivalent to its input by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are introduced; BugSpotter is a software tool. The only hand-chosen settings are the LLM temperature and the pre-selection count, and the main assumptions concern test-suite validity, the success-rate measurement model, representativeness of pre-selected exercises, and expert rating reliability.

free parameters (2)

LLM sampling temperature = 0.7
Chosen by hand in Section 3.2 to introduce variability in generated exercises; not fitted to student data but affects the diversity and difficulty of exercises used.
Number of pre-selected exercises per problem = 5
The authors pre-selected five high-quality exercises for the classroom deployment (Section 4.2) rather than using real-time generation, which directly shapes the student performance comparison.

assumptions (4)

domain assumption The test suite included with each problem P correctly and completely captures the problem specification.
Section 3.2 uses the test suite to decide whether CF is correct and CB is buggy; an incomplete test suite could mislabel correct code as buggy or vice versa.
domain assumption Student success rate on an exercise is a valid measure of exercise difficulty and debugging skill.
RQ2 and RQ3 treat success rates as the outcome measure; the paper does not validate this measurement against other indicators such as long-term course performance (acknowledged in Section 5.5).
ad hoc to paper The five pre-selected exercises per problem are representative of BugSpotter's real-time outputs.
Section 4.2 pre-selected exercises to control quality; Section 5.5 states the classroom study did not use real-time generation.
domain assumption Expert ratings of difficulty and quality are reliable and generalizable.
Only two experts rated the exercises; Cohen's kappa greater than 0.7 is reported (Section 4.1), but no evidence connects these ratings to learning outcomes beyond the small sample of three problems.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BugSpotter: Automated Generation of Code Debugging Exercises." pith.science (2026). https://pith.science/paper/QPQKZP2R

@misc{pith2026241114303,
  author       = {Pith},
  title        = {Pith review of: BugSpotter: Automated Generation of Code Debugging Exercises},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPQKZP2R}},
  note         = {Machine review of arXiv:2411.14303}
}

read the original abstract

Debugging is an essential skill when learning to program, yet its instruction and emphasis often vary widely across introductory courses. In the era of code-generating large language models (LLMs), the ability for students to reason about code and identify errors is increasingly important. However, students frequently resort to trial-and-error methods to resolve bugs without fully understanding the underlying issues. Developing the ability to identify and hypothesize the cause of bugs is crucial but can be time-consuming to teach effectively through traditional means. This paper introduces BugSpotter, an innovative tool that leverages an LLM to generate buggy code from a problem description and verify the synthesized bugs via a test suite. Students interact with BugSpotter by designing failing test cases, where the buggy code's output differs from the expected result as defined by the problem specification. This not only provides opportunities for students to enhance their debugging skills, but also to practice reading and understanding problem specifications. We deployed BugSpotter in a large classroom setting and compared the debugging exercises it generated to exercises hand-crafted by an instructor for the same problems. We found that the LLM-generated exercises produced by BugSpotter varied in difficulty and were well-matched to the problem specifications. Importantly, the LLM-generated exercises were comparable to those manually created by instructors with respect to student performance, suggesting that BugSpotter could be an effective and efficient aid for learning debugging.

Figures

Figures reproduced from arXiv: 2411.14303 by the authors.

**Figure 1.** Illustration of a debugging exercise from BugSpotter for Problem 1, where a student’s objective is to design a failing test [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** BugSpotter’s exercise generation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Prompt for asking LLMs to generate buggy codes. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Results of the expert-based quality assessment. BugSpotter leverages LLMs from OpenAI’s GPT family [ [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Student success rates w.r.t. expert-assessed difficulty [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 7.** Figure 7: Illustration of a debugging exercise for Problem 2. The exercise contains a Type 1 buggy code, as it passes some of the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Illustration of a debugging exercise for Problem 3. The exercise contains a Type 3 buggy code, as it does not check [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations
cs.SE 2024-12 conditional novelty 7.0 of 10

In a large crowd-sourced experiment, natural-language descriptions of buggy algorithms improved debugging accuracy over Python code for users who understood the task, while hints mainly helped users working with code.
From Automation to Cognition: Redefining the Roles of Educators and Generative AI in Computing Education
cs.CY 2024-12 conditional novelty 3.0 of 10

Computing educators propose redesigning take-home assignments to include and assess student use of generative AI, while shifting educator focus to metacognitive skill development.

Reference graph

Works this paper leans on

43 extracted references · 40 canonical work pages · cited by 2 Pith papers

[1]

Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla

Umair Z. Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla. 2020. Synthesizing Tasks for Block-based Programming. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)

work page 2020
[2]

Georg Brandl, Matthäus Chajdas, and Jean Abou-Samra. 2006. Pygments. https: //pygments.org/

work page 2006
[3]

Serena Caraco, Nelson Lojo, Michael Verdicchio, and Armando Fox. 2024. Gen- erating Multi-Part Autogradable Faded Parsons Problems From Code-Writing Exercises. In Proceedings of the Technical Symposium on Computer Science Educa- tion (SIGCSE)

work page 2024
[4]

Mei-Wen Chen, Cheng-Chih Wu, and Yu-Tzu Lin. 2013. Novices’ Debugging Behaviors in VB Programming. In Proceedings of the Learning and Teaching in Computing and Engineering (LaTiCE)

work page 2013
[5]

William G Cochran. 1952. The 𝜒2 Test of Goodness of Fit. The Annals of Mathematical Statistics (1952)

work page 1952
[6]

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement (1960)

work page 1960
[7]

Heffernan, Tanja Käser, Steven Moore, Anna N

Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla. 2024. Generative AI for Education (GAIED): Advances, Opportunities, and Challenges. CoRR abs/2402.01580 (2024)

arXiv 2024
[8]

Becker, and Brent N

Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

work page 2024

Show all 43 references

[9]

Tempero, and Jacob Hendrickx

Paul Denny, Andrew Luxton-Reilly, Ewan D. Tempero, and Jacob Hendrickx
[10]

Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit

Paul Denny, James Prather, Brett A. Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit. 2019. A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. InKoli Calling International Conference on Computing Education Research (Koli Calling)

2019
[11]

Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N

Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2024. Computing Education in the Era of Generative AI.Commun. ACM (2024)

2024
[12]

Sue Fitzgerald, Gary Lewandowski, Renée McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: Finding, Fixing and Flailing, a Multi-institutional Study of Novice Debuggers. Computer Science Education 18 (2008)

2008
[13]

Ahana Ghosh, Sebastian Tschiatschek, Sam Devlin, and Adish Singla. 2022. Adap- tive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes. In Proceeding of the International Conference on Artificial Intelligence in Education AIED

2022
[14]

Andre Del Carpio Gutierrez, Paul Denny, and Andrew Luxton-Reilly. 2024. Eval- uating Automatically Generated Contextualised Programming Exercises. In Pro- ceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2024
[15]

Mollie Jordan, Kevin Ly, and Adalbert Gerald Soosai Raj. 2024. Need a Program- ming Exercise Generated in Your Native Language? ChatGPT’s Got Your Back: Automatic Generation of Non-English Programming Exercises Using OpenAI GPT-3.5. InProceedings of the Technical Symposium on ...

2024
[16]

Kafai, David DeLiema, Deborah A

Yasmin B. Kafai, David DeLiema, Deborah A. Fields, Gary Lewandowski, and Colleen M. Lewis. 2019. Rethinking Debugging as Productive Failure for CS Edu- cation. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2019
[17]

Ko, Thomas D

Amy J. Ko, Thomas D. LaToza, Stephen Hull, Ellen A. Ko, William Kwok, Jane Quichocho, Harshitha Akkaraju, and Rishin Pandit. 2019. Teaching Explicit Pro- gramming Strategies to Adolescents. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2019
[18]

Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2024. Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks

2024
[19]

Martin Labaj and Mária Bieliková. 2014. Utilization of Exercise Difficulty Rating by Students for Recommendation. In Proceedings of the International Conference on Web-Based Learning (ICWL)

2014
[20]

Reeves, Paul Denny, James Prather, and Brett A

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent N. Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2023
[21]

Chen Li, Emily Chan, Paul Denny, Andrew Luxton-Reilly, and Ewan D. Tempero
[22]

Nelson Lojo and Armando Fox. 2022. Teaching Test-Writing As a Variably- Scaffolded Programming Pattern. In Proceedings of the Conference on Innovation and Technology in Computer Science Education (ItiCSE)

2022
[23]

Qianou Ma, Hua Shen, Kenneth Koedinger, and Sherry Tongshuang Wu. 2024. How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging. In Proceeding of the International Conference on Artificial Intelligence in Education (AIED)

2024
[24]

Stephen MacNeil, Paul Denny, Andrew Tran, Juho Leinonen, Seth Bernstein, Arto Hellas, Sami Sarsa, and Joanne Kim. 2024. Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models. In Proceedings of the Australasian Computing Education Con...

2024
[25]

Renée McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: A Review of the Literature from an Educational Perspective. Computer Science Education (2008)

2008
[26]

Ismael Villegas Molina, Audria Montalvo, Benjamin Ochoa, Paul Denny, and Leo Porter. 2024. Leveraging LLM Tutoring Systems for Non-Native English Speakers in Introductory CS Courses

2024
[27]

Laurie Murphy, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: the Good, the Bad, and the Quirky – A Qualitative Analysis of Novices’ Strategies. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2008
[28]

Manh Hung Nguyen, Sebastian Tschiatschek, and Adish Singla. 2024. Large Language Models for In-Context Student Modeling: Synthesizing Student’s Be- havior in Visual Programming from One-Shot Observation. In Proceedings of the International Conference on Educational Data Mining (EDM)

2024
[29]

OpenAI. 2023. ChatGPT. https://openai.com/blog/chatgpt

2023
[30]

OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

2024
[31]

OpenAI. 2024. OpenAI Platform Models. https://platform.openai.com/docs/ models

2024
[32]

Victor-Alexandru Pădurean and Adish Singla. 2024. Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks

2024
[33]

Victor-Alexandru Pădurean, Georgios Tzannetos, and Adish Singla. 2024. Neural Task Synthesis for Visual Programming. Transactions on Machine Learning Research (TMLR) (2024)

2024
[34]

Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. In Proceedings of the International Conference on Educational Data Min...

2023
[35]

Tung Phung, Victor-Alexandru Padurean, José Cambronero, Sumit Gulwani, To- bias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. In Proceedings of the Conference on Internation...

2023
[36]

Tung Phung, Victor-Alexandru Padurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. 2024. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hi...

2024
[37]

Kate Sanders et al. 2013. The Canterbury QuestionBank: Building a Repository of Multiple-Choice CS1 and CS2 Questions. In Proceedings of the Working Group Reports of the Conference on Innovation and Technology in Computer Science Education (ItiCSE)

2013
[38]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Gen- eration of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the Conference on International Computing Education Research (ICER)

2022
[39]

Smith, Paul Denny, and Max Fowler

David H. Smith, Paul Denny, and Max Fowler. 2024. Prompting for Compre- hension: Exploring the Intersection of Explain in Plain English Questions and Prompt Writing. In Proceedings of the Conference on Learning @ Scale (L@S)

2024
[40]

Chi, Quoc V

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models. In Proceedings of the Annual Conference on Neural Information Processing Systems...

2022
[41]

Jacqueline Whalley, Amber Settle, and Andrew Luxton-Reilly. 2021. Novice Reflections on Debugging. InProceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2021
[2011]

In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

CodeWrite: Supporting Student-Driven Practice of Java. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)
[2019]

In Proceedings of the Australasian Computing Education Conference (ACE)

Towards a Framework for Teaching Debugging. In Proceedings of the Australasian Computing Education Conference (ACE)

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla

Umair Z. Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla. 2020. Synthesizing Tasks for Block-based Programming. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)

work page 2020

[2] [2]

Georg Brandl, Matthäus Chajdas, and Jean Abou-Samra. 2006. Pygments. https: //pygments.org/

work page 2006

[3] [3]

Serena Caraco, Nelson Lojo, Michael Verdicchio, and Armando Fox. 2024. Gen- erating Multi-Part Autogradable Faded Parsons Problems From Code-Writing Exercises. In Proceedings of the Technical Symposium on Computer Science Educa- tion (SIGCSE)

work page 2024

[4] [4]

Mei-Wen Chen, Cheng-Chih Wu, and Yu-Tzu Lin. 2013. Novices’ Debugging Behaviors in VB Programming. In Proceedings of the Learning and Teaching in Computing and Engineering (LaTiCE)

work page 2013

[5] [5]

William G Cochran. 1952. The 𝜒2 Test of Goodness of Fit. The Annals of Mathematical Statistics (1952)

work page 1952

[6] [6]

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement (1960)

work page 1960

[7] [7]

Heffernan, Tanja Käser, Steven Moore, Anna N

Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla. 2024. Generative AI for Education (GAIED): Advances, Opportunities, and Challenges. CoRR abs/2402.01580 (2024)

arXiv 2024

[8] [8]

Becker, and Brent N

Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

work page 2024