Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming

Ibrahim Albluwi; Paul Denny; Umar Alkafaween

REVIEW 3 major objections 4 minor 58 references

Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4 can generate autograder test suites that match or beat instructor-written ones for most CS1 problems.

desk verdict A transparent empirical study that mostly supports its claim that GPT-4-generated autograder suites are usable with instructor review; the headline metric hides two pipeline failures, but the per-problem detail makes the paper honest enough for peer review. read the letter →

arxiv 2411.09261 v2 pith:64X6GEWE submitted 2024-11-14 cs.CY cs.AI

Umar Alkafaween , Ibrahim Albluwi , Paul Denny This is my paper · ORCID

classification cs.CYcs.AI

keywords LLMlargelanguagemodelautogradingtestsuitegenerationCS1GPT-4computingeducationintroductoryprogramming

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model can take over the most tedious part of setting up an autograder: writing the test suite. The authors give GPT-4 each problem's statement and reference solution, collect edge-case and random tests, run more than 25,000 real student submissions against them, and compare the verdicts with an instructor-written suite. On 26 CS1 problems they report that the LLM suites pass 92.8% of valid solutions, versus 100% for the instructor suites, but flag invalid solutions as valid only 0.2% of the time, versus 14.6% for instructors. They also show that four constraint-violating tests generated by the LLM caused 1,224 valid solutions to be marked invalid, so the practical claim is that the tests are good enough to use with review, not to use blindly. If this holds, instructors can adopt autograding at a fraction of the current test-writing cost and catch more flawed solutions than before.

What carries the argument

The load-bearing mechanism is a two-stage prompt pipeline plus a crash filter. Stage one asks GPT-4 to rewrite any problem statement into a fixed JSON summary with sections for scenario, inputs, outputs, example, and limits, using a reflection-style pass to fix inconsistencies. Stage two gives that summary plus the reference solution to the model and asks it to enumerate edge cases, reflect on them, then emit a Python script for whole programs or a C test template for functions that produces one edge-case test per case plus 100 random tests. The pipeline keeps a test only if running the reference solution does not crash it, and it defines expected outputs by executing the reference solution with a fixed random seed for randomized tests. That filter is what turns raw LLM output into an executable autograder suite, and it is also the point where constraint-invalid tests slip through.

What would settle it

Add a constraint validator that rejects tests violating the stated limits, such as source equals destination, left greater than right, or negative indices, and rerun the 26 suites. If the 1,224 false failures in problems 2 and 16 vanish while the 23 missed invalid solutions remain at 23, the paper's central claim is confirmed; if new invalid solutions appear or false failures persist, the claim needs revision.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a zero-shot, two-prompt GPT-4 workflow reproduces the grading behavior of instructor-authored test suites. The model first converts the problem statement into a fixed structured summary, then generates edge-case tests plus 100 randomized tests from that summary and the reference solution; any generated test that crashes the reference solution is rejected, and the reference solution is used to produce expected outputs. Run on 26 CS1 problems and 26,787 compiling submissions, after excluding mismatch cases the authors attribute to undefined behavior or problem ambiguity, this yields a suite that identifies 92.8% of valid solutions and misclassifies invalid solutions as valid only 0.2% of the time, compared with 14.6% for the instructor suites. The main failure mode is not missed bugs but a small number of tests that violate the problem statement's own constraints, and the paper traces those to prompting details and argues they are fixable.

Load-bearing premise

The paper accepts a generated test as valid unless it crashes the reference solution; it never independently checks that test inputs satisfy the problem statement, and its own invalid tests in problems 2 and 16 show that this assumption fails and causes valid solutions to be marked invalid.

Editorial extensions

If this is right

An instructor could generate a first-pass test suite for a CS1 problem in about one minute and for about ten US cents, which lowers the cost of adopting autograders that give instant feedback.
For most problems the LLM suite catches invalid student solutions that the instructor suite missed, including hard-coded special cases and wrong data types, so autograded correctness marks become stricter.
The main risk of using these suites is false failure: a handful of invalid generated tests can fail large numbers of valid solutions, so instructor review of the generated tests is needed before deployment.
The same generation pass can be used as a problem-statement review tool: tests whose inputs violate stated constraints expose ambiguities or gaps, prompting instructors to clarify limits and edge-case behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: a constraint validator added after generation would fix most of the reported false failures, because the failures concentrate in a tiny number of tests that violate explicit input constraints; a script checking each test's inputs against the structured summary's limits section should be the first improvement tested.
My inference: the paper's validity labels come from comparing with a reference output, so on problems where the specification is ambiguous an LLM-generated suite can only encode one interpretation; the study's decision to exclude such mismatches masks how often that happens in practice.
My inference: because the per-problem cost is fixed and the failures are clustered, the approach is likely to transfer to other CS1 languages, but this needs testing; the authors note that C's null-terminator was the source of an invalid range test, so language-specific pitfalls matter.
A testable extension: generate suites for problems with machine-checkable constraints and compare recall before and after a constraint-satisfaction filter; if the filter removes the false failures without adding false passes, the central claim is robust to the crash-check assumption.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful paper. It is the first evaluation I know of that runs LLM-generated test suites for CS1 problems against a large corpus of real student attempts (33,749). The authors compare against instructor suites and provide a taxonomy of mismatches. The main result—LLM suites identify most valid solutions and catch more invalid ones than the instructor suites in a majority of problems—holds for 23 of the 26 problems. For those problems, the LLM had no false negatives; across all problems it had 99.8% precision and a 0.2% false-positive rate. That is real, useful evidence that instructors can use GPT-4 to draft autograders, as long as they review the tests.

The soft spots are real but the paper mostly owns them. The biggest one is the crash-based validity filter. Section 3.2 only rejects tests that crash the reference solution. Non-crashing invalid tests in problems 2 and 16 survived and caused 1,224 valid solutions to be graded invalid. The paper reports this transparently in Section 4.1.1 and the discussion, but the headline aggregate 92.8% recall in Table 10 mixes these test-generation failures with the grading behavior. The stress-test note says the central claim is therefore not robustly established. I think that is too strong: the claim is about the end-to-end pipeline, and the paper shows exactly where it fails. The per-problem tables let any reader separate the parts. What is not robust is the abstract's 'at least as comprehensive' without the caveat 'after the problem-specific failures are excluded from the averages.'

The second concern is the exclusion of 'Other Mismatches,' which removes over 1,000 solutions, 757 of them from problem 19. The authors say these are due to undefined behavior or ambiguity, but no static analysis or sanitizer was used to confirm. Some of those could be instructor-suite problems, and excluding them before computing metrics is a judgment call. It does not change the qualitative picture, but it should be noted as a threat to the numbers.

The repeated-attempt counting is minor: multiple submissions by the same student are treated as independent, which overstates the effective sample size. Again, it does not move the conclusion.

Bottom line: this is a serious empirical paper with a real dataset, transparent failure analysis, and a practical contribution. It deserves peer review. The revision should soften the abstract wording and either fix the validity filter or report a secondary recall after filtering invalid tests. I would bring it to reading group and likely cite it.

Referee Report

3 major / 4 minor

Summary. The paper evaluates whether GPT-4 can generate autograder test suites for CS1 programming problems when given the problem statement and a reference solution. Using 26 problems and 33,749 student submissions from an introductory C course, the authors compare LLM-generated suites with instructor-written suites on three research questions: whether the LLM suites correctly identify valid solutions, how comprehensive they are relative to the instructor suites, and what problem-statement ambiguities they expose. The aggregate results reported in Table 10 are that LLM suites achieve 92.8% recall and 99.8% precision on valid/invalid classifications, versus 100% recall and 93.1% precision for the instructor suites. The paper also reports a cost of about one minute and ten US cents per problem and argues that LLM-generated suites can be useful for autograding and for improving problem statements.

Significance. If the results hold, the paper makes a useful practical contribution: it demonstrates on a large, real student dataset that LLM-generated test suites can be competitive with instructor-written suites, and it is unusually transparent in reporting the exact problems where the LLM failed, including invalid tests in problems 2 and 16 and a test-structure issue in problem 10. The per-problem breakdowns in Tables 4 and 6, the explicit discussion of ambiguity-driven mismatches, and the named failure categories are strengths that will help future work reproduce and improve the pipeline. The main weakness, discussed below, is that the headline recall metric is contaminated by tests that the paper itself identifies as invalid; this needs to be corrected before the central claim can be taken at face value.

major comments (3)

[Section 3.2 and Table 10] The test-validity filter used in the pipeline rejects only tests that crash the reference solution, yet the paper's own Section 5 identifies four non-crashing invalid tests in problems 2 and 16 (source and destination in the same cell; left greater than right; negative index). These invalid tests account for 1,224 of the 1,234 solutions that the LLM suite marks invalid in Table 8 (363 in problem 2 plus 861 in problem 16). Consequently, the headline recall of 92.8% in Table 10 does not measure how often the LLM grades valid solutions correctly on valid tests; it conflates test-generation validity failures with grading behavior. Please re-analyze RQ1 and RQ2 after discarding invalid tests, or equivalently report per-problem recall on valid tests only, and discuss what the 1,224 false negatives imply for the claim that LLM-generated suites correctly identify most valid solutions when used without additional filtering.
[Section 4.2, Table 5] The ground-truth labels for mismatching solutions come from an unspecified manual review. Since 3,568 mismatches and 1,051 'Other' solutions are manually classified, the confusion matrices in Tables 8-10 are only as trustworthy as that review. The paper should report who performed the review, whether the reviewers were blind to the source of the grades, how many raters were involved, and how disagreements were resolved. Without this information, readers cannot assess the reliability of the 15,907 versus 17,141 split between valid and invalid labels that drives all of the reported metrics.
[Section 4.1.3, Table 6] The 1,051 'Other Mismatches' are excluded from RQ1 and RQ2 on the grounds that they are caused by undefined behavior or ambiguities rather than by either suite. This exclusion is defensible for attribution, but the statement that these cases 'cannot be considered a mistake on either side' shifts the definition of correctness away from the actual problem statements: a test that triggers undefined behavior on a valid solution, or that relies on an ambiguous assumption, is arguably an invalid test for autograding purposes. Please provide a sensitivity analysis that includes these solutions in the aggregate metrics, or justify why excluding them cannot change the ranking of the two suites.

minor comments (4)

[Section 5, RQ1 discussion] The text says 'The LLM is instructed to print the values of any pointers that are passed to the function as can be seen in Table 7 in point 3 in the system prompt'; Table 7 is a results table, and the relevant instruction appears in Listing C7. Please fix the cross-reference.
[References [2], [3], [4]] The references to the Ruby application, the Python runner, and the system prompts do not include URLs or repository identifiers, which makes the claimed artifacts difficult to locate and verify. Please add persistent links or data archives.
[Section 5.2] The paper notes that LLM outputs are non-deterministic and that a temperature of 0.2 was used, but it does not report the date of the API calls or provide the per-problem random seed values. Including this information would improve reproducibility.
[Table 1] Problems 14 and 21 are said to be 'in red' in Table 1, but the coloring is not visible in a monochrome rendering; consider using a symbol or footnote to mark excluded problems.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an empirical benchmark against external student submissions with manual adjudication of mismatches, and no fitted parameters or self-citation chains are load-bearing.

full rationale

The paper's central claim is that LLM-generated test suites can correctly identify most valid solutions and are often at least as comprehensive as instructor suites. This is established empirically: 33,749 student submissions from an external course are run against both instructor and LLM test suites, and all 3,568 mismatching grades are manually reviewed to determine solution validity. The validity label is therefore not defined by the LLM's own outputs; it comes from human adjudication of each disputed solution, supplemented by the instructor suite only for initially matching grades. No constants are fitted to the student data, no prediction is constructed from the evaluated quantity, and no prior result by the same authors is invoked to force the conclusion. The use of the reference solution to generate expected outputs is a standard oracle practice and is not circular: it is an input to the test-generation pipeline, but the paper's evaluation target is the resulting test suite's grading behavior on independent student code, not the reference solution's behavior. The acknowledged invalid tests in problems 2 and 16 are correctly reported as LLM errors and manually excluded from the validity determination, so they weaken the generality of the headline result but do not make the derivation circular. The paper is self-contained against external benchmarks, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced. The load-bearing assumptions concern the validity of the reference solution, the crash-based validity filter for generated tests, the construction of ground-truth validity labels, and the independence of student attempts. The free parameters are prompt and configuration choices rather than fitted constants.

free parameters (3)

LLM sampling temperature = 0.2
Chosen to reduce randomness in GPT-4 outputs. No sensitivity analysis is reported, and Section 5.2 admits LLM results are not deterministic.
Random tests per problem = 100
The prompt requests 100 random tests when possible. Coverage and the resulting metrics depend on this count, and no ablation is reported.
Per-problem random seed = problem-specific integer
The function-implementation test harness calls srand with a fixed per-problem seed to stabilize outputs. The choice of seed is arbitrary and could affect which random tests are generated.

assumptions (4)

domain assumption The reference solution correctly implements the problem statement, and its outputs define expected outputs for all generated tests.
Used throughout Sections 3.2 and 3.3. If the reference solution embeds assumptions or bugs, the expected outputs encode them.
domain assumption A solution's true validity can be determined by agreement of the two test suites plus manual review of only mismatching solutions.
Section 4.2 and Table 5. Solutions that both suites pass or both suites fail are not manually inspected, so hidden bugs missed by both are labeled valid.
ad hoc to paper A test is valid if it does not crash the reference solution.
Section 3.2 rejects only crashing tests. Section 5 shows invalid non-crashing tests for problems 2 and 16, so this assumption is violated by the paper's own data.
domain assumption Repeated attempts by the same student can be counted as independent observations.
Section 3.1.2 uses 33,749 attempts, and the confusion matrices in Tables 8 and 9 treat all compiling attempts as separate samples even though many come from the same students.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming." pith.science (2026). https://pith.science/paper/64X6GEWE

@misc{pith2026241109261,
  author       = {Pith},
  title        = {Pith review of: Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64X6GEWE}},
  note         = {Machine review of arXiv:2411.09261}
}

read the original abstract

Automatically graded programming assignments provide instant feedback to students and significantly reduce manual grading time for instructors. However, creating comprehensive suites of test cases for programming problems within automatic graders can be time-consuming and complex. The effort needed to define test suites may deter some instructors from creating additional problems or lead to inadequate test coverage, potentially resulting in misleading feedback on student solutions. Such limitations may reduce student access to the well-documented benefits of timely feedback when learning programming. In this work, we evaluate the effectiveness of using Large Language Models (LLMs), as part of a larger workflow, to automatically generate test suites for CS1-level programming problems. Each problem's statement and reference solution are provided to GPT-4 to produce a test suite that can be used by an autograder. We evaluate our proposed approach using a sample of 26 problems, and more than 25,000 attempted solutions to those problems, submitted by students in an introductory programming course. We compare the performance of the LLM-generated test suites against the instructor-created test suites for each problem. Our findings reveal that LLM-generated test suites can correctly identify most valid solutions, and for most problems are at least as comprehensive as the instructor test suites. Additionally, the LLM-generated test suites exposed ambiguities in some problem statements, underscoring their potential to improve both autograding and instructional design.

Figures

Figures reproduced from arXiv: 2411.09261 by the authors.

**Figure 2.** (a) Chaining LLM Prompts and (b) combining results [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 53 canonical work pages

[1]

Nimisha Agarwal and Amey Karkare. 2022. LEGenT: Localiz ing errors and gen- erating testcases for CS1. In Proceedings of the Ninth ACM Conference on Learn- ing@ Scale. 102–112

work page 2022
[2]

Umar Alkafaween. 2024. Ruby code to run Python generated LLM unit tests

work page 2024
[3]

Umar Alkafaween. 2024. A small app that talks to OpenAI to generate tests for CS1 programming problems

work page 2024
[4]

Umar Alkafaween. 2024. System prompts to generate unit tests for Python

work page 2024
[5]

Imen Azaiz, Oliver Deckarm, and Sven Strickroth. 2023. A I-enhanced Auto- correction of Programming Exercises: How Eﬀective is GPT-3 .5? arXiv preprint arXiv:2311.10737 (2023)

arXiv 2023
[6]

Imen Azaiz, Natalie Kiesler, and Sven Strickroth. 2024. Feedback-Generation for Programming Exercises With GPT-4. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 . 31–37

work page 2024
[7]

Xue Bai, Ade Ola, Somasheker Akkaladevi, and Yingjin Cui . 2016. ENHANCING THE LEARNING PROCESS IN PROGRAMMING COURSES THROUGH AN AU- TOMATED FEEDBACK AND ASSIGNMENT MANAGEMENT SYSTEM. Issues in Information Systems 17, 3 (2016)

work page 2016
[8]

Becker, Paul Denny, James Finnie-Ansley, Andre w Luxton-Reilly, James Prather, and Eddie Antonio Santos

Brett A. Becker, Paul Denny, James Finnie-Ansley, Andre w Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Har d - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1(Toronto ON, Canada)(SIGCSE 2023). Ass...

work page doi:10.1145/3545 2023

Show all 58 references

[9]

Douglas Bengtsson and Axel Kaliﬀ. 2023. Assessment Accu racy of a Large Lan- guage Model on Programming Assignments

2023
[10]

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023)

2023 arXiv
[11]

Becker, and Brent N

Paul Denny, Juho Leinonen, James Prather, Andrew Luxto n-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Promp t Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Educa...

2024
[12]

Paul Denny, Stephen MacNeil, Jaromir Savelka, Leo Port er, and Andrew Luxton-Reilly. 2024. Desirable Characteristics for AI Tea ching Assis- tants in Programming Education. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (Milan, Italy...

2024
[13]

Paul Denny, James Prather, Brett A Becker, James Finnie -Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N Reeves, Eddie Antonio Santos, and Sami Sarsa. 2024. Computing education in the era of gener ative AI. Com- mun. ACM 67, 2 (2024), 56–67

2024
[14]

Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Q ayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Ara v Agarwal, et al
[15]

Piotr Duch and Tomasz Jaworski. 2018. Dante-Automated Assessments Tool for Students’ Programming Assignments. In 2018 11th International Conference on Human System Interaction (HSI) . IEEE, 162–168

2018
[16]

Tommy Färnqvist and Fredrik Heintz. 2016. Competition and feedback through automated assessment in a data structures and algorithms course. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Co mputer Science Education. 130–135

2016
[17]

Aldo Gordillo. 2019. Eﬀect of an instructor-centered t ool for automatic assess- ment of programming assignments on students’ perceptions a nd performance. Sustainability 11, 20 (2019), 5568

2019
[18]

Qiang Hao and Michail Tsikerdekis. 2019. How automated feedback is delivered matters: Formative feedback and knowledge transfer. In 2019 IEEE Frontiers in Education Conference (FIE). IEEE, 1–6

2019
[19]

Binyuan Hui. 2023. An awesome and curated list of best code-LLM for research

2023
[20]

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 201 8. A systematic litera- ture review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE) 19, 1 (2018), 1–43

2018
[21]

Hassan Khosravi, Paul Denny, Steven Moore, and John Sta mper. 2023. Learner- sourcing in the age of AI: Student, educator and machine part nerships for con- tent creation. Computers and Education: Artiﬁcial Intelligence 5 (2023), 100151

2023
[22]

Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa , Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Co de Explana- tions Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer...

2023 doi
[23]

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, P aul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Mod els to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Sym- posium on Computer Science Education V. 1 (Toronto ON,...

2023
[24]

Mark Liﬃton, Brad E Sheese, Jaromir Savelka, and Paul De nny. 2023. Codehelp: Using large language models with guardrails for scalable su pport in program- ming classes. In Proceedings of the 23rd Koli Calling International Conferen ce on Computing Education Research. 1–11

2023
[25]

Richard Lobb and Jenny Harlow. 2016. Coderunner: A tool for assessing com- puter programming skills. ACM Inroads 7, 1 (2016), 47–51

2016
[26]

Marcus Messer, Neil CC Brown, Michael Kölling, and Miao jing Shi. 2023. Ma- chine learning-based automated grading and feedback tools for programming: A meta-analysis. In Proceedings of the 2023 Conference on Innovation and Tech- nology in Computer Science Education V. 1 . 491–497

2023
[27]

Joydeep Mitra. 2023. Studying the impact of auto-grade rs giving immediate feedback in programming assignments. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 388–394

2023
[28]

Filippa Nilsson and Jonatan Tuvstedt. 2023. GPT-4 as an Automatic Grader: The accuracy of grades set by GPT-4 on introductory programming assignments

2023
[29]

OpenAI. 2023. A small app that talks to OpenAI to generate tests for CS1 program- ming problems

2023
[30]

OpenAI. 2024. About GPT4 and GPT4 Turbo

2024
[31]

José Carlos Paiva, José Paulo Leal, and Álvaro Figueira . 2022. Automated assess- ment in computer science education: A state-of-the-art rev iew. ACM Transac- tions on Computing Education (TOCE) 22, 3 (2022), 1–40

2022
[32]

Sagar Parihar, Ziyaan Dadachanji, Praveen Kumar Singh , Rajdeep Das, Amey Karkare, and Arnab Bhattacharya. 2017. Automatic grading a nd feedback us- ing program repair for introductory programming courses. In Proceedings of the 2017 ACM conference on innovation and technology ...

2017
[33]

Tung Phung, Victor-Alexandru Pădurean, José Cambrone ro, Sumit Gulwani, To- bias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares . 2023. Genera- tive ai for programming education: Benchmarking chatgpt, g pt-4, and human tutors. In Proceedings of the 2023 ACM Conference ...

2023
[34]

Vreda Pieterse. 2013. Automated Assessment of Program ming Assignments. CSERC 13 (2013), 4–5. Alkafaween et al

2013
[35]

James Prather, Brett A Becker, Michelle Craig, Paul Den ny, Dastyni Loksa, and Lauren Margulieux. 2020. What do we think we think we are doin g? Metacog- nition and self-regulation in programming. In Proceedings of the 2020 ACM con- ference on international computing education...

2020
[36]

Beck er, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Koh n, Andrew Luxton- Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N

James Prather, Paul Denny, Juho Leinonen, Brett A. Beck er, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Koh n, Andrew Luxton- Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. The Robots Are Here: ...

2023
[37]

James Prather, Raymond Pettit, Kayla McMurry, Alani Pe ters, John Homer, and Maxine Cohen. 2018. Metacognitive diﬃculties faced by novi ce programmers in automated assessment tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research . 41–50

2018
[38]

Dhananjai M Rao. 2019. Experiences with auto-grading i n a systems course. In 2019 IEEE Frontiers in Education Conference (FIE) . IEEE, 1–8

2019
[39]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic gen- eration of programming exercises and code explanations usi ng large language models. In Proceedings of the 2022 ACM Conference on International Comp uting Education Research-Volume 1. 27–43

2022
[40]

Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2 023. An empirical evaluation of using large language models for automated uni t test generation. IEEE Transactions on Software Engineering (2023)

2023
[41]

Mark Sherman, Sarita Bassil, Derrell Lipman, Nat Tuck, and Fred Martin. 2013. Impact of auto-grading on an introductory computing course . Journal of Com- puting Sciences in Colleges 28, 6 (2013), 69–75

2013
[42]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthi k Narasimhan, and Shunyu Yao. 2024. Reﬂexion: Language agents with verbal rei nforcement learn- ing. Advances in Neural Information Processing Systems 36 (2024)

2024
[43]

Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using large language models to generate junit tests: An empirical study . In Proceedings of the 28th International Conference on Eval...

2024
[44]

Ján Skalka and Martin Drlík. 2023. Development of autom atic source code evalu- ation tests using grey-box methods: A programming education case study. IEEE Access 11 (2023), 106772–106792

2023
[45]

Smith, Paul Denny, and Max Fowler

David H. Smith, Paul Denny, and Max Fowler. 2024. Prompt ing for Compre- hension: Exploring the Intersection of Explain in Plain Eng lish Questions and Prompt Writing. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (Atlanta, GA, USA) (L@S ’24). Association f...

2024 doi
[46]

Thomas Staubitz, Hauke Klement, Jan Renz, Ralf Teusner , and Christoph Meinel
[47]

Anne Venables and Liz Haywood. 2003. Programming stude nts NEED in- stant feedback!. In Proceedings of the ﬁfth Australasian conference on Computin g education-Volume 20. 267–272

2003
[48]

Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Su rvey, landscape, and vision. IEEE Transactions on Software Engineering (2024)

2024
[49]

Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Trans- parent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–22. Automating Autograding: Large...

2022
[52]

Use as many large containers as possible, to minimise the t otal number of containers
[53]

For example, let’s say that an order for 31 items is placed

Do not ship any containers which are not completely full This second rule means that any left over items, which could o nly be placed into a partially full container, will be scrapp ed. For example, let’s say that an order for 31 items is placed. Al so, assume that the capacit...
[54]

how many items can b e packed in a large container)

The capacity of a large container (i.e. how many items can b e packed in a large container)
[55]

how many items can b e packed in a small container)

The capacity of a small container (i.e. how many items can b e packed in a small container)
[56]

In addition, you must display the number of items that will be scrapped (be cause they can’t be placed into a full container)

The total number of items in the order Your program should calculate and display the total number o f containers, of each size, that will be needed to ship the ord er. In addition, you must display the number of items that will be scrapped (be cause they can’t be placed into a...
[57]

move the pellet horizontally (left or right) if necessary , until it lines up with the destination
[58]

1” (representing the work er) and a single “2

move the pellet vertically (up or down) if necessary, unti l it reaches the destination The route that you calculate should be indicated by setting a ll array elements on the route to the value 3. Note, as implied by the algorithm above, the pellet must move horizontally ﬁrst ...
[2015]

In2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)

Towards practical programming exercises and automat ed assessment in Massive Open Online Courses. In2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) . IEEE, 23–30
[2024]

In Proceedings of the 26th Australasian Computing Education Conference

A comparative study of AI-generated (GPT-4) and human -crafted MCQs in programming education. In Proceedings of the 26th Australasian Computing Education Conference. 114–123

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Nimisha Agarwal and Amey Karkare. 2022. LEGenT: Localiz ing errors and gen- erating testcases for CS1. In Proceedings of the Ninth ACM Conference on Learn- ing@ Scale. 102–112

work page 2022

[2] [2]

Umar Alkafaween. 2024. Ruby code to run Python generated LLM unit tests

work page 2024

[3] [3]

Umar Alkafaween. 2024. A small app that talks to OpenAI to generate tests for CS1 programming problems

work page 2024

[4] [4]

Umar Alkafaween. 2024. System prompts to generate unit tests for Python

work page 2024

[5] [5]

Imen Azaiz, Oliver Deckarm, and Sven Strickroth. 2023. A I-enhanced Auto- correction of Programming Exercises: How Eﬀective is GPT-3 .5? arXiv preprint arXiv:2311.10737 (2023)

arXiv 2023

[6] [6]

Imen Azaiz, Natalie Kiesler, and Sven Strickroth. 2024. Feedback-Generation for Programming Exercises With GPT-4. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 . 31–37

work page 2024

[7] [7]

Xue Bai, Ade Ola, Somasheker Akkaladevi, and Yingjin Cui . 2016. ENHANCING THE LEARNING PROCESS IN PROGRAMMING COURSES THROUGH AN AU- TOMATED FEEDBACK AND ASSIGNMENT MANAGEMENT SYSTEM. Issues in Information Systems 17, 3 (2016)

work page 2016

[8] [8]

Becker, Paul Denny, James Finnie-Ansley, Andre w Luxton-Reilly, James Prather, and Eddie Antonio Santos

Brett A. Becker, Paul Denny, James Finnie-Ansley, Andre w Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Har d - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1(Toronto ON, Canada)(SIGCSE 2023). Ass...

work page doi:10.1145/3545 2023