{"id":"d2a15f81-053b-4b1a-bae8-99da539b0b20","arxiv_id":"2411.09261","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-4-generated test suites for 26 CS1 problems identified 92.8% of valid student solutions and caught invalid solutions more often than instructor suites did.","lead":"This paper asks whether GPT-4 can write the hidden test cases used to grade introductory programming assignments. The answer matters because writing good test cases is one of the main time costs of automated grading, and the AI-generated tests caught most valid solutions and found extra bugs in most of the 26 problems tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pipeline's crash-based test validity filter is unsound: non-crashing invalid tests in problems 2 and 16 caused 1,224 valid solutions to fail, so the reported 92.8% recall does not reflect the LLM's ability to grade valid tests.","rationale":"The reader identified the crash-based validity filter as the weakest assumption, and the paper's own examples (problems 2 and 16) demonstrate it is false. This is the most load-bearing issue because the RQ1 recall metric is directly computed from solutions that fail due to invalid tests; without a valid-test guarantee, the headline numbers do not cleanly support the central claim. The paper is transparent about these cases, and the overall conclusion that LLM-generated suites are usable with instructor review remains plausible, so a conditional verdict is appropriate. The proposed test would isolate whether the flaw is confined to the filter or reflects a deeper inconsistency in the problem summaries. No change to the reader's conditional verdict is needed.","tokens_in":42822,"tokens_out":10897,"duration_ms":108500,"concrete_test":"Re-run the LLM test-generation pipeline for all 26 problems with an additional validation step: after generation, check every test (edge-case and random) against the 'Inputs' section of the detailed problem statement produced by Prompt 1, and discard any test whose inputs violate those constraints. Then recompute Tables 8-10 and the per-problem false-negative counts. If the recall for problems 2 and 16 rises to ~100% and the aggregate recall exceeds 98%, the crash-only filter is confirmed as the cause of the reported false failures; if invalid non-crashing tests still pass the constraint check, the problem statement summarization itself is unreliable and manual review of tests is necessary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 accepts a generated test unless it crashes the reference solution; there is no independent check that test inputs satisfy the problem's stated constraints. The paper's own results refute this premise: Problem 2's LLM suite includes a test with source and destination in the same cell, and Problem 16's suite has tests with left > right and negative indices. These invalid tests caused 363 and 861 valid solutions respectively to be graded invalid (Table 4/Table 6), contributing 1,224 of the 1,234 LLM false negatives in Table 8. Thus the aggregate 92.8% recall in Table 10 mixes the LLM's test-generation failures with its grading behavior on valid tests; a pipeline that filtered invalid tests would show higher recall, and one where these failures are more frequent would show much lower recall. The central claim that 'LLM-generated test suites can correctly identify most valid solutions' is therefore not robustly established: it depends on the unsupported assumption that non-crashing generated tests are valid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether GPT-4 can generate autograder test suites for CS1 programming problems when given the problem statement and a reference solution. Using 26 problems and 33,749 student submissions from an introductory C course, the authors compare LLM-generated suites with instructor-written suites on three research questions: whether the LLM suites correctly identify valid solutions, how comprehensive they are relative to the instructor suites, and what problem-statement ambiguities they expose. The aggregate results reported in Table 10 are that LLM suites achieve 92.8% recall and 99.8% precision on valid/invalid classifications, versus 100% recall and 93.1% precision for the instructor suites. The paper also reports a cost of about one minute and ten US cents per problem and argues that LLM-generated suites can be useful for autograding and for improving problem statements.","tokens_in":42887,"tokens_out":5710,"duration_ms":63623,"significance":"If the results hold, the paper makes a useful practical contribution: it demonstrates on a large, real student dataset that LLM-generated test suites can be competitive with instructor-written suites, and it is unusually transparent in reporting the exact problems where the LLM failed, including invalid tests in problems 2 and 16 and a test-structure issue in problem 10. The per-problem breakdowns in Tables 4 and 6, the explicit discussion of ambiguity-driven mismatches, and the named failure categories are strengths that will help future work reproduce and improve the pipeline. The main weakness, discussed below, is that the headline recall metric is contaminated by tests that the paper itself identifies as invalid; this needs to be corrected before the central claim can be taken at face value.","major_comments":[{"comment":"The test-validity filter used in the pipeline rejects only tests that crash the reference solution, yet the paper's own Section 5 identifies four non-crashing invalid tests in problems 2 and 16 (source and destination in the same cell; left greater than right; negative index). These invalid tests account for 1,224 of the 1,234 solutions that the LLM suite marks invalid in Table 8 (363 in problem 2 plus 861 in problem 16). Consequently, the headline recall of 92.8% in Table 10 does not measure how often the LLM grades valid solutions correctly on valid tests; it conflates test-generation validity failures with grading behavior. Please re-analyze RQ1 and RQ2 after discarding invalid tests, or equivalently report per-problem recall on valid tests only, and discuss what the 1,224 false negatives imply for the claim that LLM-generated suites correctly identify most valid solutions when used without additional filtering.","section":"Section 3.2 and Table 10"},{"comment":"The ground-truth labels for mismatching solutions come from an unspecified manual review. Since 3,568 mismatches and 1,051 'Other' solutions are manually classified, the confusion matrices in Tables 8-10 are only as trustworthy as that review. The paper should report who performed the review, whether the reviewers were blind to the source of the grades, how many raters were involved, and how disagreements were resolved. Without this information, readers cannot assess the reliability of the 15,907 versus 17,141 split between valid and invalid labels that drives all of the reported metrics.","section":"Section 4.2, Table 5"},{"comment":"The 1,051 'Other Mismatches' are excluded from RQ1 and RQ2 on the grounds that they are caused by undefined behavior or ambiguities rather than by either suite. This exclusion is defensible for attribution, but the statement that these cases 'cannot be considered a mistake on either side' shifts the definition of correctness away from the actual problem statements: a test that triggers undefined behavior on a valid solution, or that relies on an ambiguous assumption, is arguably an invalid test for autograding purposes. Please provide a sensitivity analysis that includes these solutions in the aggregate metrics, or justify why excluding them cannot change the ranking of the two suites.","section":"Section 4.1.3, Table 6"}],"minor_comments":[{"comment":"The text says 'The LLM is instructed to print the values of any pointers that are passed to the function as can be seen in Table 7 in point 3 in the system prompt'; Table 7 is a results table, and the relevant instruction appears in Listing C7. Please fix the cross-reference.","section":"Section 5, RQ1 discussion"},{"comment":"The references to the Ruby application, the Python runner, and the system prompts do not include URLs or repository identifiers, which makes the claimed artifacts difficult to locate and verify. Please add persistent links or data archives.","section":"References [2], [3], [4]"},{"comment":"The paper notes that LLM outputs are non-deterministic and that a temperature of 0.2 was used, but it does not report the date of the API calls or provide the per-problem random seed values. Including this information would improve reproducibility.","section":"Section 5.2"},{"comment":"Problems 14 and 21 are said to be 'in red' in Table 1, but the coloring is not visible in a monochrome rendering; consider using a symbol or footnote to mark excluded problems.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a computing-education venue and has a solid empirical core. The main risk is that the aggregate metrics in Table 10 are presented as evidence for the abstract's claim even though 1,224 of the 1,234 false negatives come from tests that the manuscript itself calls invalid. This is fixable by re-analysis and by making the manual-review protocol explicit. I would be willing to look at a revised version that addresses the three major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful paper. It is the first evaluation I know of that runs LLM-generated test suites for CS1 problems against a large corpus of real student attempts (33,749). The authors compare against instructor suites and provide a taxonomy of mismatches. The main result—LLM suites identify most valid solutions and catch more invalid ones than the instructor suites in a majority of problems—holds for 23 of the 26 problems. For those problems, the LLM had no false negatives; across all problems it had 99.8% precision and a 0.2% false-positive rate. That is real, useful evidence that instructors can use GPT-4 to draft autograders, as long as they review the tests.\n\nThe soft spots are real but the paper mostly owns them. The biggest one is the crash-based validity filter. Section 3.2 only rejects tests that crash the reference solution. Non-crashing invalid tests in problems 2 and 16 survived and caused 1,224 valid solutions to be graded invalid. The paper reports this transparently in Section 4.1.1 and the discussion, but the headline aggregate 92.8% recall in Table 10 mixes these test-generation failures with the grading behavior. The stress-test note says the central claim is therefore not robustly established. I think that is too strong: the claim is about the end-to-end pipeline, and the paper shows exactly where it fails. The per-problem tables let any reader separate the parts. What is not robust is the abstract's 'at least as comprehensive' without the caveat 'after the problem-specific failures are excluded from the averages.'\n\nThe second concern is the exclusion of 'Other Mismatches,' which removes over 1,000 solutions, 757 of them from problem 19. The authors say these are due to undefined behavior or ambiguity, but no static analysis or sanitizer was used to confirm. Some of those could be instructor-suite problems, and excluding them before computing metrics is a judgment call. It does not change the qualitative picture, but it should be noted as a threat to the numbers.\n\nThe repeated-attempt counting is minor: multiple submissions by the same student are treated as independent, which overstates the effective sample size. Again, it does not move the conclusion.\n\nBottom line: this is a serious empirical paper with a real dataset, transparent failure analysis, and a practical contribution. It deserves peer review. The revision should soften the abstract wording and either fix the validity filter or report a secondary recall after filtering invalid tests. I would bring it to reading group and likely cite it.","headline":"A transparent empirical study that mostly supports its claim that GPT-4-generated autograder suites are usable with instructor review; the headline metric hides two pipeline failures, but the per-problem detail makes the paper honest enough for peer review.","tokens_in":43540,"tokens_out":2398,"would_cite":true,"duration_ms":23857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4 can generate autograder test suites that match or beat instructor-written ones for most CS1 problems.","keywords":["LLM","large language model","autograding","test suite generation","CS1","GPT-4","computing education","introductory programming"],"falsifier":"Add a constraint validator that rejects tests violating the stated limits, such as source equals destination, left greater than right, or negative indices, and rerun the 26 suites. If the 1,224 false failures in problems 2 and 16 vanish while the 23 missed invalid solutions remain at 23, the paper's central claim is confirmed; if new invalid solutions appear or false failures persist, the claim needs revision.","tokens_in":42472,"feed_emoji":"🤖","tokens_out":8776,"duration_ms":91961,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can take over the most tedious part of setting up an autograder: writing the test suite. The authors give GPT-4 each problem's statement and reference solution, collect edge-case and random tests, run more than 25,000 real student submissions against them, and compare the verdicts with an instructor-written suite. On 26 CS1 problems they report that the LLM suites pass 92.8% of valid solutions, versus 100% for the instructor suites, but flag invalid solutions as valid only 0.2% of the time, versus 14.6% for instructors. They also show that four constraint-violating tests generated by the LLM caused 1,224 valid solutions to be marked invalid, so the practical claim is that the tests are good enough to use with review, not to use blindly. If this holds, instructors can adopt autograding at a fraction of the current test-writing cost and catch more flawed solutions than before.","feed_headline":"GPT-4 writes autograder tests that rival instructor suites","feed_subtitle":"In a 26-problem study, LLM suites passed 92.8% of valid solutions and caught invalid code instructors missed.","key_machinery":"The load-bearing mechanism is a two-stage prompt pipeline plus a crash filter. Stage one asks GPT-4 to rewrite any problem statement into a fixed JSON summary with sections for scenario, inputs, outputs, example, and limits, using a reflection-style pass to fix inconsistencies. Stage two gives that summary plus the reference solution to the model and asks it to enumerate edge cases, reflect on them, then emit a Python script for whole programs or a C test template for functions that produces one edge-case test per case plus 100 random tests. The pipeline keeps a test only if running the reference solution does not crash it, and it defines expected outputs by executing the reference solution with a fixed random seed for randomized tests. That filter is what turns raw LLM output into an executable autograder suite, and it is also the point where constraint-invalid tests slip through.","core_discovery":"On the paper's own terms, the discovery is that a zero-shot, two-prompt GPT-4 workflow reproduces the grading behavior of instructor-authored test suites. The model first converts the problem statement into a fixed structured summary, then generates edge-case tests plus 100 randomized tests from that summary and the reference solution; any generated test that crashes the reference solution is rejected, and the reference solution is used to produce expected outputs. Run on 26 CS1 problems and 26,787 compiling submissions, after excluding mismatch cases the authors attribute to undefined behavior or problem ambiguity, this yields a suite that identifies 92.8% of valid solutions and misclassifies invalid solutions as valid only 0.2% of the time, compared with 14.6% for the instructor suites. The main failure mode is not missed bugs but a small number of tests that violate the problem statement's own constraints, and the paper traces those to prompting details and argues they are fixable.","pith_inferences":["My inference: a constraint validator added after generation would fix most of the reported false failures, because the failures concentrate in a tiny number of tests that violate explicit input constraints; a script checking each test's inputs against the structured summary's limits section should be the first improvement tested.","My inference: the paper's validity labels come from comparing with a reference output, so on problems where the specification is ambiguous an LLM-generated suite can only encode one interpretation; the study's decision to exclude such mismatches masks how often that happens in practice.","My inference: because the per-problem cost is fixed and the failures are clustered, the approach is likely to transfer to other CS1 languages, but this needs testing; the authors note that C's null-terminator was the source of an invalid range test, so language-specific pitfalls matter.","A testable extension: generate suites for problems with machine-checkable constraints and compare recall before and after a constraint-satisfaction filter; if the filter removes the false failures without adding false passes, the central claim is robust to the crash-check assumption."],"forward_implications":["An instructor could generate a first-pass test suite for a CS1 problem in about one minute and for about ten US cents, which lowers the cost of adopting autograders that give instant feedback.","For most problems the LLM suite catches invalid student solutions that the instructor suite missed, including hard-coded special cases and wrong data types, so autograded correctness marks become stricter.","The main risk of using these suites is false failure: a handful of invalid generated tests can fail large numbers of valid solutions, so instructor review of the generated tests is needed before deployment.","The same generation pass can be used as a problem-statement review tool: tests whose inputs violate stated constraints expose ambiguities or gaps, prompting instructors to clarify limits and edge-case behavior."],"supporting_citations":[{"why":"This citation supplies the CodeRunner autograder whose exported submissions for the 26 course problems form the dataset for the evaluation.","marker":"[25]"},{"why":"This citation supplies the Reflexion strategy that the prompt design uses when asking the LLM to reflect on and correct its edge cases before finalizing test code.","marker":"[42]"},{"why":"This citation is adduced as benchmark evidence that GPT-4 is capable at code generation, motivating the choice of the model that the entire study uses.","marker":"[19]"},{"why":"This citation identifies gpt-4-0125-preview as the specific OpenAI model used, so it pins down which model the study's numbers describe.","marker":"[30]"}],"fun_headline_variants":["LLM autograders pass 92.8% of valid code, beat instructor catches","GPT-4 test suites rival instructor grading in CS1 study","LLM-generated tests catch more invalid code than human suites","Study: GPT-4 autograder tests slash false passes from 14.6% to 0.2%","LLM suites find valid code 92.8% of the time, expose ambiguities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper accepts a generated test as valid unless it crashes the reference solution; it never independently checks that test inputs satisfy the problem statement, and its own invalid tests in problems 2 and 16 show that this assumption fails and causes valid solutions to be marked invalid.","fun_headline_variants_meta":{"raw":{"variants":["LLM autograders pass 92.8% of valid code, beat instructor catches","GPT-4 test suites rival instructor grading in CS1 study","LLM-generated tests catch more invalid code than human suites","Study: GPT-4 autograder tests slash false passes from 14.6% to 0.2%","LLM suites find valid code 92.8% of the time, expose ambiguities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2700,"prompt_tokens":990,"completion_tokens":1710,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":1601}},"tokens_in":606,"tokens_out":1710,"duration_ms":14629,"temperature":1.0,"reasoning_tokens":1601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:50:30.172498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Add a constraint validator that rejects tests violating the stated limits, such as source equals destination, left greater than right, or negative indices, and rerun the 26 suites. If the 1,224 false failures in problems 2 and 16 vanish while the 23 missed invalid solutions remain at 23, the paper's central claim is confirmed; if new invalid solutions appear or false failures persist, the claim needs revision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This citation supplies the CodeRunner autograder whose exported submissions for the 26 course problems form the dataset for the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This citation supplies the Reflexion strategy that the prompt design uses when asking the LLM to reflect on and correct its edge cases before finalizing test code."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This citation is adduced as benchmark evidence that GPT-4 is capable at code generation, motivating the choice of the model that the entire study uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This citation identifies gpt-4-0125-preview as the specific OpenAI model used, so it pins down which model the study's numbers describe."}],"review_version":1}