{"id":"8828427a-ab77-4d3a-8052-8bac0cfff379","arxiv_id":"2411.14303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-generated debugging exercises, validated by a test suite, matched instructor-written exercises in student success rates across three introductory C programming problems.","lead":"BugSpotter uses a large language model to generate buggy code from programming problem descriptions, then checks the bugs against a test suite. A classroom study with 741 students found that these generated debugging exercises produced similar student success rates to exercises written by an instructor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-significant chi-square tests do not establish comparability; the central claim needs equivalence testing or confidence intervals.","rationale":"The reader correctly identified pre-selection of exercises (Section 4.2) as a threat to external validity, and their rationale also mentioned the need for equivalence testing or confidence intervals. However, the weakest_assumption field emphasizes the curation gap rather than the more fundamental statistical inference error in Section 5.3: non-significance is not evidence of equivalence. That latter issue is more load-bearing because it undermines the central claim even for the curated set of exercises actually deployed. The expert evaluation (RQ1) and difficulty-alignment results (RQ2) provide credible, useful evidence for the tool's ability to generate diverse and appropriately difficult exercises, and the paper is transparent about several limitations. No internal inconsistency or fabrication is present; the problem is a correctness risk in the statistical interpretation of RQ3. Since the appropriate remedy is a reanalysis or additional data rather than a rejection of the entire contribution, the existing CONDITIONAL verdict remains appropriate. The proposed concrete check would settle whether the comparability claim can be maintained or must be substantially weakened.","tokens_in":12108,"tokens_out":3463,"duration_ms":35883,"concrete_test":"Reanalyze the existing RQ3 data with equivalence testing: compute 90% confidence intervals for the difference in success rates (LLM-generated minus instructor-created) for each problem and overall, and compare them against a pre-registered equivalence margin of ±10 percentage points (or another educationally justified bound). If the entire interval lies within the margin for all problems, the comparability claim is supported; if any interval crosses the margin, the current data do not establish equivalence and the claim should be weakened to 'no significant difference was detected.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in Section 5.3 and repeated in the abstract and conclusion, is that LLM-generated debugging exercises are 'comparable in difficulty' to instructor-created ones. The only quantitative support is a set of chi-square tests whose p-values all exceed 0.05, with the lowest at 0.065. This is an invalid equivalence argument: a non-significant p-value means only that the data do not provide strong evidence against the null hypothesis of equal success rates; it does not quantify support for equality. With roughly 741 students split across 10 exercises per problem (~74 per arm), the tests have limited power to detect educationally meaningful differences. The observed direction is consistently toward higher instructor success rates, and Problem 3 has p=0.065, so the data are at least suggestive of a difference. Without pre-specified equivalence margins, confidence intervals for the success-rate differences, or a power analysis, the claim that the exercises are comparable is not supported by the statistical evidence presented. The limitations section (5.5) acknowledges pre-selection and lack of long-term correlation, but does not flag this inference gap. This is the most load-bearing issue because the paper's practical recommendation—educators can substitute BugSpotter exercises without harming student performance—rests on demonstrating equivalence, not merely on failing to detect a difference. The pre-selection issue identified by the reader (Section 4.2) is real, but it compounds rather than replaces this statistical problem: even for the curated exercises actually studied, comparability has not been established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BugSpotter, a tool that uses an LLM to generate buggy C code from problem specifications, validates the code against a test suite, and presents students with debugging exercises in which they must design failing test cases. The authors report an expert-based quality/diversity evaluation (RQ1), an analysis of difficulty alignment between expert labels and student success (RQ2), and a classroom study comparing student success on LLM-generated versus instructor-created debugging exercises (RQ3). The classroom study involved 741 students and three problems, with five pre-selected LLM-generated exercises and five instructor-created exercises per problem. The paper claims that LLM-generated exercises are comparable in difficulty to instructor-created ones and suggests that BugSpotter can reduce educator workload.","tokens_in":12365,"tokens_out":3071,"duration_ms":27468,"significance":"If the central claims are properly supported, this is a useful contribution to computing education: it addresses debugging practice, an underserved area, and provides an automated pipeline with a concrete validation step. Strengths include the well-specified exercise-generation and validation methodology, the use of an external baseline of instructor-created exercises, expert annotation with reported inter-rater agreement, and a deployed classroom evaluation. The paper also ships a public demo, which aids reproducibility. However, the statistical support for the central 'comparable difficulty' claim is currently weak, and the classroom study evaluates a curated subset rather than the live generation pipeline, so the significance of the practical recommendation is not yet established.","major_comments":[{"comment":"The claim that LLM-generated exercises are 'comparable in difficulty' to instructor-created exercises rests on chi-square tests with p-values 0.675, 0.580, 0.065, and 0.140. A non-significant p-value only indicates failure to reject the null hypothesis of equal success rates; it does not provide positive evidence for equivalence, particularly with limited power and with Problem 3 close to conventional thresholds. The paper should report confidence intervals for the success-rate differences, a pre-specified equivalence margin, or an equivalence test (e.g., TOST), and ideally a power analysis; without this, the central practical recommendation in the abstract and Section 5.3 is not statistically supported.","section":"§5.3, Figure 6"},{"comment":"The classroom evaluation did not use BugSpotter's real-time generation pipeline: the authors pre-selected 5 high-quality, diverse exercises per problem, as stated in Section 4.2, and Section 5.5 admits this. Consequently, student success rates measure the curated exercises, not the uncurated output that the tool would produce in live use; this weakens the claim that BugSpotter 'produced' exercises comparable to instructor-created ones. The Limitations section should be extended to state that the comparability result applies to the curated subset and does not yet establish the effectiveness of unmoderated generation.","section":"§4.2 and §5.5"},{"comment":"The difficulty-diversity analysis in Section 5.2 labels the two easiest exercises as 'easy', the next two as 'medium', and the last as 'hard' after expert ranking, and then shows that student success rates decrease with these labels. With only five exercises per problem and no error bars or inferential test, this is a weak basis for the claim of 'strong alignment' between expert-assessed difficulty and student performance; the paper should provide per-exercise success counts or confidence intervals and temper the wording accordingly.","section":"§5.2, Figure 5"}],"minor_comments":[{"comment":"The text says '741 data points per problem' but the study assigns 741 students across ten exercises; please clarify whether the contingency table is per problem with 741 students or whether 741 is the total across all conditions, and report the per-arm counts.","section":"§5.3"},{"comment":"The heading 'Print A verage Rainfall' contains a stray space; it should be 'Print Average Rainfall'.","section":"Figure 8"},{"comment":"The phrase '5 instructed-created' should be '5 instructor-created'.","section":"§4.2"},{"comment":"The phrase 'greater than0.7' is missing a space; it should read 'greater than 0.7'.","section":"§4.1"},{"comment":"The web application supports Python exercises while the classroom study uses C; this discrepancy is acknowledged only implicitly and deserves a sentence of discussion in the limitations.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid tool-and-study contribution for SIGCSE, but the abstract and conclusion state a comparability claim that the current statistical analysis cannot support. The authors already acknowledge the pre-selection limitation, so the main fix is to align the claims with the evidence, add equivalence testing or confidence intervals, and clarify that the classroom result applies to curated exercises. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBugSpotter is a real, useful addition to the LLM-for-education toolbox. The exercise format is the best part: a student sees a C problem specification and a buggy implementation, then has to supply an input, the buggy output, and the correct output, with all three checked by actually running both programs. That turns debugging practice into something auto-gradable and forces students to engage with the specification rather than just poking at code. The generation pipeline is also sensible: prompt an LLM for buggy/fixed pairs with explanations, filter out non-compiling or unrelated code, and require that the fixed version passes the problem's test suite while the buggy version fails at least one test. There is no circularity here; the evaluation compares against instructor-created exercises, not against the pipeline's own outputs.\n\nThe classroom study is a serious effort: 741 students, random assignment across five LLM-generated and five instructor-written exercises per problem. The expert rubric, with two annotators and kappa above 0.7, is reasonable. The authors are also candid in Section 5.5 about pre-selection, the lack of long-term correlation with course performance, and the absence of a direct educator-workload measurement.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing. Section 5.3 and the abstract claim \"comparable difficulty\" based solely on chi-square tests where every p-value exceeds 0.05, with the lowest at 0.065. Non-significance is not evidence of equivalence. The observed success rates are consistently lower for LLM-generated exercises, and the sample sizes give limited power to detect educationally meaningful differences. To support the substitution claim, the paper needs confidence intervals, a pre-specified equivalence margin, or a power analysis. This is a statistical-inference gap, not a sign of sloppy execution; the authors simply reached one step past what the data support.\n\nThe pre-selection issue is real but secondary. The study evaluates a curated sample of five exercises per problem, not the live generation pipeline, so the deployment overstates what an uncurated tool would deliver. The authors acknowledge this. A minor mismatch: the public demo supports Python while the evaluation is in C.\n\nWho is this for: computing education researchers and instructors considering LLM-based exercise generation. It deserves a serious referee and likely acceptance with a requested revision that reanalyzes the RQ3 data using interval estimates and softens the comparability language. I would cite it and bring it to a reading group.","headline":"A genuinely useful debugging-exercise generator with a solid classroom study, but the comparability claim rests on non-significant chi-square tests and a curated exercise set.","tokens_in":12918,"tokens_out":2294,"would_cite":true,"duration_ms":23656,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BugSpotter generates debugging exercises with an LLM, and a 741-student classroom study finds students perform on them as well as on instructor-written exercises.","keywords":["debugging","programming education","exercise generation","large language models","test case design","BugSpotter","computing education","automated validation"],"falsifier":"Run BugSpotter in real time without pre-selection in the same course and compare student success rates against instructor-created exercises on the same problems; if the chi-square test shows a significant difference (p < 0.05) or the uncurated pipeline frequently fails validation, the comparability claim would not transfer to live use.","tokens_in":1434,"feed_emoji":"🐞","tokens_out":2000,"duration_ms":61679,"temperature":0.7,"pith_summary":"BugSpotter is a tool that turns a plain programming-problem specification into a debugging exercise: a large language model generates a buggy version of the intended function and a corrected version, and an automated stage checks both against the problem's test suite before showing the exercise to a student. The student's task is not to fix the code but to design a failing test case (input, buggy output, correct output), which requires reading the specification carefully and localizing the bug. The paper's central claim is that such LLM-generated exercises are usable in real classrooms: expert review rated them as well-matched and diverse, and in a deployment with 741 students, success rates on generated exercises were statistically indistinguishable from success rates on exercises hand-crafted by an instructor. If this holds, educators could generate debugging practice material automatically, easing a part of teaching that is often inconsistent and labor-intensive.","feed_headline":"LLM-generated debugging exercises match instructor-written ones","feed_subtitle":"In a 741-student class, success rates on LLM-made exercises were statistically indistinguishable from hand-crafted ones.","key_machinery":"The load-bearing mechanism is a two-stage generation-and-validation pipeline. In the generation stage, the LLM is prompted to reason about likely student bugs and output ten JSON tuples, each containing a buggy code, its fixed version, and an explanation. In the validation stage, a candidate is kept only if the fixed version compiles and passes the problem's entire test suite, the buggy version compiles and fails at least one test (by wrong output or runtime error, excluding hangs), and execution stays within a time limit. The student-facing exercise asks for a failing test case satisfying three machine-checkable criteria: the input makes buggy and fixed outputs differ, the reported correct output matches the fixed code, and the reported buggy output matches the buggy code. This design makes exercise quality depend on executable checks rather than on human curation.","core_discovery":"On the paper's own terms, BugSpotter establishes that LLM-generated debugging exercises can be substituted for instructor-authored ones without measurable loss in student performance. Across three single-function C problems, 741 introductory students each attempted exercises from a random assignment of five LLM-generated and five instructor-created variants per problem, and chi-square tests found no significant difference in success rates by source; the smallest p-value was 0.065 on one problem, with an aggregate p-value of 0.140. Expert ratings of difficulty also aligned with observed student success: exercises classed easy, medium, and hard produced descending success rates. The paper further reports that a cheaper model generated exercises comparable in quality and diversity to a more expensive one, and passed validation more often, making the pipeline cost-effective.","pith_inferences":["If the tool were used without pre-selection, average exercise difficulty could shift from what the study measured, since the authors chose five high-quality exercises per problem rather than sampling all validated output.","The p-value of 0.065 on one problem leaves room for a real difficulty gap on certain problems; more problems and larger samples would tell whether aggregate comparability holds everywhere.","Since a correct attempt requires matching both the buggy and correct outputs exactly, the exercise format doubles as a test of specification comprehension, which the paper does not directly measure.","Fine-tuning the generator on actual student bug-fixing attempts could make the exercises resemble natural novice errors more closely than the current prompt asking for bugs students may make."],"forward_implications":["Instructors can produce debugging exercises for single-function C problems without hand-crafting bugs, with the validation stage replacing human quality control.","Because the cheaper model performed comparably to the more expensive one, cost need not block adoption of the pipeline.","Students get practice reading problem specifications and designing discriminating test cases, not just fixing code.","Exercises can be sorted or recommended by difficulty, since expert-ranked difficulty predicted student success rates.","The same pipeline can be extended to other languages and to multi-function or class-based problems, as the demo already supports Python."],"supporting_citations":[{"why":"Supplies the language models used for buggy-code generation in the pipeline.","marker":"[31]"},{"why":"Motivates the prompt design that asks the model to reason about likely bugs before generating buggy and fixed code.","marker":"[40]"},{"why":"Provides the metacognitive-scaffolding rationale for having students generate test cases before programming.","marker":"[10]"},{"why":"Informs the expert quality rubric and frames debugging as hypothesis generation, a key reference for the evaluation design.","marker":"[23]"},{"why":"Supplies the chi-square test used to compare student success rates on LLM-generated versus instructor-created exercises.","marker":"[5]"},{"why":"Provides Cohen's kappa, used to verify substantial agreement between expert evaluators on rubric ratings.","marker":"[6]"},{"why":"Supplies the tokenizer used to compute edit distance between buggy and fixed code for the quality rubric.","marker":"[2]"}],"fun_headline_variants":["AI debugging exercises match instructor-written ones in class","BugSpotter: LLM-generated debugging tasks equal human-crafted","Study finds AI debugging practice as good as human-made","For debugging skills, AI exercises rival expert-created ones","In 741-student test, AI debugging exercises match humans"],"cache_read_input_tokens":14976,"weakest_assumption_plain":"The classroom comparison assumes that five pre-selected exercises per problem, hand-picked for quality and diversity, represent what BugSpotter would generate in regular use; the study did not test real-time generation.","fun_headline_variants_meta":{"raw":{"variants":["AI debugging exercises match instructor-written ones in class","BugSpotter: LLM-generated debugging tasks equal human-crafted","Study finds AI debugging practice as good as human-made","For debugging skills, AI exercises rival expert-created ones","In 741-student test, AI debugging exercises match humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2024,"prompt_tokens":945,"completion_tokens":1079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":998}},"tokens_in":561,"tokens_out":1079,"duration_ms":9754,"temperature":1.0,"reasoning_tokens":998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:18:43.180063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BugSpotter in real time without pre-selection in the same course and compare student success rates against instructor-created exercises on the same problems; if the chi-square test shows a significant difference (p < 0.05) or the uncurated pipeline frequently fails validation, the comparability claim would not transfer to live use.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the language models used for buggy-code generation in the pipeline."},{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"Motivates the prompt design that asks the model to reason about likely bugs before generating buggy and fixed code."},{"cited_title":"Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit","cited_arxiv_id":null,"evidence_quote":"Provides the metacognitive-scaffolding rationale for having students generate test cases before programming."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Informs the expert quality rubric and frames debugging as hypothesis generation, a key reference for the evaluation design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the chi-square test used to compare student success rates on LLM-generated versus instructor-created exercises."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Cohen's kappa, used to verify substantial agreement between expert evaluators on rubric ratings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the tokenizer used to compute edit distance between buggy and fixed code for the quality rubric."}],"review_version":1}