{"id":"38f8a298-661c-45a7-b1c4-91e61b5794b0","arxiv_id":"2506.12278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TestCase-Eval introduces fault coverage and fault exposure tasks over 500 Codeforces problems and finds that 19 LLMs, led by Qwen3-32B, remain far below human expert performance.","lead":"This paper presents TestCase-Eval, a new benchmark that measures how well large language models generate test cases for Codeforces algorithm problems. In tests across 19 models, even the best model catches only 43.8% of planted faults, while human experts catch 93.3%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing validation of LLM-generated test inputs makes the oracle-comparison scores (43.8% vs. 93.3%) uninterpretable until a parser and constraint checker are specified.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap in the paper: the evaluation assumes every LLM-generated test input is syntactically valid and within constraints, but no validation is described. This is the right concern because the entire benchmark score is produced by a black-box oracle comparison. If invalid inputs are common, the reported fault coverage and exposure rates can be inflated without reflecting genuine test-generation ability, and the central numeric claims become unverifiable. The concern is not fatal to the benchmark's potential value: it is fixable by adding a validation layer, reporting invalid-output rates, and recomputing scores, which is why the appropriate verdict remains conditional rather than accept or reject. I agree with the reader's judgment that the gap between LLMs and humans is likely real, but the exact 43.8% and 93.3% figures should not be taken at face value until this pipeline step is documented and tested. My concrete check directly tests whether the concern lands: if a small audit shows negligible invalid outputs and stable scores, the concern is resolved; if it shows high invalidity or large score shifts, the paper's headline needs revision. The reader and I are aligned on both the concern and the needed fix.","tokens_in":13848,"tokens_out":6980,"duration_ms":96246,"concrete_test":"Re-run the released pipeline on 50 Task2 instances from Qwen3-32B and 50 from GPT-4.1; for each generated response, write a per-problem validator that checks parseability, input-format conformance, and constraint satisfaction, then recompute the fault exposure rate using only valid inputs. If more than 5% of responses are invalid or the recomputed exposure rate shifts by more than 5 absolute points, the reported scores need correction and the comparison against the human baseline is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result depends entirely on the evaluation pipeline described in Sections 3.4 and Appendix C.2: each LLM output is treated as a test input, executed against a correct and a faulty solution, and any output difference is scored as fault exposure or coverage. The paper never specifies how model responses are extracted from CoT prompts that ask the model to \"Think step by step\" before emitting the input, and it never describes any validation that the emitted string is syntactically well-formed or satisfies the problem's input constraints. If a generated string contains explanatory prose, a code fence, or a value outside the stated bounds, the oracle comparison is no longer a meaningful test: the correct solution can crash or exhibit undefined behavior, and a differing crash or exit status can be counted as a detected fault. This would inflate Cov@N and Fault Exposure Rate in a model- and problem-dependent way. Because every reported number, including the headline gap between Qwen3-32B and human experts, passes through this unvalidated oracle, the benchmark's quantitative conclusions are not secure without an explicit input-validation step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TestCase-Eval, a benchmark for evaluating LLM-generated test cases on algorithm problems. It consists of 500 Codeforces problems from 2024 and a large collection of human-written correct and incorrect solutions. The benchmark defines two tasks: (1) Fault Coverage, where a model generates N test inputs and the union of incorrect submissions detected by these inputs is measured as Cov@N; and (2) Fault Exposure, where a model must generate a single test input that triggers a bug in a given faulty solution, measured as the Fault Exposure Rate. Nineteen LLMs are evaluated under direct-output and CoT prompting. The central empirical claim is that the benchmark is very challenging: the best model (Qwen3-32B) scores 43.8% on Fault Exposure, far below the human-expert rate of 93.3%, and even top models achieve only moderate Fault Coverage.","tokens_in":14075,"tokens_out":5158,"duration_ms":62578,"significance":"If the benchmark is valid, it fills a real gap in LLM test-generation evaluation by moving beyond line/branch coverage to fault-oriented metrics that matter for algorithmic problems. The dataset construction has notable strengths: problems are from 2024 (outside typical pretraining cutoffs), large scale (500 problems, tens of thousands of human submissions), and the two-task design separates broad coverage from targeted fault triggering. The authors also provide experimental evidence on 19 models, including reasoning models, and compare CoT vs direct prompting. However, the quantitative conclusions currently rest on an evaluation pipeline that does not validate the syntactic well-formedness or constraint satisfaction of LLM-generated test inputs. This makes the oracle comparison potentially unreliable and the headline gap (43.8% vs 93.3%) not yet interpretable. The benchmark construction and task definitions are a useful contribution, but the scoring pipeline must be made explicit and validated before the numerical results can be taken as evidence about LLM capability.","major_comments":[{"comment":"The evaluation treats the raw LLM output string directly as a test input without any specified extraction, parsing, or validation step. The CoT prompts in Figures 5 and 7 ask the model to 'Think step by step' and then output inside a code block, so the raw response may include reasoning text, markdown fences, or the code block delimiters themselves; no procedure is described for extracting the actual input. Furthermore, there is no check that the extracted string is syntactically valid or respects the problem's input bounds. Running an invalid or out-of-range input against the oracle and the faulty solution can cause crashes, timeouts, or undefined behavior in both programs, and any difference in exit status or crash signature is counted as 'fault exposure.' This inflates Cov@N and Fault Exposure Rate in a model- and problem-dependent way. Because every reported number, including the headline Qwen3-32B vs human expert comparison, passes through this unvalidated oracle, the quantitative conclusions are not secure. Please specify the exact extraction and validation procedure, report the rate of parse failures and constraint violations per model, and confirm that all scored inputs are well-formed and within stated bounds.","section":"Section 3.4 and Appendix C.2"},{"comment":"The human-expert performance is measured on a randomly sampled subset of 20 problems, while the model scores in Table 1 are apparently aggregated over all 500 problems. If the model scores are not computed on the same 20-problem subset, the headline comparison '93.3% vs 43.8%' confounds model capability with problem difficulty. For example, the 20-problem subset may be easier or harder than the full benchmark. Please report model performance on the same 20-problem subset used for the human evaluation, and clearly state whether the model numbers in Table 1 include the human-evaluated problems. Without this, the central claim of a large human-model gap is not rigorously established.","section":"Section 4.1 and Table 1"},{"comment":"The paper gives inconsistent counts for the incorrect-submission pool. Section 3.1 states 'we collect 200 incorrect submissions' per problem and a total of '100,000 submissions,' while Appendix B.2 reports a final set of 118,611 incorrect solutions across 500 problems, with an average of 237 per problem. These numbers affect the denominator of Cov@N (which uses |F_total|) and the sampling pool for Task 2. Please reconcile the counts and clarify whether the 100,000 figure in the abstract refers to a sample before filtering or a different stage of the pipeline.","section":"Section 3.1 vs Appendix B.2"},{"comment":"The filtering criteria in Appendix B.2 state that each sampled solution 'must be semantically valid and executable, passing compilation and basic test cases without syntax or runtime errors,' yet Table 2 and the accompanying analysis include Runtime Error (RE) as a major fault category. This is a direct contradiction. If RE submissions are included in the evaluation, the stated criterion is false; if RE is only a target error type that is not part of the filtered pool, the table's RE column needs a different explanation. Please clarify the inclusion or exclusion of RE submissions and adjust the filtering description accordingly.","section":"Section B.2 and Table 2"},{"comment":"The oracle is described as 'ten correct human-written solutions' that 'consistently produce identical outputs for the same test inputs.' The paper does not specify how the oracle output is determined during evaluation: is it the output of a single reference solution, a majority vote of the ten, or an agreement check that discards inputs where the ten disagree? For a valid generated input, if the ten correct solutions do not all produce the same output, the notion of a 'correct output' is undefined and the fault-exposure decision is ambiguous. Please state the oracle aggregation rule and what happens on disagreement.","section":"Section 3.1 (oracle definition)"}],"minor_comments":[{"comment":"The abstract says '100,000 human-crafted solutions,' while Appendix B.2 reports 118,611; please use consistent numbers throughout the paper and in the abstract.","section":"Abstract and Section 3.1"},{"comment":"The evaluation pipeline diagram shows 'Correct code' and 'Incorrect code' feeding into a sandbox but has no input-validation step; add an explicit 'input parsing and constraint check' box to reflect the required procedure.","section":"Figure 1"},{"comment":"The claim that 'Task 2 yields more stable and reproducible results across multiple evaluation runs, whereas Task 1 scores exhibit higher variance' is presented without evidence; either include variance or confidence intervals, or soften the claim.","section":"Section 4.2"},{"comment":"The prompts in Figures 5–8 ask for 'a challenging test input' (singular), but Task 1 reports Cov@N for N up to 20; clarify whether N independent generations are sampled per problem, and whether each generation may itself contain multiple test cases (as suggested in Appendix B.4).","section":"Appendix C.2 (prompts)"},{"comment":"The 'Human Expert' row is based on a 20-problem subset; add a table footnote or marker indicating this, so readers are not misled into comparing it directly with model rows over 500 problems.","section":"Table 1"},{"comment":"The citation for the sandbox environment is given as 'ExecEval (Khan et al., 2023)', but the reference list entry is titled 'xcodeeval'; please correct the citation to match the actual work used.","section":"References"},{"comment":"The Limitations section acknowledges lack of detailed error analysis and performance-bottleneck assessment, but does not acknowledge the input-validation gap or the human-baseline subset mismatch; please extend the limitations to cover these methodological issues.","section":"Limitations section"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially valuable benchmark and a clear empirical study, but the evaluation pipeline lacks a critical input-validation step that is essential for the oracle comparison to be meaningful. The human-baseline comparison also needs to be placed on the same problem subset. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The authors should also make the dataset and evaluation code publicly available with clear extraction and validation routines; the abstract's dataset link appears incomplete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate benchmark paper with a genuinely new task and a large dataset. The main claim that LLMs are far behind humans on targeted fault exposure is probably true, but the numbers as reported are not something to quote yet.\n\nWhat is new: the fault exposure task, where the model is given a specific faulty program and must generate one input that breaks it, is a real advance over coverage-style benchmarks. The fault coverage task extends TestEval by scoring against a large pool of human incorrect submissions rather than line/branch coverage. The dataset construction is mostly careful: problems from 2024 outside pretraining, special-judge problems excluded, and ten correct solutions run to establish an oracle.\n\nThe soft spots are real. Most important: the paper never says how LLM outputs are extracted and validated. The CoT prompt asks the model to 'Think step by step' before emitting a test input, but there is no description of stripping reasoning tokens or checking that the final string is syntactically well-formed and within constraints. If an input is malformed, execution may crash or produce undefined behavior, and the oracle comparison can then count noise as a detection. The scores could be inflated in a model-dependent way. This is fixable with a parser and a constraint checker, and the authors need to describe it.\n\nSecond: the human baseline comes from 20 randomly sampled problems, while models are evaluated on all 500. With 20 problems the human scores have wide error bars, and the paper reports none. A lucky or unlucky draw could shift the 93.3% number. The gap is so large that the qualitative conclusion would probably survive, but the exact comparison is not fair as presented.\n\nThird: the submission counts don't line up. Abstract says 100,000, main text says 200 per problem (100,000 total), appendix says final set is 118,611 with an average of 237 per problem. These need to be reconciled. Task 1 details also need a bit more precision about which pool is used for Cov@N and whether the same bugs are used across models.\n\nNone of this is fatal. The benchmark tasks are meaningful, the dataset is large, and the weaknesses are about reporting and missing validation rather than the underlying idea. The paper deserves a serious referee, but it needs a solid revision before the specific scores should be used.\n\nWho this is for: anyone working on LLM test generation or code evaluation benchmarks, and to a lesser extent software testing researchers interested in adversarial input generation. I'd bring it to a reading group and I'd probably cite it once the validation question is answered.","headline":"A useful new benchmark with a genuinely novel fault-exposure task, but the headline numbers need a validation and reporting pass before they should be quoted.","tokens_in":14584,"tokens_out":3466,"would_cite":true,"duration_ms":40991,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TestCase-Eval, a 500-problem benchmark of 2024 Codeforces contests and 100,000 human incorrect solutions, reports that even the best LLM tested exposes only 43.8% of targeted faults, versus 93.3% for human experts.","keywords":["test-case generation","LLM evaluation","fault coverage","fault exposure","Codeforces","algorithm problems","benchmark","competitive programming"],"falsifier":"Take a random sample of LLM-generated test inputs from both tasks, validate each one against the problem's stated input format and constraints, drop or repair the invalid ones, and recompute the coverage and exposure rates. If a large share turns out to be malformed, the current gap would shrink and the central comparison would be an artifact; if nearly all inputs are valid, the gap stands.","tokens_in":13652,"feed_emoji":"🐛","tokens_out":9647,"duration_ms":95617,"temperature":0.7,"pith_summary":"TestCase-Eval asks whether LLMs can produce test cases that actually catch faulty algorithm code, not just execute lines or branches. The paper builds a benchmark from 500 Codeforces problems from 2024 and 100,000 real human incorrect submissions, and defines two measurable tasks: Fault Coverage, the fraction of known faulty submissions caught by a set of generated inputs, and Fault Exposure, whether a single input crafted for one faulty submission makes it fail. Across 19 open and proprietary LLMs, the best model reaches 43.8% on Fault Exposure while two human experts reach 93.3%; coverage of diverse faults also trails human performance, though by less. If the benchmark measures what it claims, current LLMs are a long way from expert-level adversarial test generation for algorithm problems.","feed_headline":"Best AI model catches 43.8% of planted bugs; human experts catch 93.3%","feed_subtitle":"A new 500-problem benchmark measures LLM fault coverage and exposure; best model lands below half of human performance.","key_machinery":"The carrying mechanism is an oracle-comparison scoring pipeline: a generated test input is run against a set of known-correct human reference solutions and against a sampled set of known-faulty human submissions, and the input 'detects' a fault when the faulty program's output differs from the oracle's. Fault Coverage aggregates these hits across N generated inputs (Cov@N), while Fault Exposure evaluates a single generated input against one faulty submission. The dataset is constructed to make this meaningful: problems from 2024 to avoid pretraining contamination, no special-judge problems, ten correct solutions verified to agree on outputs, and 200 incorrect submissions per problem stratified by error type, language, and the index of the first failing test.","core_discovery":"TestCase-Eval's central claim is that LLM test-case generation should be judged by whether the generated inputs expose real incorrect implementations, and that on that standard all 19 evaluated models fall short of human experts. In the harder Fault Exposure task, the strongest model scores 43.8% against a 93.3% human-expert rate; even the best Fault Coverage numbers (Qwen3-32B at 50.8% cov@1 and 95.7% cov@20) sit just below human experts (56.2% and 97.2%). The paper also finds that reasoning-oriented LLMs outperform general-purpose LLMs, chain-of-thought prompting helps, Wrong Answer and Runtime Error faults are easier to expose than Time/Memory Limit Exceeded faults, and open-source models compete with proprietary ones.","pith_inferences":["Editorial inference: the same oracle-comparison setup is a ready-made training reward: any generated input that splits a known-correct solution from a known-buggy one gives a cheap scalar signal, and the current 43.8% ceiling suggests large headroom.","Editorial inference: the difficulty labels, based on the index of the first failed Codeforces test, are only a proxy; grouping failures by error type and input-size sensitivity would give a cleaner picture of which faults are genuinely hard.","Editorial inference: a constraint and format validator over generated inputs would test the benchmark's own validity, since malformed inputs can make correct and incorrect programs fail alike.","Editorial inference: the benchmark's 2024-only design makes it a natural instrument for tracking year-over-year progress in adversarial test generation, in the same way coding benchmarks track code generation."],"forward_implications":["If the measured gap is real, LLM-written test sets cannot yet substitute for human adversarial testing on hard algorithm problems, particularly for time- and memory-limit faults.","Because reasoning-oriented models and CoT prompting consistently raise scores, the bottleneck is partly deliberate analysis of possible failure modes rather than code-generation fluency.","The benchmark's error-type breakdown suggests that progress on asymptotic-complexity reasoning would transfer directly to better detection of TLE and MLE submissions.","The competitive open-source results imply that future gains may come from task-specific reasoning training rather than from larger proprietary API models alone."],"supporting_citations":[{"why":"Supplies the closest prior benchmark and the line/branch-coverage baseline that TestCase-Eval argues is too easy, motivating the two fault-based tasks.","marker":"TestEval (Wang et al., 2024)"},{"why":"Provides the sandbox environment that executes correct and faulty submissions against generated inputs; the entire scoring pipeline relies on it.","marker":"(Khan et al., 2023)"},{"why":"Gives the contamination-free evaluation design that TestCase-Eval follows by restricting problems to post-training-period 2024 contests.","marker":"(Jain et al., 2024)"}],"fun_headline_variants":["LLMs catch 43.8% of planted bugs; humans catch 93.3%","New benchmark shows LLMs lag humans in test-case generation","Best LLM exposes 43.8% of faults vs 93.3% for humans","TestCase-Eval: LLMs fall short on fault exposure","LLM test generation: 19 models, all below human fault catch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every LLM-generated test input is syntactically valid and satisfies the problem's constraints, so a differing output from a faulty submission really reflects that submission's bug rather than a malformed input; the paper does not describe any check of this.","fun_headline_variants_meta":{"raw":{"variants":["LLMs catch 43.8% of planted bugs; humans catch 93.3%","New benchmark shows LLMs lag humans in test-case generation","Best LLM exposes 43.8% of faults vs 93.3% for humans","TestCase-Eval: LLMs fall short on fault exposure","LLM test generation: 19 models, all below human fault catch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1294,"prompt_tokens":849,"completion_tokens":445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":465,"tokens_out":445,"duration_ms":4844,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:54:12.235743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of LLM-generated test inputs from both tasks, validate each one against the problem's stated input format and constraints, drop or repair the invalid ones, and recompute the coverage and exposure rates. If a large share turns out to be malformed, the current gap would shrink and the central comparison would be an artifact; if nearly all inputs are valid, the gap stands.","supporting_citations":[],"review_version":1}