{"id":"6079dfc8-1075-4e14-8bb0-ee817774dfec","arxiv_id":"2507.06920","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SAGA, a human-LLM workflow, generates tests that catch 90.62% of known human bugs on a new benchmark, though those bugs also helped create the tests.","lead":"This paper argues that current test suites for grading AI-generated code are too sparse and too similar, letting bugs slip through. The authors build SAGA, a human-AI method that generates harder tests from both correct and buggy human solutions, and a new benchmark to measure test quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline DR/VAcc gains in Tables 2 and 3 are evaluated on the same human wrong submissions SAGA's Differential Analysis consumes, so they establish fit to known bugs, not generalized verifier quality.","rationale":"Good-faith reading: the paper targets a real problem--LLM-generated test suites are homogeneous and miss human-like bugs--and its CodeCompass construction is a plausible way to make benchmarks stricter. The pass@1 comparisons are a useful independent sanity check. The weakest point is not the idea but the evaluation protocol. Because SAGA's Differential Analysis consumes the exact wrong submissions that define DR/VAcc, the large gains over random/TestChain/EvalPlus in Table 2 are expected even if SAGA contributes nothing beyond echoing the prompt-set bugs. The paper's formal Definition 2 makes this sharper: it claims a property over all incorrect programs but measures a finite set that is part of the generator's input. The held-out-bug split is cheap and decisive. I agree with the reader's weakest_assumption, and I do not think any other concern (e.g., the approximate saturation bound with unmeasured rho_eff, or the lack of released data) changes the verdict. REJECT is appropriate on current evidence, so verdict_should_be is UNCHANGED.","tokens_in":24035,"tokens_out":6308,"duration_ms":73833,"concrete_test":"Split each problem's human incorrect submissions into a 70% 'prompt set' (visible to SAGA's Differential Analysis) and 30% 'held-out set', stratified by problem and difficulty, repeated over 5 random folds. Regenerate SAGA and all Table 2 baselines at n=50 using only the prompt-set bugs, then recompute DR@50 and VAcc@50 on the held-out bugs. If SAGA's held-out VAcc@50 is not significantly above the random Input-Interpreter held-out VAcc@50 (bootstrap CI), the reported 32.58% reflects memorization of the prompt bugs rather than generalization. Also recompute Table 3's VAcc@40 on the held-out subset only; release per-problem results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: SAGA's Differential Analysis (Sec. 3.1) is given incorrect human submissions S_wrong, pairs each with the user's corrected version, and generates inputs I_diff where S_wrong(I_diff) != S_correct(I_diff); these inputs are explicitly added to the suite. The headline metrics in Sec. 2.1/Table 2 are then computed on TCGBench-Lite using the same S_wrong collection (avg. 41.41 per problem, Sec. 3.2.1). Consequently, DR@50 = 90.62% and VAcc@50 = 32.58% measure how well the suite covers the very bugs shown to the generator, not whether it detects unseen incorrect programs. Definition 2 formally defines VAcc over all S in S_wrong(P), an infinite set; the experiments replace this with a finite, generator-linked set. Table 3's CodeCompass-vs-LCB-v6 VAcc gain is contaminated in the same way, because those verifier-quality metrics are also computed against the TCGBench-Lite human bugs. The CodeCompass pass@1 drop in Figures 7-8 is a partially independent signal, but it is not the basis for the paper's numeric TCG claims and is itself limited to 101 shared AtCoder problems without a matched-test-count analysis. The paper's own Appendix H.2 ablation shows that removing Differential Analysis costs 8.35 points of VAcc@50 on Codeforces; that is precisely the channel through which evaluation-set leakage enters. Without held-out bugs, the central claim is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the quality of test suites used to evaluate LLM-generated code. It defines Detection Rate (DR) and Verifier Accuracy (VAcc) metrics, constructs TCGBench (1,840 competitive programming problems with an average of 36.66 incorrect human submissions per problem), and proposes SAGA, a human-LLM collaborative test-case generation framework that leverages both correct solutions (S_human) and incorrect submissions (S_wrong) via Multidimensional and Differential Analysis. The paper reports DR@50=90.62% and VAcc@50=32.58% for SAGA on a 270-problem subset, claims a 10.78 percentage-point higher VAcc than LiveCodeBench-v6 on a shared AtCoder subset, and derives an asymptotic saturation bound for DR. It also introduces TCGCoder-7B, a distilled 7B model for test generation.","tokens_in":24263,"tokens_out":7758,"duration_ms":76300,"significance":"The problem is important and the paper contributes a substantial benchmark and a plausible method. The idea of mining both correct and incorrect human solutions is valuable, and the paper includes extensive experiments and releases code/data. However, the headline results are not interpretable as generalized verifier quality because the evaluation is circular: the same incorrect submissions that seed SAGA's Differential Analysis are used to compute DR and VAcc. Consequently, the significance of the proposed method for detecting unseen bugs is currently untested. The theoretical saturation result also rests on an unproven approximating model.","major_comments":[{"comment":"The evaluation of SAGA is circular: Differential Analysis consumes the TCGBench-Lite wrong submissions S_wrong to produce inputs I_diff, and Table 2 then reports DR@50=90.62% and VAcc@50=32.58% computed on the same S_wrong set (average 41.41 per problem). These numbers measure how well the generated suite covers the exact bugs shown to the generator, not its ability to detect unseen incorrect programs. The paper's own ablation in Appendix H.2 shows that removing Differential Analysis lowers VAcc@50 by 8.35 points on Codeforces, confirming that the reported gains flow through this leakage channel. A held-out set of human bugs that is never shown to the generator must be used to support the claimed generalized verifier quality.","section":"§3.1 and §3.2.1, Table 2"},{"comment":"Definition 2 defines VAcc over the infinite set S_wrong(P) of all incorrect solutions, but the experiments replace it with the finite collection of human submissions used to build the test suites. The paper provides no argument that this finite sample is representative of the infinite set. The reported VAcc values are therefore not estimates of the quantity in Definition 2. The authors should either redefine VAcc for finite evaluation sets or supply a representativeness analysis (e.g., convergence of VAcc as the number of bugs grows).","section":"§2.1, Definition 2"},{"comment":"Corollary 1 and the supporting Theorem 1 rest on an approximating effective-sample-size model whose parameters p_bar and rho_bar_eff are not estimated from data. The proof is a sketch, and the empirical validation in Figure 4 shows only qualitative saturation without a quantitative fit. This is insufficient to support the stated asymptotic bound or the claim that the experiments 'validate the impact of correlation rho'. I recommend presenting this as a heuristic or providing a rigorous derivation and model fit.","section":"§2.2 and Appendix C, Corollary 1"},{"comment":"The paper reports point estimates without uncertainty quantification. The comparison with LiveCodeBench-v6 in Table 3 is based on only 101 shared AtCoder problems, and the 10.78 percentage-point VAcc gap and the pass@1 re-ranking in Figures 7–8 could easily arise from sampling noise. Bootstrap confidence intervals or per-problem standard errors should be reported for all headline metrics.","section":"Tables 2–3, Figures 7–8"}],"minor_comments":[{"comment":"Section 3.2 states SAGA achieves DR 93.81% and VAcc 41.33% on the full TCGBench, whereas Table 2 reports DR@50=90.62% and VAcc@50=32.58% on TCGBench-Lite; the discrepancy is not explained.","section":"§3.2 vs Table 2"},{"comment":"The abstract mentions a 12.14% Verifier Accuracy improvement, while the contributions list says 15.86% and Table 2 shows a 15.86 percentage-point gain over Input-Interpreter; clarify which comparison is meant.","section":"Abstract and Contributions"},{"comment":"The term 'Shuman' is used without definition; it appears to denote the set of correct human solutions (S_correct). Please define it at first use.","section":"§3.1"},{"comment":"In Table 4, 'Avg.Swrong/Problem' is a typo; it should be 'Avg. S_wrong/Problem'.","section":"Table 4"},{"comment":"The claim that TCGCoder-7B 'outperforms all these established baselines' is ambiguous because SAGA itself beats TCGCoder-7B on most metrics; rephrase to say it outperforms the non-SAGA baselines.","section":"§3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern raised in the reader's report is valid and is the main obstacle to acceptance. The paper's central claims about generalized verifier quality cannot be supported with the current evaluation design. I believe a major revision that introduces a held-out bug set and re-estimates all headline metrics could make the paper publishable. Also, the inconsistency between the full TCGBench and TCGBench-Lite results should be resolved. The theoretical result (Corollary 1) needs either proof or reframing as a heuristic. Given the significance of the benchmark and the method, I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real attempt at a real bottleneck, but the headline numbers are circular. SAGA's Differential Analysis is fed the same human wrong submissions that define DR and VAcc, so Tables 2 and 3 mostly show how well the suite covers the bugs it was built on. That doesn't make the paper useless, but it does mean the central claim—better verifier accuracy—is unproven.\n\nWhat is genuinely useful: the formal definitions of Detection Rate and Verifier Accuracy are a step up from ad hoc coverage metrics; TCGBench is a worthwhile asset with real human WA/TLE submissions; the dual-pronged idea of mining both correct and incorrect solutions is sensible; and the CodeCompass pass@1 comparison, where model scores drop and re-rank versus LiveCodeBench-v6, is a partially independent sanity check that the synthesized tests are stricter. The saturation corollary is basically Kish's design effect restated for test generation, and the paper cites Kish—fine.\n\nThe soft spots, in order: (1) The circularity. Definition 2 defines VAcc over all incorrect programs, but experiments only use the finite Swrong that SAGA was shown. The Appendix H.2 ablation quantifies the problem: dropping Differential Analysis costs 8.35 VAcc points, which is exactly the leakage channel. (2) No held-out evaluation, no error bars, and no released code or data, so the finite-sample proxy for the infinite-set metric is unverified. (3) The saturation bound depends on an effective correlation parameter that is never estimated independently; it's a plausible model, not a proven limit.\n\nOne thing in the paper's favor: it doesn't hide the mechanism. The pipeline is described clearly enough that a reader can spot the circularity. The fix is straightforward—rerun on wrong submissions held out from generation, report per-problem variance, release the artifacts. As it stands, I would not accept the numbers, but I would send it to review, because the problem is important and the framework, once cleaned up, could be a useful contribution to how we build and judge code benchmarks.","headline":"Real bottleneck, plausible method, but the headline verifier-accuracy numbers are circular because the generator and the evaluation share the same human bug set.","tokens_in":24900,"tokens_out":2027,"would_cite":true,"duration_ms":22542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAGA, a human-LLM test generator, catches 90.6% of hidden code bugs and produces benchmarks that re-rank top models.","keywords":["test-case generation","code generation evaluation","verifier accuracy","detection rate","LLM code benchmarks","human-LLM collaboration","RLVR","test-suite diversity"],"falsifier":"Hold out a set of wrong solutions that SAGA never saw, such as submissions collected from contests released after its training cutoff, and run SAGA-built verifiers on them. If verifier accuracy stays near the random-interpreter baseline on that held-out set, the claim that SAGA excels at detecting unseen human errors is refuted.","tokens_in":1725,"feed_emoji":"🧪","tokens_out":3158,"duration_ms":121781,"temperature":0.7,"pith_summary":"Standard code benchmarks use small test suites that miss many bugs, so LLM scores are inflated and reinforcement learning from verifiable rewards gets noisy feedback. The paper argues this happens because LLM-generated test cases cluster around the same systematic failure patterns, while human bugs are diverse and dispersed. It formalizes two metrics, detection rate and verifier accuracy, builds TCGBench from recent competitive-programming problems with tens of thousands of real incorrect submissions, and proposes SAGA, a pipeline that combines correct human solutions and incorrect human submissions to generate targeted tests. On its 270-problem curated subset, SAGA reaches a 90.62% detection rate and 32.58% verifier accuracy at 50 tests, and the benchmark it synthesizes, CodeCompass, scores 10.78 percentage points higher on verifier accuracy than LiveCodeBench-v6 suites on a shared subset. If the method holds up, it offers a way to build stricter, less gameable code benchmarks and improve the reward signals used to train code models.","feed_headline":"New tests catch 90.6% of hidden code bugs","feed_subtitle":"SAGA pairs human error insights with LLM reasoning to build stricter benchmarks that re-rank leading models.","key_machinery":"SAGA is the central mechanism: a two-pronged, prompt-driven pipeline. Multidimensional Analysis uses roughly ten correct human solutions per problem to deconstruct their defensive logic into mathematical constraints such as equivalence classes and boundary values, then generates targeted adversarial inputs; Differential Analysis pairs each incorrect submission with its author's later corrected solution and asks the LLM where the two diverge, directly exposing constraint-handling and edge-case errors. Generated Python case scripts carry math explanations and self-validation code, and a ground-truth interpreter turns inputs into outputs. The paper also supplies a theoretical backstop: if test-detection events are positively correlated with effective average correlation $\\bar{\\rho}_{\\mathrm{eff}}$ and average per-test detection probability $\\bar{p}$, detection rate saturates at $1-(1-\\bar{p})^{1/\\bar{\\rho}_{\\mathrm{eff}}}<1$ as the number of tests grows, which motivates breaking correlation rather than merely adding tests.","core_discovery":"The paper's central claim is that the verifier, not the generator, is the weak link in LLM code evaluation, and that verification quality can be engineered by feeding an LLM structured human priors from both sides of the error boundary. SAGA (Strategic Adversarial & Constraint-differential GenerAtive workflow) analyzes correct human solutions to extract constraints and boundary strategies, and differentially analyzes incorrect human submissions against their corrected versions to find inputs where the wrong program diverges; Python case scripts then generate test inputs, a ground-truth interpreter produces outputs, and self-validation code checks the inputs before use. On TCGBench-Lite, SAGA reaches a detection rate of 90.62% and a verifier accuracy of 32.58% at 50 tests, compared with 81.07% and 16.72% for the random input-interpreter baseline, and the CodeCompass suites it synthesizes raise verifier accuracy by 10.78 percentage points over LiveCodeBench-v6 on the shared subset. The paper interprets these gains as evidence that test-case generation should be judged not by the number of tests but by per-test potency and inter-test correlation, and that human error diversity is an input to generation, not a byproduct of it.","pith_inferences":["The most informative follow-up experiment would be time-sliced: train SAGA on submissions before a cutoff, then score its verifiers on submissions collected after the cutoff; a large drop would bound how much of the reported gap carries over to genuinely unseen bugs.","The differential-analysis step is a general recipe, pair any incorrect artifact with its corrected counterpart and ask an LLM where they diverge, which could transfer to bug-finding in reasoning chains or proof checking.","The saturation bound is a general statement about correlated tests, so the same reasoning could be used to audit any test-suite generator, including non-LLM fuzzers, by estimating its effective correlation from an accuracy-versus-size curve.","Distilling SAGA into a 7B model suggests test-case synthesis could eventually run inside the training loop at scale, not just at benchmark construction time."],"forward_implications":["Code generation leaderboards built this way should show lower, and more separable, pass@1 numbers; in the paper's shared subset the relative drop averaged 9.56% and two model pairs swapped ranks.","RLVR pipelines that use SAGA-style verifiers should receive sharper reward signals and face less reward hacking, because more wrong programs are actually rejected.","More tests alone cannot close the gap: the saturation corollary and measurements show random input sampling plateaus far below complete detection.","The SAGA-distilled 7B specialist reaches a detection rate of 89.44% and a verifier accuracy of 29.11% at 50 tests, beating larger-model baselines, so high-quality test-case generation can be made affordable.","The same recipe can scale to larger problem sets, providing training data with more accurate rewards."],"supporting_citations":[{"why":"is the random input-interpreter baseline that SAGA's detection-rate and verifier-accuracy gains are measured against.","marker":"[19]"},{"why":"is the mutation-based baseline (EvalPlus) and the prior evidence that expanding tests sharply lowers pass rates.","marker":"[27]"},{"why":"is the direct-generation baseline (TestChain) and the prior work on decoupling input and output generation.","marker":"[22]"},{"why":"is the source of the sparse 7.7-tests-per-problem observation that motivates stronger verifiers.","marker":"[6]"},{"why":"is the benchmark whose verifiers the paper audits, finding that 50% of problems fail to detect known errors.","marker":"[37]"},{"why":"is the RLVR training framework whose reward accuracy the paper argues depends on verifier quality.","marker":"[8]"},{"why":"supplies the effective-sample-size concept behind the saturation bound on detection rate.","marker":"[20]"}],"fun_headline_variants":["SAGA: Human-guided tests catch 90.6% of LLM code faults","Improved verifier accuracy: SAGA adds 10.78 points over LiveCodeBench","Testing beats generation: New method finds 90.6% of hidden bugs","LLM code verification: Human-guided tests boost accuracy by 10.78%"],"cache_read_input_tokens":26880,"weakest_assumption_plain":"The headline detection and verifier-accuracy numbers are computed against the same human wrong submissions that SAGA was shown, so the proof does not yet cover wrong programs SAGA has never seen.","fun_headline_variants_meta":{"raw":{"variants":["SAGA: Human-guided tests catch 90.6% of LLM code faults","Improved verifier accuracy: SAGA adds 10.78 points over LiveCodeBench","Testing beats generation: New method finds 90.6% of hidden bugs","LLM code verification: Human-guided tests boost accuracy by 10.78%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2406,"prompt_tokens":1056,"completion_tokens":1350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":1262}},"tokens_in":672,"tokens_out":1350,"duration_ms":12265,"temperature":1.0,"reasoning_tokens":1262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:51:43.328187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of wrong solutions that SAGA never saw, such as submissions collected from contests released after its training cutoff, and run SAGA-built verifiers on them. If verifier accuracy stays near the random-interpreter baseline on that held-out set, the claim that SAGA excels at detecting unseen human errors is refuted.","supporting_citations":[{"cited_title":"Codeforces cots","cited_arxiv_id":null,"evidence_quote":"is the benchmark whose verifiers the paper audits, finding that 50% of problems fail to detect known errors."},{"cited_title":"Survey sampling","cited_arxiv_id":null,"evidence_quote":"supplies the effective-sample-size concept behind the saturation bound on detection rate."}],"review_version":1}