{"id":"43467f7d-78d4-4a54-8700-fdfae76b39f5","arxiv_id":"2508.09101","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.","lead":"This paper presents a fully automated pipeline in which LLMs write coding problems, reference solutions, and test cases, with outputs checked by a sandbox that runs 20 languages. The result, AutoCodeBench, contains 3,920 multilingual problems where the best tested model still scores only about 52% pass@1.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth is the LLM-generated solution, not an independent spec; the paper's own audit admits 12.4% invalid in six languages, and the 14 translated languages are unaudited, so Pass@1 rankings may not be robust to known noise.","rationale":"The reader's weakest assumption is the central issue: sandbox execution makes tests self-consistent with an LLM-written solution, not correct with respect to an independent problem specification. The paper's own Table 8 shows 87.6% validity in six languages, so this is not a hypothetical failure. What makes it load-bearing for the headline claim is that the benchmark's scores and difficulty are aggregate statistics over all 3,920 problems; a 12.4% invalid rate is large enough to alter rankings, and the 14 translated languages are even less secure because they are produced by translation and are not manually audited. The reader also rightly notes the difficulty filter is based on a single model family, but that is secondary: even a perfectly unbiased difficulty filter cannot repair test outputs derived from an arbitrary implementation. I am not changing the reader's verdict: CONDITIONAL remains appropriate because the paper is honest, open-sources the sandbox, and the quality issue is acknowledged, but the dataset release and independent audit are necessary before accepting the benchmark as a reliable multilingual evaluation tool.","tokens_in":61135,"tokens_out":4533,"duration_ms":51475,"concrete_test":"Release the per-instance manual audit labels for the six audited languages, and independently audit a stratified sample of at least 100 problems per translated language using the same protocol. Then recompute Pass@1 (Tables 4 and 5) after excluding invalid instances. If the relative ordering of the top five models changes, or the leading average shifts by more than 2 points, the benchmark's difficulty and ranking claims are not robust to the known noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AutoCodeGen's validity claim (§2.2.2) is that test outputs come from executing the LLM-generated reference solution in a sandbox, then the problem statement is written to match that solution. This only guarantees consistency with one arbitrary implementation; it does not guarantee the tests encode a correct, unambiguous statement. The manual audit (§4.1, Table 8) finds 87.6% valid problem–test pairs in six languages, with incomplete problem descriptions as the most frequent issue (Appendix B). Pass@1 and the union upper bound are computed over all 3,920 problems, so a 12.4% known contamination rate can shift scores and possibly rankings. The remaining 14 languages are built by approximate translation (§2.2.5) and receive no manual audit, leaving the validity of most of the benchmark unmeasured. The load-bearing assumption—that the LLM solution is correct ground truth—is thus violated at a known, non-negligible rate and at an unknown rate in the majority of languages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoCodeGen, a fully automated pipeline that generates multilingual code-generation benchmarks from Stack-Edu code snippets. The pipeline works in reverse order: an LLM evolves a self-contained solution, test inputs are generated and executed in a multilingual sandbox to obtain outputs, a problem statement is then written to match the solution and tests, and finally three filtering stages remove too-easy, low-quality, or redundant items. The resulting AutoCodeBench contains 3,920 problems across 20 languages, plus AutoCodeBench-Lite and AutoCodeBench-Complete. The authors evaluate over 30 models and report that the best model reaches only 52.4% Pass@1, that reasoning models outperform non-reasoning models, and that multi-logic problems cause a consistent performance drop. The paper also contributes a multilingual sandbox and a manual audit of six languages reporting 87.6% valid problem-test pairs.","tokens_in":61415,"tokens_out":3592,"duration_ms":42124,"significance":"If the central validity claim holds, the paper makes a useful contribution: it demonstrates a scalable, human-free pipeline for constructing multilingual code benchmarks and provides a large, balanced evaluation suite with unusually broad language coverage. The strengths are real: the execution-based test-output generation is machine-checked, the evaluation spans more than 30 models, the authors are transparent about their manual audit and about possible model bias, and they release a sandbox that could benefit the community. The significance is conditional, however, on the validity evidence. The manual audit reports 12.4% invalid instances in the six audited languages and no audit for the other 14 languages, and the difficulty filter is defined by a DeepSeek model's pass rate while DeepSeek-family models are also evaluated. These issues bear directly on the headline rankings and on the claim that AutoCodeBench is a high-difficulty benchmark, so the contribution is defensible but requires additional robustness evidence before the central claims can be accepted as stated.","major_comments":[{"comment":"The test outputs are produced by executing an LLM-generated reference solution, and the problem statement is then written to describe that same solution. This guarantees consistency with one implementation, not correctness with respect to an independent specification. The paper's own manual audit finds only 87.6% valid problem-test pairs in the six audited languages, with incomplete problem descriptions as the most frequent issue (Appendix B). Despite this, Pass@1 and the union 'Current Upper Bound' in Table 4 are computed over all 3,920 problems with no exclusion of the flagged invalid instances. A 12.4% known contamination rate is large enough to shift model scores and possibly rankings, especially because the failure mode is not random noise but a systematic mismatch between the stated problem and the hidden reference behavior. Please provide a sensitivity analysis (e.g., recompute sc","section":"§2.2.2, §2.2.3, §4.1, Table 8"},{"comment":"Fourteen of the twenty languages are constructed by approximate translation from unused data, and the manual audit covers only Python, C++, Java, JavaScript, Go, and Shell. The 87.6% validity figure therefore applies to a minority of the benchmark languages, and the validity of the majority is unmeasured. Translation is not a validity-preserving operation for this pipeline: the translation prompt in Figure 15 rewrites both the reference solution and the test functions, which can introduce new naming mismatches, precision issues, or altered edge-case behavior. The paper should either extend the manual audit to a sample of translated languages or explicitly restrict the validity claim to the six directly generated languages until such evidence is provided.","section":"§2.2.5, §4.1"},{"comment":"Difficulty labels and the 'easy-problem filter' are both defined by the pass rate of DeepSeek-Coder-V2-Lite, and DeepSeek-family models are among the evaluated models in Table 4. The paper acknowledges a possible favorable bias toward DeepSeek and attempts to measure it in Table 7, but that measurement covers only five models and 3,600 sampled instances in six languages, with no confidence intervals or statistical test. Because the filter removes problems that DeepSeek-Coder-V2-Lite solves, it can differentially affect the measured Pass@1 of DeepSeek-family models relative to other families. Please report the effect of each filtering stage on all evaluated model families, or otherwise quantify the bias more rigorously; the current evidence is too limited to support the claim in §4.2 that the impact is 'minimal.'","section":"§2.2.4, Table 2, §4.2, Table 7"},{"comment":"The LLM-as-Critic prompt in Figure 10 instructs the critic to 'assum[e] the problem is completely correct' and to evaluate only whether the test function is consistent with the problem. Appendix B then reports that the most frequent defect is incomplete problem descriptions. The automated critic is therefore structurally unable to catch the dominant failure mode, and the manual audit is the only validity evidence for the pipeline. This reinforces the need for the robustness analysis requested above; without it, the paper's claim that AutoCodeGen 'ensures the correctness and completeness of test cases' is stronger than the evidence supports.","section":"§2.2.4, Appendix B"}],"minor_comments":[{"comment":"AutoCodeBench is described as 'Human-Free' in Table 1, but §4.1 reports a manual audit by six professional annotators. Please clarify that the claim refers to the generation pipeline, not to post-hoc verification.","section":"Table 1 and §4.1"},{"comment":"The completion-based benchmark is called AutoCodeBench-Complete in §3.7 and AutoCodeBench-Comp in the Conclusion. Please use one consistent name.","section":"§2.2.6, §3.7, Conclusion"},{"comment":"The label 'Current Upper Bound' is potentially misleading: it is the union of problems solved by at least one evaluated model, not an upper bound in the mathematical sense. Consider renaming it 'Union Pass@1' or 'Ensemble Upper Bound.'","section":"Table 4"},{"comment":"Heading contains a typo: 'Multi-T urn' should be 'Multi-Turn.'","section":"§3.6"},{"comment":"The sentence 'We sample answers for each problem ten times using the model and validate the correctness via sandbox execution' should specify whether 'correctness' refers to the generated solution or to the model's sampled answer. This affects the interpretation of the 25.1% Python filtering figure.","section":"§2.2.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the reverse-order pipeline—generate the solution, execute test inputs in a sandbox, then write the problem statement from the solution—is a real departure from prior synthesis work, and it does produce a large, balanced multilingual resource that didn't exist. Second, the paper's own audit puts known validity at 87.6% on six languages, and the other 14 are translated and unaudited. That is the single most important number: 12.4% of the audited problems are known to be misaligned, and the rest are unknown.\n\nThe engineering is solid. The sandbox is a reusable artifact. The evaluation is broad, covering 30+ models across 20 languages, and the results are plausible. The authors also do something rare: they report a manual audit, publish the critic prompts, and discuss bias in the generation process. The paper is honest about several limitations that other work would hide.\n\nThe soft spots are real but contained. The ground truth is an LLM-generated solution, not an independent spec, so the tests reward whatever arbitrary implementation choices the generator made. The difficulty labels come from a pass-rate filter using DeepSeek-Coder-V2-Lite, and DeepSeek models are among those evaluated, making the difficulty rankings partly self-referential. The bias analysis in Table 7 is thoughtful but stops short of a quantitative adjustment. And the abstract's claim that the workflow 'ensures correctness and completeness' is not supported by the audit; the authors later allow incomplete test coverage, which is fine, but the abstract should match the evidence.\n\nThe manual audit is the right idea, but 87.6% is a single number without inter-annotator agreement. That matters because the annotation task is subtle—the annotators judge whether test functions align with a problem statement that was reverse-generated from those very functions. The fact that incomplete problem descriptions are the most common issue suggests the pipeline sometimes produces questions that underspecify the intended behavior.\n\nWho this is for: anyone building or using multilingual code benchmarks. The dataset and sandbox will likely be cited and reused. The leaderboard itself should be read with caution until independent validation lands.\n\nI'd send this to review. The core contribution is real, the limitations are known and partially disclosed, and the community will benefit from the artifact. But I'd tell the editors to require release of the exact dataset, prompts, and seeds, and to insist on validation coverage of the translated languages before publication.","headline":"Reverse-order generation is a real contribution, but the leaderboard's reliability rests on the authors' own 87.6% audit of six languages and zero evidence on the other 14.","tokens_in":61930,"tokens_out":3642,"would_cite":true,"duration_ms":39400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLMs can build valid, hard, multilingual code benchmarks without manual annotation by generating test inputs, executing them against an LLM-written reference solution in a sandbox, and writing each problem statement l","keywords":["code generation benchmark","multilingual evaluation","LLM-generated benchmarks","LLM-sandbox interaction","reverse-order problem generation","test case generation","AutoCodeBench","AI code evaluation"],"falsifier":"In a random sample of AutoCodeBench problems, have human programmers (or a different model family) write their own test cases directly from the problem statements, then run all models that pass the original generated tests on these independent tests. If pass rates drop sharply, the generated tests encode the reference solution's specific output choices and edge behaviors rather than the requirements stated in the problem.","tokens_in":61023,"feed_emoji":"💻","tokens_out":14815,"duration_ms":134940,"temperature":0.7,"pith_summary":"AutoCodeGen is a fully automated pipeline that turns raw code snippets into finished programming problems with executable test cases. Instead of asking an LLM to write tests from a problem statement, it reverses the order: an LLM first evolves a snippet into a self-contained reference solution, writes test inputs on that basis, executes solution plus inputs in a multilingual sandbox to obtain the expected outputs, and composes the problem statement last to describe the solution. Test outputs come from execution rather than generation, so the tests are internally consistent by construction; the problem is authored to match the solution, so the benchmark rewards reproducing that solution's behavior. The result, AutoCodeBench, holds 3,920 problems across 20 languages; the best of more than 30 evaluated models scores 52.4% Pass@1, and the combined upper bound across all models is 74.8%. The paper argues this makes large-scale, high-difficulty multilingual evaluation feasible without human curation, and it ships a simplified variant, a base-model completion variant, and a multilingual sandbox.","feed_headline":"No humans needed: LLMs write a 3,920-problem code benchmark","feed_subtitle":"AutoCodeBench runs 30+ models across 20 languages; the best scores just 52.4% Pass@1.","key_machinery":"The load-bearing mechanism is the reverse-order, execution-verified test generation loop: the reference solution is written first, test inputs are written second, expected outputs are obtained by running the solution on those inputs in the multilingual sandbox, and the problem statement is synthesized last to fit the solution and its tests. What does the work is that the sandbox execution converts test-output correctness from an LLM judgment call into a runtime fact, which is what lets the pipeline claim completeness and correctness of test cases without human review; the reverse order is what ensures the problem, solution, and tests describe the same artifact.","core_discovery":"The central claim is that reverse-order generation makes benchmark construction reliable enough to be fully automated, without manual annotation. The pipeline starts from real-world code snippets, evolves each into a self-contained reference solution with a strong code LLM, generates public and private test-input functions, executes the solution together with those inputs in a multilingual sandbox so the test outputs are computed rather than guessed, and then prompts the model to write a problem statement that matches the solution's function names, signatures, and input/output format, embedding the public tests as examples. Three filtering stages follow: a weak model removes problems it can","pith_inferences":["Editorial inference: the same reverse-order loop can serve as a training-data generator, which the paper itself flags as possible; a further consequence the paper does not explore is that benchmarks and training sets could co-evolve from one pipeline, letting a model family be evaluated on problems its own lineage generated.","Editorial inference: because expected outputs derive from a single reference solution, Pass@1 on this benchmark likely measures fidelity to that solution's behavior—naming, output formatting, edge-case choices—as much as algorithmic correctness; scoring the same problems against independent reference solutions would separate the two.","Editorial inference: the manual audit covered only six of the twenty languages, and the other fourteen pass through an extra translation step, so the 87.6% validity figure may not hold for low-resource languages; re-auditing a sample of translated problems would show where incomplete descriptions concentrate."],"forward_implications":["If the pipeline is sound, code benchmarks can be created and refreshed automatically for new languages or domains, removing the human annotation bottleneck that limits current benchmark scale.","The gap between the strongest model (52.4%) and the all-models union upper bound (74.8%) implies that combining strengths across models, or distilling cross-model solutions, has substantial headroom on these tasks.","Problems that require implementing multiple coordinated functions or classes consistently lower every model's score, identifying multi-logic synthesis as a specific weakness current models share.","Execution feedback from the sandbox raises scores for all tested models across three refinement turns, making sandbox-in-the-loop refinement a practical, model-scale-independent lever for improving code generation.","The completion-based subset extends high-difficulty multilingual evaluation to base models via few-shot inputs, a regime the paper argues previous base-model benchmarks do not cover."],"supporting_citations":[{"why":"Supplies Stack-Edu, the source of real-world educational code snippets that seed the solution-generation step.","marker":"(Allal et al., 2025)"},{"why":"Provides The Stack v2, the large code corpus from which Stack-Edu is derived, grounding the claimed diversity and practicality of the seeds.","marker":"(Lozhkov et al., 2024)"},{"why":"Provides the generator model used for solution evolution, test integration, problem generation, labeling, and translation in the pipeline.","marker":"(DeepSeek-AI, 2025a)"},{"why":"Provides the reasoning model used as LLM-as-critic in the quality-control filtering stage.","marker":"(DeepSeek-AI, 2025b)"},{"why":"KodCode is the prior approach that generates complete test cases directly; AutoCodeGen contrasts its execution-based test-output generation with this method.","marker":"(Xu et al., 2025)"},{"why":"CodeIO is the prior input-only generation approach that AutoCodeGen explicitly contrasts with when motivating its sandbox-based output generation.","marker":"(Li et al., 2025)"},{"why":"Evol-Instruct is the representative evolution-based synthesis method that this workflow extends and differentiates from.","marker":"(Luo et al., 2024)"},{"why":"OSS-Instruct/Magicoder is the prior open-source-snippet-based synthesis line that AutoCodeGen builds on and modifies into reverse-order generation.","marker":"(Wei et al., 2024b)"},{"why":"Defines HumanEval and the Pass@1 metric used for evaluating all models in this paper.","marker":"(Chen et al., 2021)"},{"why":"BigCodeBench serves as a manually annotated benchmark baseline in the comparisons and as evidence that incomplete problem descriptions are a general benchmark-creation challenge.","marker":"(Zhuo et al., 2025)"}],"fun_headline_variants":["Benchmark generation goes fully automated with reverse-order LLM pipeline","No manual annotations: LLMs produce a 3,920-problem multilingual suite","AutoCodeBench: LLMs build code tests, score 52.4% Pass@1 at best","How to test coding LLMs? Have LLMs write the tests","Reverse-order generation: the key to reliable LLM-made code benchmarks"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that each LLM-generated reference solution is correct ground truth: test outputs are computed by executing that solution and the problem is written to describe it, so any bug or under-specified edge behavior in the solution is encoded as expected behavior—and the paper's own six-language audit found only 87.6% of problem–test pairs valid, meaning more than one in ten violate this premise.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark generation goes fully automated with reverse-order LLM pipeline","No manual annotations: LLMs produce a 3,920-problem multilingual suite","AutoCodeBench: LLMs build code tests, score 52.4% Pass@1 at best","How to test coding LLMs? Have LLMs write the tests","Reverse-order generation: the key to reliable LLM-made code benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3369,"prompt_tokens":804,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2478}},"tokens_in":548,"tokens_out":2565,"duration_ms":19266,"temperature":1.0,"reasoning_tokens":2478,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:12:08.774617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a random sample of AutoCodeBench problems, have human programmers (or a different model family) write their own test cases directly from the problem statements, then run all models that pass the original generated tests on these independent tests. If pass rates drop sharply, the generated tests encode the reference solution's specific output choices and edge behaviors rather than the requirements stated in the problem.","supporting_citations":[],"review_version":1}