{"id":"08cc3974-855b-49ce-b671-c4a05fc59d27","arxiv_id":"2412.13464","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Self-training with execution feedback, where generated tests are validated against ground-truth code and generated code is filtered by validated tests, improves code and test generation on APPS.","lead":"The authors built GenX, a pipeline that trains a code model and a test model together and uses execution results to correct and filter each other's outputs, producing a larger training set. On the APPS benchmark, models trained on this augmented data beat the model trained on the full original dataset on the top-1 code metric and on test generation, though the gains are small and the evaluation has caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmented tests are never validated against official APPS tests, so rejection sampling may retain code that passes easy self-generated tests but fails hidden tests, potentially manufacturing the reported gains.","rationale":"The reader's weakest_assumption is that augmented tests are representative enough of hidden tests to filter code without false positives. I agree that this is the load-bearing point. The paper's entire pipeline—test augmentation, code rejection sampling, and the final comparison in Table 3—rests on the validity of this assumption. Section 3.4 explicitly claims that the large number of generated tests lets the authors 'confidently judge the correctness of the newly generated code and avoid false positives,' but no experiment validates this against official APPS tests. The proposed held-out test check would settle whether augmented tests are genuinely predictive of hidden-test performance. If the check fails, the central claim that APPS+ training outperforms APPS training would be unsupported, because the reported pass@1 gains could reflect selection for easy self-generated tests rather than improved code generation. If the check succeeds, the conditional acceptance is appropriate. I therefore recommend keeping the conditional verdict while making the representativeness validation an explicit acceptance condition. This is not an objection to the method's overall plausibility; the idea of using execution feedback to augment both code and tests is reasonable, and the paper provides some supporting evidence such as improved pass@1 on HumanEval and MBPP in Appendix A. However, the lack of direct validation of the augmented test distribution is the weakest link in the central argument, and it should be resolved empirically.","tokens_in":11140,"tokens_out":4932,"duration_ms":50865,"concrete_test":"Hold out a random half of the official test cases for a subset of APPS training problems that have multiple official tests. Generate augmented tests as in Section 3.3 using the ground-truth solution, then use Section 3.4 rejection sampling to filter generated code with those augmented tests. Finally, compute the pass rate of the retained code on the held-out official tests, and compare it with the pass rate of code retained by the original official tests. If code passing augmented tests fails held-out official tests at a substantially higher rate, the Table 3 gains are likely overfitting to the augmented test distribution rather than reflecting true correctness improvement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim depends on the Section 3.4 assertion that, after test augmentation, 'we can confidently judge the correctness of the newly generated code and avoid false positives.' The augmented tests are produced by executing model-generated inputs on one retained ground-truth solution (Section 3.3). Nothing in the paper verifies that these tests are representative of the official APPS test distribution or that they have sufficient discriminative power to filter code without false positives. Table 5 is indirect evidence at best: it shows that using APPS+ tests yields better pass@1 than using APPS's sparse tests, but it does not measure whether code retained by augmented tests would pass the official hidden tests. Because the headline result in Table 3 is pass@1 on official APPS test cases, any systematic bias in the augmented tests—e.g., toward easy or formulaic inputs—can inflate the reported gains by training the model on code that is optimized for weak tests rather than for genuine problem-solving. The concern is amplified by the fact that APPS+ contains fewer code solutions per problem than APPS (20.14 vs. 23.45), so the improvement cannot be attributed to data quantity alone; it must come from test quality, and that quality is never directly assessed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenX, a training and inference framework that jointly augments code and test generation data on the APPS dataset using execution feedback. Starting from a reduced dataset (APPS-), it iteratively generates test inputs with a fine-tuned test-generation model, executes them on the single retained ground-truth solution to obtain outputs, and then uses the resulting augmented test suite to filter generated code solutions via rejection sampling. This produces an augmented dataset (APPS+) on which code and test generation models are trained. At inference, a dual-critic scoring algorithm (Algorithm 2) ranks generated code and tests from a pass/fail matrix without ground truth. Experiments on APPS, HumanEval, and MBPP report that models trained on APPS+ achieve better pass@1 on APPS test cases than models trained on APPS- or the original APPS, and that the proposed scoring function outperforms CodeT on code and especially test ranking.","tokens_in":11435,"tokens_out":5063,"duration_ms":43548,"significance":"If the central claim is correct, the paper offers a practical way to escape the dependence of execution-feedback code generation on pre-existing, dense test suites: a model can bootstrap its own tests from a single ground-truth solution and use them to filter correct code. The framework is clearly described and the inclusion of test-generation metrics, a CodeT comparison, and additional HumanEval/MBPP experiments are strengths. The dual-critic scoring algorithm is simple and shows consistent ranking improvements. However, the main empirical claim rests on small pass@1 differences obtained on a subsampled APPS test set without error bars, and a key assumption—that augmented tests are representative enough to avoid false positives in rejection sampling—is not directly validated. The contribution is therefore promising but needs stronger evidence before the headline result can be accepted.","major_comments":[{"comment":"The headline claim that APPS+ outperforms APPS relies on pass@1 differences of 1.5, 0.7, and 0.8 percentage points for intro, interview, and competition levels, each evaluated on 100 problems. No error bars, confidence intervals, or multiple-seed results are reported, so these differences may be within sampling noise. The abstract and §4.2 should either be qualified or supported by repeated runs with variance estimates.","section":"§4.1, Table 3"},{"comment":"The rejection-sampling filter in §3.4 uses augmented tests that are generated by executing sampled inputs on the single ground-truth solution (§3.3). The assertion that this allows us to 'confidently judge the correctness of the newly generated code and avoid false positives' is not directly verified: the paper never checks whether code retained by augmented tests also passes the official APPS hidden tests. Table 5 only compares filtering with APPS+ tests versus APPS tests in terms of final pass@1; it does not measure the representativeness of the augmented tests. Please add an experiment that evaluates code retained by augmented tests on the official hidden tests, or report the pass rate of augmented tests against the official tests.","section":"§3.3–§3.4, Table 5"},{"comment":"The evaluation uses a 300-problem subset of APPS (100 per difficulty) after filtering out problems without valid solutions and keeping only the shortest solution for each problem. This is a modified benchmark, so the reported pass@k numbers are not directly comparable to prior APPS results and the generalizability of the central claim is limited. Please report results on the full APPS test set or provide a clear justification for the subset and its relationship to the official benchmark.","section":"§4.1, Evaluation"},{"comment":"The n@k metric is defined as 'selecting the top-n scored code solutions (which may exceed n) and then calculating their pass@k.' This is ambiguous: if the selected set can contain more than n solutions, the effective number of candidates differs from n, making the metric incomparable to standard pass@k. Please state exactly how n@k is computed, including how ties are handled, and ensure the CodeT comparison in Table 4 uses identical selection rules.","section":"§4.1, n@k definition; Tables 3–4"},{"comment":"The pass num metric can be inflated by trivial or redundant tests (e.g., constant outputs). The paper prunes tests by length and notes that some outputs are 'too trivial' in §3.3, but no quantitative measure of augmented-test discriminative power is reported. Because the entire data-augmentation pipeline depends on test quality, please report the distribution of augmented test outputs, the fraction of non-trivial tests, or the pass rate of augmented tests against official hidden tests.","section":"§3.3, Table 2"}],"minor_comments":[{"comment":"The phrase 'concurrently trains a code generation model and a test generation model' is misleading because Algorithm 1 trains the test model and the code model sequentially (first M_t in the test-augmentation loop, then M_c in the code-augmentation loop). Please rephrase.","section":"Abstract"},{"comment":"The sentence 'To ensure diversity, we remove duplicate test cases' does not specify the deduplication criterion (exact input-output match, input only, or normalized JSON). Please clarify.","section":"§3.3"},{"comment":"The claim that Algorithm 2 'typically converges within a few hundred iterations' is supported only by simulation; please report the number of iterations n used in the experiments and state whether convergence was checked for all problems in the test set.","section":"§3.5"},{"comment":"The pass rate for APPS+ on competition problems (26.93%) is lower than for APPS- (27.50%), yet §4.2 states that APPS+ outperforms APPS- in both pass rate and pass num. This inconsistency should be discussed.","section":"Table 2"},{"comment":"The y-axis label 'Score' is defined only in the caption as the product of pass rate and pass num; please label the axes directly and include units or a short explanation in the figure itself.","section":"Figures 2 and 3"},{"comment":"The concluding limitation paragraph correctly acknowledges that text-rich error messages are not used; this is a useful statement of scope. There is a minor capitalization inconsistency at the start of the last paragraph ('Currently, We do not utilize...').","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper does not mention whether code or data will be released, and several hyperparameter choices (temperatures, sampling budgets, length thresholds) are given only in prose. Given that the evaluation is on a modified 300-problem subset of APPS, the editor should be aware that the headline numbers are not directly comparable to standard APPS benchmarks. The main uncertainty is whether the modest pass@1 gains are significant and whether the augmented tests are truly representative; these are fixable with additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take: the method is a reasonable extension of execution-feedback code generation, but the headline claim is under-supported because the augmented tests are never validated against the official APPS tests, and the reported gains are small and may be within noise.\n\nWhat's actually new: unlike CodeT and similar inference-time scoring approaches, GenX uses execution feedback to iteratively expand both test and code training data, with one model's outputs filtered by the other. That loop is the contribution, and it's honestly described. The reciprocal scoring function (Algorithm 2) is a normalized variant of CodeT's consensus method; the direct comparison against CodeT is good to see.\n\nWhere it's soft: the main weakness is the one you flagged. The augmented tests are generated by executing model-generated inputs on a single retained ground-truth solution (Section 3.3), but the paper never shows these tests are representative of the official hidden tests or that they have enough discriminative power. Section 3.4 states we \"can confidently judge the correctness of the newly generated code and avoid false positives,\" but that confidence is never empirically checked. Table 5 is indirect evidence at best: it shows that more tests help, not that the specific augmented tests are good filters. The headline gains in Table 3 (e.g., competition pass@1 from 0.30 to 1.10) are small, with no error bars, and the n@k metric is ambiguous—\"selecting the top-n scored code solutions (which may exceed n)\" doesn't parse. The APPS+ dataset also has fewer solutions per problem than APPS (20.14 vs 23.45), so the comparison isn't controlled. No released artifacts, so independent verification is impossible. These are fixable with more careful evaluation, but as-is the central claim is not fully supported.\n\nWho it's for: people working on code generation with limited test annotations, and anyone interested in mutual training of code and test models. It deserves peer review because the idea is plausible and the experiments are non-trivial, but it needs major revisions.\n\nMy recommendation: send it to review, but ask for error bars, a clearer definition of n@k, and ideally an analysis that directly validates augmented tests against official test outcomes. The stress-test note holds up; the paper should address it head-on.","headline":"A plausible execution-feedback loop, but the evaluation never checks whether the augmented tests actually predict official test outcomes, and the reported gains are small and possibly noise.","tokens_in":11904,"tokens_out":2626,"would_cite":false,"duration_ms":24427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-made tests and code push pass@1 past full APPS data","keywords":["code generation","test generation","execution feedback","data augmentation","rejection sampling","dual critic scoring","APPS dataset","self-training"],"falsifier":"Take the code solutions that pass the augmented test suite but fail the official APPS test labels; if this false-positive rate is high, the pass@1 gain on APPS+ would reflect overfitting to the augmented tests rather than genuine improvement. A direct measurement of that rate on a held-out subset of APPS problems would settle whether the augmentation loop is the cause of the reported gains.","tokens_in":10952,"feed_emoji":"🧪","tokens_out":4903,"duration_ms":38601,"temperature":0.7,"pith_summary":"The paper proposes GenX, a framework that trains code generation and test generation models together, using execution results to augment both. Starting from a small seed dataset with one solution per problem and few or no tests, the framework iteratively generates test inputs, executes them on the ground-truth solution to get correct outputs, then uses the growing test set to filter newly generated code solutions, keeping only those that pass. After three such iterations, models trained on the augmented data achieve higher code pass@1 on the APPS dataset than models trained on the original, larger APPS data, and higher test pass num. The paper also introduces a dual-critic scoring function that ranks generated code and tests against each other at inference time, with no ground truth needed.","feed_headline":"Self-made tests and code push pass@1 past full APPS data","feed_subtitle":"Starting from one solution and almost no tests, iterative execution checkpoints beat training on the original set.","key_machinery":"The load-bearing mechanism is the two-stage augmentation loop (Algorithm 1) combined with the dual-critic scoring function (Algorithm 2). In stage one, a test generation model samples test inputs, which are executed on the single ground-truth solution to obtain correct outputs, so model errors in predicting outputs are corrected by execution. In stage two, a code generation model samples many candidate solutions, and rejection sampling keeps only those that pass the accumulated test suite. The scoring function converts the pass/fail matrix between generated code and generated tests into normalized scores by iteratively updating each code's score as the share of test scores it passes and each test's score as the share of code scores it passes; this is used at inference to rank candidates without any ground truth.","core_discovery":"The central discovery is that iterative data augmentation driven by execution feedback can improve both code and test generation. With only the shortest correct solution per problem and almost no initial tests (APPS-), the framework synthesizes new test cases by running generated inputs through that solution, then rejects incorrect code samples by running them against the growing test suite. The resulting dataset (APPS+) yields a code generation model whose pass@1 exceeds the model trained on the full APPS dataset (23.40 vs 21.90 on introductory problems, 2.20 vs 1.50 on interview, 1.10 vs 0.30 on competition), and a test generation model with higher pass num. The paper claims the augmentation loop, not the original data, is what drives the improvement.","pith_inferences":["The dual-critic scoring is a mutual-reinforcement scheme akin to HITS or eigenvector centrality on the bipartite code–test graph; the paper does not analyze its convergence conditions or whether the final scores correspond to a principled notion of quality, but the reported results suggest a testable ranking hypothesis.","Whether the augmented tests are representative of the hidden distribution is the main open question; a direct comparison of augmented-test pass status against official APPS test labels on held-out problems would validate the filtering premise.","The approach's reliance on a single ground-truth solution for test-output correction is a latent risk: if that solution is itself incorrect on some inputs, the 'corrected' tests would codify wrong outputs; using multiple seed solutions or majority-vote outputs could mitigate this, but the paper keeps a single solution.","Because the paper leaves text-rich error messages as future work, the rejection-sampling loop could plausibly be strengthened by feeding runtime error text back into the code model rather than discarding failed samples."],"forward_implications":["Models trained on GenX-augmented data beat models trained on the full APPS dataset on code pass@1 across all three difficulty levels, so a small seed set plus execution feedback can substitute for large annotated training data.","Test generation improves substantially in pass num on APPS+, suggesting the augmentation loop produces more diverse, valid tests, not just more code.","The dual-critic scoring function outperforms the CodeT consensus-set scoring for ranking both code and tests, indicating that normalized mutual pass rates are a better inference-time selection signal.","The gains transfer to HumanEval and MBPP, where the APPS+-trained model improves pass@1 over the APPS--trained model."],"supporting_citations":[{"why":"Supplies the APPS benchmark and its annotated test cases, which the paper uses as seed data, evaluation, and baseline.","marker":"Hendrycks et al. (2021)"},{"why":"Defines CodeT, the dual-execution consensus scoring baseline the paper compares against for ranked code and test selection.","marker":"Chen et al. (2022)"},{"why":"AlphaCode's use of example tests for massive filtering and test mutation motivates the rejection-sampling and test-generation strategy.","marker":"Li et al. (2022)"},{"why":"Provides the DeepSeek Coder 5.7b base model that is fine-tuned in all experiments.","marker":"Guo et al. (2024)"},{"why":"Originates the self-training idea the augmentation loop extends to code and test generation.","marker":"Scudder (1965)"},{"why":"Self-Instruct demonstrates LLM-generated data can improve training, the general premise for data augmentation.","marker":"Wang et al. (2022)"}],"fun_headline_variants":["Execution feedback beats full data in code/test generation","Self-made tests boost code generation pass@1","Iterative execution checkpoints outperform original APPS data","Code and test generation improve via self-augmentation","From one solution, execution loop beats full dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that running generated inputs on the one retained ground-truth solution produces test cases that are representative enough of the hidden test distribution to filter generated code without letting incorrect solutions through.","fun_headline_variants_meta":{"raw":{"variants":["Execution feedback beats full data in code/test generation","Self-made tests boost code generation pass@1","Iterative execution checkpoints outperform original APPS data","Code and test generation improve via self-augmentation","From one solution, execution loop beats full dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1342,"prompt_tokens":837,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":453,"tokens_out":505,"duration_ms":4513,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:06:10.202369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the code solutions that pass the augmented test suite but fail the official APPS test labels; if this false-positive rate is high, the pass@1 gain on APPS+ would reflect overfitting to the augmented tests rather than genuine improvement. A direct measurement of that rate on a held-out subset of APPS problems would settle whether the augmentation loop is the cause of the reported gains.","supporting_citations":[],"review_version":1}