Pith. sign in

REVIEW 5 major objections 6 minor 15 references

GenX: Mastering Code and Test Generation with Execution Feedback

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-made tests and code push pass@1 past full APPS data

desk verdict A plausible execution-feedback loop, but the evaluation never checks whether the augmented tests actually predict official test outcomes, and the reported gains are small and possibly noise. read the letter →

arxiv 2412.13464 v1 pith:T3IEQNYJ submitted 2024-12-18 cs.SE cs.CL

Nan Wang , Yafei Liu , Chen Chen , Haonan Lu This is my paper · ORCID
classification cs.SEcs.CL
keywords codegenerationtestexecutionfeedbackdataaugmentationrejectionsamplingdualcriticscoringAPPSdatasetself-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GenX, a framework that trains code generation and test generation models together, using execution results to augment both. Starting from a small seed dataset with one solution per problem and few or no tests, the framework iteratively generates test inputs, executes them on the ground-truth solution to get correct outputs, then uses the growing test set to filter newly generated code solutions, keeping only those that pass. After three such iterations, models trained on the augmented data achieve higher code pass@1 on the APPS dataset than models trained on the original, larger APPS data, and higher test pass num. The paper also introduces a dual-critic scoring function that ranks generated code and tests against each other at inference time, with no ground truth needed.

What carries the argument

The load-bearing mechanism is the two-stage augmentation loop (Algorithm 1) combined with the dual-critic scoring function (Algorithm 2). In stage one, a test generation model samples test inputs, which are executed on the single ground-truth solution to obtain correct outputs, so model errors in predicting outputs are corrected by execution. In stage two, a code generation model samples many candidate solutions, and rejection sampling keeps only those that pass the accumulated test suite. The scoring function converts the pass/fail matrix between generated code and generated tests into normalized scores by iteratively updating each code's score as the share of test scores it passes and each test's score as the share of code scores it passes; this is used at inference to rank candidates without any ground truth.

What would settle it

Take the code solutions that pass the augmented test suite but fail the official APPS test labels; if this false-positive rate is high, the pass@1 gain on APPS+ would reflect overfitting to the augmented tests rather than genuine improvement. A direct measurement of that rate on a held-out subset of APPS problems would settle whether the augmentation loop is the cause of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery is that iterative data augmentation driven by execution feedback can improve both code and test generation. With only the shortest correct solution per problem and almost no initial tests (APPS-), the framework synthesizes new test cases by running generated inputs through that solution, then rejects incorrect code samples by running them against the growing test suite. The resulting dataset (APPS+) yields a code generation model whose pass@1 exceeds the model trained on the full APPS dataset (23.40 vs 21.90 on introductory problems, 2.20 vs 1.50 on interview, 1.10 vs 0.30 on competition), and a test generation model with higher pass num. The paper claims the augmentation loop, not the original data, is what drives the improvement.

Load-bearing premise

The pipeline assumes that running generated inputs on the one retained ground-truth solution produces test cases that are representative enough of the hidden test distribution to filter generated code without letting incorrect solutions through.

Editorial extensions

If this is right

  • Models trained on GenX-augmented data beat models trained on the full APPS dataset on code pass@1 across all three difficulty levels, so a small seed set plus execution feedback can substitute for large annotated training data.
  • Test generation improves substantially in pass num on APPS+, suggesting the augmentation loop produces more diverse, valid tests, not just more code.
  • The dual-critic scoring function outperforms the CodeT consensus-set scoring for ranking both code and tests, indicating that normalized mutual pass rates are a better inference-time selection signal.
  • The gains transfer to HumanEval and MBPP, where the APPS+-trained model improves pass@1 over the APPS--trained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dual-critic scoring is a mutual-reinforcement scheme akin to HITS or eigenvector centrality on the bipartite code–test graph; the paper does not analyze its convergence conditions or whether the final scores correspond to a principled notion of quality, but the reported results suggest a testable ranking hypothesis.
  • Whether the augmented tests are representative of the hidden distribution is the main open question; a direct comparison of augmented-test pass status against official APPS test labels on held-out problems would validate the filtering premise.
  • The approach's reliance on a single ground-truth solution for test-output correction is a latent risk: if that solution is itself incorrect on some inputs, the 'corrected' tests would codify wrong outputs; using multiple seed solutions or majority-vote outputs could mitigate this, but the paper keeps a single solution.
  • Because the paper leaves text-rich error messages as future work, the rejection-sampling loop could plausibly be strengthened by feeding runtime error text back into the code model rather than discarding failed samples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes GenX, a training and inference framework that jointly augments code and test generation data on the APPS dataset using execution feedback. Starting from a reduced dataset (APPS-), it iteratively generates test inputs with a fine-tuned test-generation model, executes them on the single retained ground-truth solution to obtain outputs, and then uses the resulting augmented test suite to filter generated code solutions via rejection sampling. This produces an augmented dataset (APPS+) on which code and test generation models are trained. At inference, a dual-critic scoring algorithm (Algorithm 2) ranks generated code and tests from a pass/fail matrix without ground truth. Experiments on APPS, HumanEval, and MBPP report that models trained on APPS+ achieve better pass@1 on APPS test cases than models trained on APPS- or the original APPS, and that the proposed scoring function outperforms CodeT on code and especially test ranking.

Significance. If the central claim is correct, the paper offers a practical way to escape the dependence of execution-feedback code generation on pre-existing, dense test suites: a model can bootstrap its own tests from a single ground-truth solution and use them to filter correct code. The framework is clearly described and the inclusion of test-generation metrics, a CodeT comparison, and additional HumanEval/MBPP experiments are strengths. The dual-critic scoring algorithm is simple and shows consistent ranking improvements. However, the main empirical claim rests on small pass@1 differences obtained on a subsampled APPS test set without error bars, and a key assumption—that augmented tests are representative enough to avoid false positives in rejection sampling—is not directly validated. The contribution is therefore promising but needs stronger evidence before the headline result can be accepted.

major comments (5)
  1. [§4.1, Table 3] The headline claim that APPS+ outperforms APPS relies on pass@1 differences of 1.5, 0.7, and 0.8 percentage points for intro, interview, and competition levels, each evaluated on 100 problems. No error bars, confidence intervals, or multiple-seed results are reported, so these differences may be within sampling noise. The abstract and §4.2 should either be qualified or supported by repeated runs with variance estimates.
  2. [§3.3–§3.4, Table 5] The rejection-sampling filter in §3.4 uses augmented tests that are generated by executing sampled inputs on the single ground-truth solution (§3.3). The assertion that this allows us to 'confidently judge the correctness of the newly generated code and avoid false positives' is not directly verified: the paper never checks whether code retained by augmented tests also passes the official APPS hidden tests. Table 5 only compares filtering with APPS+ tests versus APPS tests in terms of final pass@1; it does not measure the representativeness of the augmented tests. Please add an experiment that evaluates code retained by augmented tests on the official hidden tests, or report the pass rate of augmented tests against the official tests.
  3. [§4.1, Evaluation] The evaluation uses a 300-problem subset of APPS (100 per difficulty) after filtering out problems without valid solutions and keeping only the shortest solution for each problem. This is a modified benchmark, so the reported pass@k numbers are not directly comparable to prior APPS results and the generalizability of the central claim is limited. Please report results on the full APPS test set or provide a clear justification for the subset and its relationship to the official benchmark.
  4. [§4.1, n@k definition; Tables 3–4] The n@k metric is defined as 'selecting the top-n scored code solutions (which may exceed n) and then calculating their pass@k.' This is ambiguous: if the selected set can contain more than n solutions, the effective number of candidates differs from n, making the metric incomparable to standard pass@k. Please state exactly how n@k is computed, including how ties are handled, and ensure the CodeT comparison in Table 4 uses identical selection rules.
  5. [§3.3, Table 2] The pass num metric can be inflated by trivial or redundant tests (e.g., constant outputs). The paper prunes tests by length and notes that some outputs are 'too trivial' in §3.3, but no quantitative measure of augmented-test discriminative power is reported. Because the entire data-augmentation pipeline depends on test quality, please report the distribution of augmented test outputs, the fraction of non-trivial tests, or the pass rate of augmented tests against official hidden tests.
minor comments (6)
  1. [Abstract] The phrase 'concurrently trains a code generation model and a test generation model' is misleading because Algorithm 1 trains the test model and the code model sequentially (first M_t in the test-augmentation loop, then M_c in the code-augmentation loop). Please rephrase.
  2. [§3.3] The sentence 'To ensure diversity, we remove duplicate test cases' does not specify the deduplication criterion (exact input-output match, input only, or normalized JSON). Please clarify.
  3. [§3.5] The claim that Algorithm 2 'typically converges within a few hundred iterations' is supported only by simulation; please report the number of iterations n used in the experiments and state whether convergence was checked for all problems in the test set.
  4. [Table 2] The pass rate for APPS+ on competition problems (26.93%) is lower than for APPS- (27.50%), yet §4.2 states that APPS+ outperforms APPS- in both pass rate and pass num. This inconsistency should be discussed.
  5. [Figures 2 and 3] The y-axis label 'Score' is defined only in the caption as the product of pass rate and pass num; please label the axes directly and include units or a short explanation in the figure itself.
  6. [§5] The concluding limitation paragraph correctly acknowledges that text-rich error messages are not used; this is a useful statement of scope. There is a minor capitalization inconsistency at the start of the last paragraph ('Currently, We do not utilize...').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed gains are measured on held-out ground-truth tests and external benchmarks; no prediction reduces by construction to an input or to a self-citation chain.

full rationale

The derivation chain is self-contained and externally anchored. Test augmentation (Section 3.3, Algorithm 1 lines 8-9) labels model-generated inputs by executing them on the retained ground-truth solution; this is an external oracle, not the model's own prediction. Code augmentation (Section 3.4, Algorithm 1 lines 19-20) keeps only solutions that pass the accumulated tests, and the headline pass@1 numbers (Table 3) are computed on the held-out APPS- test set of 300 problems, supplemented by HumanEval/MBPP in Appendix A. The final evaluation targets were not used to filter or synthesize training data, so the reported improvements are not forced by construction. Algorithm 2's dual-critic scoring is applied only at inference for ranking and is benchmarked against the external CodeT baseline; it does not enter the training labels. There are no load-bearing self-citations or imported uniqueness theorems. The appendix itself warns that HumanEval/MBPP results may not align with official evaluations due to conversion errors, which limits the strength of that external support but does not create circularity. The concern that APPS+ tests are never validated against official APPS tests is a substantive validity risk: if augmented tests are weak, retained code may still fail hidden tests. That would depress or invalidate the comparison, but it is an empirical failure mode, not a circular reduction of the result to its own assumptions.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-set hyperparameters (sampling budgets, temperatures, iteration counts) and on the assumption that augmented tests provide sufficient coverage to filter code without false positives. No invented entities are introduced; the method combines existing concepts.

free parameters (5)
  • per-difficulty sampling budget for code rejection sampling = 40 intro, 80 interview, 160 competition
    Chosen by hand in Section 3.4; larger budget for harder problems. Affects how many correct solutions are found and thus the augmented dataset.
  • number of augmentation iterations = Nt = 3, Nc = 3
    Set to 3; Figure 3 shows performance can decline after later iterations, so this choice is load-bearing.
  • sampling temperatures = test 0.8, code 0.6, +0.2 for augmentation
    Chosen for inference versus augmentation; affects diversity and pass rate.
  • test length threshold = not specified
    Tests above a 'specific length threshold' are excluded from training the test model (Section 3.3); the threshold is not reported, so the filtering is not reproducible.
  • number of test samples per problem per iteration = up to 10 samplings, each up to 10 test cases
    Arbitrary sampling budget that controls the quantity of augmented data.
assumptions (4)
  • domain assumption Executing a generated test input on the ground-truth solution produces the correct test output.
    Section 3.3 uses this to correct test outputs; it assumes the ground-truth code is correct for all sampled inputs.
  • domain assumption Generated tests that pass the ground-truth code are valid for evaluating generated code (no false positives due to low coverage).
    Section 3.4 relies on this to filter code; Section 4.3 discusses false positives but does not rule out incompleteness.
  • ad hoc to paper The APPS- artificial distribution (shortest solution, at most 10 tests for 686 problems, none for others) is a representative test-scarce scenario.
    The paper creates APPS- to simulate scarcity; the validity of the conclusions depends on this construct.
  • standard math The iterative normalization in Algorithm 2 converges to a meaningful score ranking.
    Only 'simulated convergence' is reported; no proof of convergence or of score semantics is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GenX: Mastering Code and Test Generation with Execution Feedback." pith.science (2026). https://pith.science/paper/T3IEQNYJ

@misc{pith2026241213464,
  author       = {Pith},
  title        = {Pith review of: GenX: Mastering Code and Test Generation with Execution Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3IEQNYJ}},
  note         = {Machine review of arXiv:2412.13464}
}
read the original abstract

Recent advancements in language modeling have enabled the translation of natural language into code, and the use of execution feedback to improve code generation. However, these methods often rely heavily on pre-existing test cases, which may not always be available or comprehensive. In this work, we propose a novel approach that concurrently trains a code generation model and a test generation model, utilizing execution feedback to refine and enhance the performance of both. We introduce two strategies for test and code data augmentation and a new scoring function for code and test ranking. We experiment on the APPS dataset and demonstrate that our approach can effectively generate and augment test cases, filter and synthesize correct code solutions, and rank the quality of generated code and tests. The results demonstrate that our models, when iteratively trained with an increasing number of test cases and code solutions, outperform those trained on the original dataset.

Figures

Figures reproduced from arXiv: 2412.13464 by the authors.

Figure 1
Figure 1. The execution roles under different situations, where W represents code solution, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The experimental results of test generation and code generation when replaying [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The experimental results under three iterations and four different versions (v1 to [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [1]

    Leveraging grammar and reinforcement learning for neural program synthesis

    Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. arXiv preprint arXiv:1805.04276,

    arXiv

  2. [7]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196,

    arXiv

  3. [8]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186,

    arXiv

  4. [10]

    Rltf: Reinforcement learning from unit test feedback

    Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349,

    arXiv

  5. [12]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  6. [14]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024a. Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang...

    arXiv

  7. [15]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568,

    arXiv

  8. [2013]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J ´er´emy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,

    arXiv

Show all 15 references
  1. [2014]

    Automatic unit test data generation and actor-critic reinforcement learning for code synthesis

    Philip Gorinski, Matthieu Zimmer, Gerasimos Lampouras, Derrick Goh Xin Deik, and Ignacio Iacobacci. Automatic unit test data generation and actor-critic reinforcement learning for code synthesis. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 370–384,

    2023

  2. [2017]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  3. [2019]

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution-based code generation using deep reinforcement learning. ArXiv, abs/2301.13816, 2023a. Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation using dee...

    arXiv

  4. [2021]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Sch¨arli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128,

    arXiv

  5. [2022]

    Evalu- ating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  6. [2023]

    Textbooks are all you need

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C ´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644,

    arXiv

  7. [2024]

    Mapping language to code in programmatic context

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1643–1652,

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.