Pith. sign in

REVIEW 3 major objections 4 minor 44 references

Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repeatedly fine-tuning a code model on its own interpreter-verified outputs beats standard fine-tuning, and a repair-training variant improves one-shot generation on small benchmarks.

desk verdict A transparent thesis that does real service on APPS dataset quality but overstates the bootstrapping result due to a selection confound it itself names. read the letter →

arxiv 2507.15889 v1 pith:5NWL7WUN submitted 2025-07-20 cs.SE cs.AI

Noah van der Vleuten This is my paper
classification cs.SEcs.AI
keywords programsynthesisbootstrappingself-trainingrepairinterpreterfeedbackpass@kCodeT5APPSdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that bootstrapping a program synthesis model on its own interpreter-verified outputs consistently beats regular fine-tuning, and that the same loop can teach the model to repair. Each round, the model generates candidate solutions, an interpreter checks them against unit tests, the passing solutions are kept, failed solutions are optionally sent through a repair pass with interpreter feedback, and the original pre-trained model is fine-tuned again on the accumulated data. On the 770M CodeT5 model, bootstrapping beats regular fine-tuning on both MBPP and APPS, and on APPS it matches fine-tuned models roughly 68 percent larger in parameter count. On MBPP, the author reports that training with repairs improves non-repairing greedy pass@1 by 16.3 percent over plain bootstrapping, although on APPS the repair-trained models lose to plain bootstrapping at first-attempt accuracy. The paper also reports that the example test cases in the APPS training split overlap heavily with the hidden tests, with exact matches in 89 of 1,951 problems, and that ground-truth solutions pass only 61.26 percent of these example tests, which it argues contaminates any repair or reinforcement method that relies on them.

What carries the argument

The load-bearing mechanism is the bootstrapping loop of Algorithm 1, an adaptation of the STaR self-training idea to code. Over up to 9 rounds, the current model generates one solution per training problem; an interpreter runs the code against unit tests and is the only teacher. Correct generated code is kept, and for failed solutions the model either records the ground-truth answer (plain bootstrapping) or receives its own previous code plus a feedback message and tries to repair it, with the repaired code kept if it passes and the ground truth used otherwise. Crucially, every round fine-tunes the original pre-trained model, not the previous round's checkpoint, so each iteration is a fresh supervised fit on an enriched dataset rather than a continuation. The feedback comes in two strengths: simple ("The code above is wrong. Please fix it.") and full interpreter output (error type, expected versus produced output), which lets the paper separate the effect of information content from the effect of a second chance. The experiments use the 770M encoder-decoder model codet5-large-ntp-py, whose 512-token context the paper argues is partly responsible for the repair models' poor first-attempt results on the long-prompt APPS dataset.

What would settle it

Train regular fine-tuning nine times with the same hyperparameters, pick the run with the best validation pass@1, and compare its test pass@1 with the best bootstrapped run: if the scores are statistically indistinguishable, the bootstrapping loop contributes nothing beyond checkpoint selection.

Watch

Extended reading notes

Core claim

The central discovery claim is that a model can teach itself to program better by being repeatedly fine-tuned on its own interpreter-verified output, and that this bootstrapping is better than standard supervised fine-tuning on programming-competition benchmarks. In the author's words, bootstrapping consistently outperforms regular fine-tuning. The repairing variant goes further: when a generated solution fails the hidden tests, the model is asked to repair it, with either a bare "fix it" message or full interpreter feedback, and the successful repairs enter the training set. On MBPP this repair training improves even the non-repairing, first-attempt performance of the model compared with plain bootstrapping; on APPS it instead weakens first-attempt pass@1, which the author attributes to the 512-token context of CodeT5 truncating task descriptions in repair prompts. The author also claims that a 770M bootstrapped model performs on par with fine-tuned models roughly 68 percent larger, and that repairing during inference is likely inferior to sampling the same number of fresh solutions, since the sequential repair pass is slower and does not beat parallel sampling.

Load-bearing premise

The claim that bootstrapping beats regular fine-tuning rests on an unequal comparison: the best of nine bootstrapped training runs is picked by validation score while the regular fine-tuning model is trained only once, so the gap could come from getting several rolls of the random training dice rather than from the self-training data itself.

Editorial extensions

If this is right

  • If the central claim holds, any pretrained code model can be pushed past its standard fine-tuning level on a fixed dataset without new human-written examples, because interpreter feedback replaces annotation.
  • Bootstrapping with repair training is a way to improve one-shot synthesis even when no repairing is used at inference, at least on small, clean benchmarks like MBPP.
  • For practice, the paper's results imply that at inference time parallel temperature sampling of 2k solutions is the safer default than sequential repair, since edit pass@k rarely beats pass@2k.
  • Small models become more competitive: a 770M model bootstrapped this way reached APPS sampled pass@10 on par with 1.3B fine-tuned models in the comparison, so model size is not the only lever for coding ability.
  • The APPS contamination finding means training-time repair or reinforcement methods that use example tests as reward signals should filter those tests, and published results built on them may need re-examination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading is that the main confound left open is selection: nine bootstrapped training runs are screened by validation score against a single regular fine-tune. If matched compute reproduces the gap, the self-training data is the cause; if not, the paper's method is effectively a checkpoint-selection heuristic, which the author flags in the limitations.
  • The MBPP-versus-APPS divergence suggests a concrete causal story worth testing: repair bootstrapping helps when the repair prompt still contains the full task and hurts when context truncation removes it, so a long-context model should convert the APPS result.
  • The APPS overlap findings imply that a quantitative overlap audit of the test split is a natural follow-up; the paper inspects only a few examples qualitatively, and that audit would tell the community which repair-based results remain trustworthy.
  • Beyond competition problems, the generate-verify-repair-retain loop transfers naturally to settings with any executable signal, such as build logs, pull-request test suites, or runtime assertions, where self-training on repaired outputs could replace hand-labeled bug-fix data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a bootstrapping algorithm for program synthesis in which a pre-trained CodeT5 model is repeatedly fine-tuned on its own correct outputs together with ground-truth solutions for failures, with variants that additionally train the model to repair its own incorrect code using simple or interpreter feedback. The method is evaluated on MBPP and APPS and compared with regular fine-tuning, with pass@k as the metric. The paper claims that bootstrapping consistently outperforms regular fine-tuning, that bootstrapping with repairing improves non-repairing performance relative to plain bootstrapping, and that repairing during inference is generally not better than sampling additional solutions. A separate contribution is an analysis showing that example test cases in the APPS training set overlap with hidden tests, which the authors argue can cause overfitting in bootstrapping and related methods.

Significance. If the central claim were established, the paper would demonstrate a simple self-training loop with interpreter feedback can improve small program-synthesis models without additional human-annotated data, which is a practically valuable result. The APPS dataset-quality finding is a solid and independent contribution, as many repair and reinforcement-learning methods rely on the affected example test cases. The paper is transparent about many limitations and releases its code, which supports reproducibility. However, the main empirical claim is not yet supported by the evidence as presented: the key comparison between bootstrapping and regular fine-tuning is confounded by unequal checkpoint-selection effort, and some abstract claims are contradicted by the paper's own tables. These issues affect the headline conclusion and require additional experiments or substantially weakened claims.

major comments (3)
  1. [Section 5.2.1 / 5.4.5, Tables 5.3 and 5.5] The central comparison between bootstrapping and regular fine-tuning is confounded by unequal selection effort. Section 5.2.1 states that for bootstrapping objectives 'the best bootstrapped model is selected (from the 9 bootstrapping models trained per run, the best is selected)' by validation performance, whereas regular fine-tuning contributes a single model per run. The reported differences, e.g., APPS greedy pass@1 of 0.74 vs 0.63 in Table 5.5, could plausibly arise from selecting the best of nine random initializations or validation peaks, and the paper's own Section 5.4.5 explicitly names random initialization as a possible explanation and proposes the control of fine-tuning ten regular models and selecting the best. That control was not run. An equal-selection or equal-compute comparison is necessary before the claim that bootstrapping consistently outperforms regular fine-tuning can be accepted.
  2. [Abstract, Section 5.3.2, Table 5.5] The abstract claim that 'bootstrapping consistently outperforms regular fine-tuning' is contradicted by the paper's own APPS results for the full-feedback bootstrapping variant: Table 5.5 reports greedy pass@1 of 0.41 ± 0.06 for full-feedback bootstrapping versus 0.63 ± 0.08 for regular fine-tuning. Even for plain bootstrapping, the MBPP temperature-sampling results in Table 5.4 show pass@10 of 33.27 ± 0.57 for plain bootstrapping versus 33.53 ± 1.20 for regular fine-tuning, so the word 'consistently' does not hold across the reported sampling regimes. The claim should be qualified to the specific bootstrapping variant, dataset, and decoding setting that actually support it.
  3. [Abstract, Tables 5.3 and 5.5] The statement that 'bootstrapping with repairing also improves non-repairing performance compared to regular bootstrapping during inference' is only observed on MBPP, not on APPS. On MBPP (Table 5.3), full-feedback bootstrapping has Full Pass@1 of 18.40 ± 0.28 versus 14.67 ± 1.33 for plain bootstrapping, which supports the statement. On APPS (Table 5.5), however, full-feedback bootstrapping has non-repairing greedy pass@1 of 0.41 ± 0.06 versus 0.74 ± 0.03 for plain bootstrapping, which is the opposite. The claim should be restricted to MBPP or otherwise qualified to reflect the dataset dependence.
minor comments (4)
  1. [Table 5.3] The column labels 'Simple' and 'Full' are used both for training objectives and for few-shot prompt conditions, which makes the table difficult to interpret; the two axes should be renamed or separated more clearly.
  2. [Section 3.3.2] The sentence 'in 89/1951 problems there is an exact match of the example and hidden unit tests' should state what 1951 refers to, such as the number of APPS training problems with example tests, and should report how many training problems were considered in total.
  3. [Section 4.3.4] There is a typo in the training-case bullets: 'The modelrepairs the code correctly' should read 'The model repairs the code correctly'.
  4. [Section 5.3.2 / Table 5.7] The comparison with models '68% larger' mixes model family, training data, and sampling protocol; the claim in the abstract and in contribution (2) should state these caveats explicitly rather than presenting the comparison as an equivalence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the central bootstrapping claims are empirical comparisons against external benchmarks, and the acknowledged initialization-selection confound is an experimental-design issue rather than a derivation that reduces to its inputs.

full rationale

The paper's bootstrapping algorithm (Algorithm 1) is a self-training procedure: the model generates candidate solutions, an interpreter checks them, correct samples are added to the fine-tuning set, and ground-truth solutions are used for failed attempts. This is self-training, not circularity, because the target claim (bootstrapping outperforms regular fine-tuning) is not an input to the procedure; it is evaluated on held-out MBPP and APPS test problems using standard pass@k metrics. No fitted parameter is renamed as a prediction, and no load-bearing conclusion is justified by a self-citation. The comparisons to CodeRL and Self-Edit are external related work, not argumentative premises. The paper itself flags the main threat in Section 5.4.5: 'Bootstrapping as a method could simply be making use of random variations in the training initialization,' and notes that the control of fine-tuning ten regular models and selecting the best was not run. That is a legitimate soundness concern about equal selection effort and statistical attribution, but it does not make the derivation circular, because the observed test scores are not constructed from the conclusion. The APPS example-test overlap findings are likewise independent empirical observations. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method uses no new physical or mathematical entities. It relies on standard assumptions about benchmark correctness and the validity of bootstrapping. The free parameters are hyperparameters chosen by hand or borrowed from prior work, not fitted to maximize the reported metrics in a way that would make the claims circular.

free parameters (4)
  • Number of bootstrapping steps N = 9
    Chosen by the author; performance peaks at different steps per run, and the best step is selected on validation.
  • Sampling temperature = 0.8
    Used for test-time sampling; chosen to match Zhang et al. (2023), not optimized for this method.
  • Number of samples n for pass@k = 10
    Limited by compute; may under-estimate pass@k compared to larger n in prior work.
  • Learning rate = 5e-5
    A single value used across all runs; the author notes it may be poorly chosen for some objectives.
assumptions (3)
  • domain assumption Hidden test cases correctly judge program correctness during bootstrapping and evaluation.
    Used throughout to determine which generated programs are correct and should be reinforced.
  • ad hoc to paper Example test cases in the APPS training set substantially overlap with hidden tests, justifying their removal from prompts.
    Section 3.3.2 claims an overlap and uses it to exclude example tests during training, which affects the repair model's training signal.
  • domain assumption Fine-tuning the pre-trained CodeT5 model repeatedly on bootstrapped data is a valid way to improve the model.
    The entire method relies on the assumption that self-generated correct outputs and repaired outputs provide useful training signal beyond ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing." pith.science (2026). https://pith.science/paper/5NWL7WUN

@misc{pith2026250715889,
  author       = {Pith},
  title        = {Pith review of: Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NWL7WUN}},
  note         = {Machine review of arXiv:2507.15889}
}
read the original abstract

Language models for program synthesis are usually trained and evaluated on programming competition datasets (MBPP, APPS). However, these datasets are limited in size and quality, while these language models are extremely data hungry. Additionally, the language models have a misaligned program synthesis process compared to humans. While humans iteratively develop code with the help of a compiler, most program synthesis models currently produce code in one go. To solve these issues, we introduce a bootstrapping algorithm for program synthesis, that supports teaching models how to repair. We show that bootstrapping consistently outperforms regular fine-tuning. Compared to other work, our bootstrapped model performs on par with fine-tuned models that are 68\% larger. Notably, bootstrapping with repairing also improves non-repairing performance compared to regular bootstrapping during inference. However, on our models, repairing during inference is likely inferior to simply sampling the same number of solutions. Furthermore, we find that there are issues with the example test cases in the training portion of the APPS dataset that are valuable to the community, as many repairing and reinforcement learning methods rely on them.

Figures

Figures reproduced from arXiv: 2507.15889 by the authors.

Figure 2.1
Figure 2.1. Table taken from Gunasekar et al. (2023) They use self-reported scores whenever available. Despite being trained at vastly smaller scale, phi-1 outperforms competing models on HumanEval and MBPP, except for GPT-4 (also WizardCoder obtains better HumanEval but worse MBPP) Gunasekar et al. (2023). is deemed educationally valuable, they call this: textbook quality (6B tokens). Furthermore, GPT-3.5 is used to generate s… view at source ↗
Figure 3.1
Figure 3.1. Formatted MBPP tasks from the training dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Example of a large APPS task description, taken from the train dataset. The [PITH_FULL_IMAGE:figures/full_fig_p017_3_2.png] view at source ↗
Figures from the paper (9 more)
Figure 3.3
Figure 3.3. Figure 3.3: t-SNE plots generated by entering the formatted APPS train tasks in the CodeT5 [PITH_FULL_IMAGE:figures/full_fig_p018_3_3.png]
Figure 3.4
Figure 3.4. Figure 3.4: Boxplots of the token length of each prompt in the APPS train and test dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_3_4.png]
Figure 3.5
Figure 3.5. Figure 3.5: Formatted APPS tasks from the training dataset with the example test cases marked [PITH_FULL_IMAGE:figures/full_fig_p019_3_5.png]
Figure 4.1
Figure 4.1. Figure 4.1: A visual overview of the bootstrapping algorithm (for repairing). During inference [PITH_FULL_IMAGE:figures/full_fig_p021_4_1.png]
Figure 5.1
Figure 5.1. Figure 5.1: Results of several experiments conducted while validating. The samples come from [PITH_FULL_IMAGE:figures/full_fig_p033_5_1.png]
Figure 5.2
Figure 5.2. Figure 5.2: Results of Baseline 2 (beam search) compared to the repairing models while validat [PITH_FULL_IMAGE:figures/full_fig_p034_5_2.png]
Figure 5.3
Figure 5.3. Figure 5.3: Results of several experiments conducted while validating. The samples come from [PITH_FULL_IMAGE:figures/full_fig_p035_5_3.png]
Figure 5.4
Figure 5.4. Figure 5.4: Baseline 3 of the plain model and pass@1 and repairing performance of the full [PITH_FULL_IMAGE:figures/full_fig_p035_5_4.png]
Figure 5.5
Figure 5.5. Figure 5.5: Sankey diagrams of repairing performance on the MBPP test set while sampling, [PITH_FULL_IMAGE:figures/full_fig_p037_5_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 8 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. http://arxiv.org/abs/2108.07732 Program Synthesis with Large Language Models . ArXiv:2108.07732 [cs]. http://arxiv.org/abs/2108.07732 http://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [4]

    Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DEEPCODER : LEARNING TO WRITE PROGRAMS page 20

    work page 2017

  5. [5]

    R. Balzer. 1985. https://doi.org/10.1109/TSE.1985.231877 A 15 Year Perspective on Automatic Programming . IEEE Transactions on Software Engineering\/ SE-11(11):1257--1268. Conference Name: IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.1985.231877 https://doi.org/10.1109/TSE.1985.231877

    work page arXiv 1985

  6. [6]

    Biermann

    Alan W. Biermann. 1985. https://doi.org/10.1016/S0747-7171(85)80010-9 Automatic programming : A tutorial on formal methodologies . Journal of Symbolic Computation\/ 1(2):119--142. https://doi.org/10.1016/S0747-7171(85)80010-9 https://doi.org/10.1016/S0747-7171(85)80010-9

    work page doi:10.1016/s0747-7171(85)80010-9 1985

  7. [7]

    James Bornholt. 2015. https://www.cs.utexas.edu/ bornholt/post/synthesis-explained.html Program Synthesis Explained . https://www.cs.utexas.edu/ bornholt/post/synthesis-explained.html https://www.cs.utexas.edu/ bornholt/post/synthesis-explained.html

    work page 2015

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

Show all 44 references
  1. [9]

    Bowman, Kyunghyun Cho, and Ethan Perez

    Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. 2023 a . http://arxiv.org/abs/2303.16749 Improving Code Generation by Training with Natural Language Feedback . ArXiv:2303.16749 [cs]. http://arxi...

    2023 arXiv

  2. [10]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. http://arxiv.org/abs/2207.10397 CodeT : Code Generation with Generated Tests . ArXiv:2207.10397 [cs] version: 2. http://arxiv.org/abs/2207.10397 http://arxiv.org/abs/2207.10397

    2022 arXiv

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 doi arXiv

  4. [12]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023 b . http://arxiv.org/abs/2304.05128 Teaching Large Language Models to Self - Debug . ArXiv:2304.05128 [cs]. http://arxiv.org/abs/2304.05128 http://arxiv.org/abs/2304.05128

    2023 arXiv

  5. [13]

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. http://arxiv.org/abs/2306.05284 Simple and Controllable Music Generation . ArXiv:2306.05284 [cs, eess]. http://arxiv.org/abs/2306.05284 http://arxiv.org/abs/2306.05284

    2023 arXiv

  6. [14]

    Jeffrey Dastin. 2023. https://www.reuters.com/technology/microsoft-attracting-users-its-code-writing-generative-ai-software-2023-01-25/ Microsoft attracting users to its code-writing, generative AI software . Reuters\/ https://www.reuters.com/technology/microsoft-attracting-us...

    2023

  7. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.48550/arXiv.1810.04805 BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . ArXiv:1810.04805 [cs]. https://doi.org/10.48550/arXiv.1810.04805 https://doi.o...

    2019 doi arXiv

  8. [16]

    Tenenbaum

    Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2020. https://doi.org/10.48550/arXiv.2006.08381 DreamCoder : Growing generalizable, interpretable knowledge with wake-sleep Bayes...

    2020 doi arXiv

  9. [17]

    Goldberg

    Allen T. Goldberg. 1986. https://doi.org/10.1109/TSE.1986.6312977 Knowledge-based programming: A survey of program design and construction techniques . IEEE Transactions on Software Engineering\/ SE-12(7):752--768. Conference Name: IEEE Transactions on Software Engineering. ht...

    1986

  10. [18]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Y...

    2023 arXiv

  11. [19]

    Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2022. https://doi.org/10.48550/arXiv.2207.14502 Language Models Can Teach Themselves to Program Better . ArXiv:2207.14502 [cs]. https://doi.org/10.48550/arXiv.2207.14502 https://doi.org/10.48550/arXiv.2207.14502

    2022 doi arXiv

  12. [20]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. http://arxiv.org/abs/2105.09938 Measuring Coding Challenge Competence With APPS . ArXiv:2105.09938 [cs] version:...

    2021 arXiv

  13. [21]

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. https://proceedings.neurips.cc/paper/2019/hash/7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html SPoC : Search -based Pseudocode to Code . In Advances in Neural Information ...

    2019

  14. [22]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. http://arxiv.org/abs/2207.01780 CodeRL : Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning . ArXiv:2207.01780 [cs]. http://arxiv.org/abs/2207.01780 http:/...

    2022 arXiv

  15. [23]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  16. [24]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    2022 doi

  17. [25]

    Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. 2023. http://arxiv.org/abs/2307.04349 RLTF : Reinforcement Learning from Unit Test Feedback . ArXiv:2307.04349 [cs]. http://arxiv.org/abs/2307.04349 http://arxiv.org/abs/2307.04349

    2023 arXiv

  18. [26]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. http://arxiv.org/abs/2306.08568 WizardCoder : Empowering Code Large Language Models with Evol - Instruct . ArXiv:2306.08568 [cs]. http://arxiv.org/ab...

    2023 arXiv

  19. [27]

    Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023. http://arxiv.org/abs/2305.02309 CodeGen2 : Lessons for Training LLMs on Programming and Natural Languages . ArXiv:2305.02309 [cs]. http://arxiv.org/abs/2305.02309 http://arxiv.org/abs/2305.02309

    2023 arXiv

  20. [28]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. http://arxiv.org/abs/2112.00114 Show Your Work : Scratchpads for Intermediate C...

    2021 arXiv

  21. [29]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. http://arxiv.org/abs/2306.09896 Demystifying GPT Self - Repair for Code Generation . ArXiv:2306.09896 [cs]. http://arxiv.org/abs/2306.09896 http://arxiv.org/abs/2306.09896

    2023 arXiv

  22. [30]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre - Training

    2018

  23. [31]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. http://arxiv.org/abs/2204.06125 Hierarchical Text - Conditional Image Generation with CLIP Latents . ArXiv:2204.06125 [cs]. http://arxiv.org/abs/2204.06125 http://arxiv.org/abs/2204.06125

    2022 arXiv

  24. [32]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022. http://arxiv.org/abs/2205.11487 Pho...

    2022 arXiv

  25. [33]

    Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. 2021. https://doi.org/10.48550/arXiv.2106.05784 Programming Puzzles . ArXiv:2106.05784 [cs]. https://doi.org/10.48550/arXiv.2106.05784 https://doi.org/10.48550/arXiv.2106.05784

    2021 doi arXiv

  26. [34]

    Noam Shazeer. 2019. http://arxiv.org/abs/1911.02150 Fast Transformer Decoding : One Write - Head is All You Need . ArXiv:1911.02150 [cs]. http://arxiv.org/abs/1911.02150 http://arxiv.org/abs/1911.02150

    2019 arXiv

  27. [35]

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. https://doi.org/10.48550/arXiv.2301.13816 Execution-based Code Generation using Deep Reinforcement Learning . ArXiv:2301.13816 [cs]. https://doi.org/10.48550/arXiv.2301.13816 https://doi.org/10.48550/a...

    2023 doi arXiv

  28. [36]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. http://arxiv.org/abs/1706.03762 Attention Is All You Need . ArXiv:1706.03762 [cs]. http://arxiv.org/abs/1706.03762 http://arxiv.org/abs/1706.03762

    2017 arXiv

  29. [37]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. http://arxiv.org/abs/2305.07922 CodeT5 +: Open Code Large Language Models for Code Understanding and Generation . ArXiv:2305.07922 [cs]. http://arxiv.org/abs/2305.07922 http://ar...

    2023 arXiv

  30. [38]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. https://doi.org/10.48550/arXiv.2109.00859 CodeT5 : Identifier -aware Unified Pre -trained Encoder - Decoder Models for Code Understanding and Generation . ArXiv:2109.00859 [cs]. https://doi.org/10.48550/arXiv.2109...

    2021 doi arXiv

  31. [39]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. http://arxiv.org/abs/2109.01652 Finetuned Language Models Are Zero - Shot Learners . ArXiv:2109.01652 [cs]. http://arxiv.org/abs/2109.01652 http://ar...

    2022 arXiv

  32. [40]

    Weijia Xu, Andrzej Banburski-Fahey, and Nebojsa Jojic. 2023. http://arxiv.org/abs/2305.09993 Reprompting: Automated Chain -of- Thought Prompt Inference Through Gibbs Sampling . ArXiv:2305.09993 [cs]. http://arxiv.org/abs/2305.09993 http://arxiv.org/abs/2305.09993

    2023 arXiv

  33. [41]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. http://arxiv.org/abs/2203.14465 STaR : Bootstrapping Reasoning With Reasoning . ArXiv:2203.14465 [cs]. http://arxiv.org/abs/2203.14465 http://arxiv.org/abs/2203.14465

    2022 arXiv

  34. [42]

    Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. http://arxiv.org/abs/2305.04087 Self- Edit : Fault - Aware Code Editor for Code Generation . ArXiv:2305.04087 [cs]. http://arxiv.org/abs/2305.04087 http://arxiv.org/abs/2305.04087

    2023 arXiv

  35. [43]

    Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I

    Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I. Wang. 2022. http://arxiv.org/abs/2211.16490 Coder Reviewer Reranking for Code Generation . ArXiv:2211.16490 [cs] version: 1. http://arxiv.org/abs/2211.16490 http://arxiv.org/abs/2211.16490

    2022 arXiv

  36. [44]

    Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian

    Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion https://doi.org/10.1145/3520312.3534864. In Proceedings of the 6th ACM SIGPLAN Inter...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.