Learning to Generate Unit Tests for Automated Debugging

Archiki Prasad; Elias Stengel-Eskin; Justin Chih-Yao Chen; Mohit Bansal; Zaid Khan

REVIEW 3 major objections 4 minor 56 references

Learning to Generate Unit Tests for Automated Debugging

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trained test-writers improve LLM debugging by 15 points

desk verdict A solid, useful empirical paper on generating unit tests for debugging, but the abstract overstates one comparison and the training-data overlap with evaluation benchmarks is unexamined. read the letter →

arxiv 2502.01619 v3 pith:TNIOBM2G submitted 2025-02-03 cs.SE cs.AIcs.CLcs.LG

Archiki Prasad , Elias Stengel-Eskin , Justin Chih-Yao Chen , Zaid Khan , Mohit Bansal This is my paper · ORCID

classification cs.SEcs.AIcs.CLcs.LG

keywords unittestgenerationautomateddebuggingLLMcoderepairattackrateoutputaccuracytest-timecomputeself-consistency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unit tests are the standard way to check code, but writing them by hand is expensive, and LLMs usually need a gold solution to know the expected output. This paper claims that a relatively small fine-tuned model can generate unit tests that do both things at once: expose a bug in faulty code and predict the correct output, using only the problem description and the faulty code. The authors introduce UTGen, a training recipe that creates supervision data by intentionally corrupting correct code, and UTDebug, a debugging loop that applies test-time scaling and a validation-and-backtracking step to cope with noisy generated tests. If the claim holds, automated debugging and code selection no longer depend on human-written tests or gold implementations, and even frontier debugging models benefit from cheap, generated test feedback.

What carries the argument

The machinery is a three-stage data bootstrapping recipe (UTGen) plus a robust inference loop (UTDebug). UTGen starts from gold code solutions, (I) corrupts them into faulty variants, (II) samples unit test inputs that fail on the faulty code while using the gold code to supply the correct output, and (III) relabels each output prediction with a chain-of-thought rationale generated post hoc, producing roughly 30K-70K supervised instances that jointly reward attack rate (how often the test makes faulty code fail) and output accuracy (how often the predicted test output is correct). At inference, UTDebug samples k=8 output predictions per input, keeps the majority vote only if it exceeds 50% of votes, generates n unit tests, uses one failing test as feedback, and accepts a code edit only when the pass rate over the full test suite improves; otherwise it backtracks. The combination is what allows noisy, model-generated tests to serve as reliable feedback for debugging and code selection.

What would settle it

Run a decontamination scan of UTGen's training data against HumanEval and MBPP+ (for example, fuzzy matching problem descriptions or code tokens to benchmark solutions); if overlaps exist, retrain on the cleaned set and check whether the 7.59% Accuracy ∩ Attack gain and the debugging improvements persist. A large drop toward the zero-shot prompting baseline would show the reported gains came from memorization rather than learned unit-test generation; no overlap with stable numbers would support the generalization claim.

Watch

Extended reading notes

Core claim

UTGen teaches LLMs to generate unit test input-output pairs such that the input is valid for the task, the input reveals an error in the given faulty code, and the output matches what a correct solution would return. The paper's central discovery is a trade-off: with zero-shot prompting, tests that are likely to reveal bugs tend to have harder inputs whose outputs the model cannot reliably predict, while randomly sampled tests have more accurate outputs but rarely fail. Supervised fine-tuning on bootstrapped data, built from corrupted reference solutions and chain-of-thought rationales, moves models into a sweet spot where both properties hold; UTGen reports 7.59% higher Accuracy ∩ Attack than prompting baselines. On downstream tasks, feeding these tests through UTDebug's self-consistency and backtracking improves pass@1 debugging accuracy (the share of debugged programs passing all tests) for Qwen2.5 32B by up to 15.07% absolute over no-test feedback and by over 12.35% on the hardest MBPP+ split compared with other LLM test generators, and the same tests act as a better judge of code correctness than a specialized 8B reward model.

Load-bearing premise

The load-bearing premise is that HumanEval and MBPP+ problems used for evaluation never appear among the Tulu-3 training problems that UTGen is fine-tuned on; the paper filters Tulu-3 for Python functions but reports no decontamination check, so leaked benchmark solutions would inflate the measured test-generation and debugging gains.

Editorial extensions

If this is right

LLM debugging pipelines can run without human-written unit tests: a fine-tuned 7B-32B model supplies the feedback, removing the test-availability bottleneck.
Unit-test-based judging of code correctness beats a trained 8B reward model on best-of-10 sampling, offering an alternative verification signal for code generation.
Frontier debuggers such as GPT-4o improve substantially when given UTGen-generated tests, showing test quality matters even for very strong debuggers.
The debugging gains persist as the number of generated unit tests grows, and UTGen remains better than random test sampling when both are scaled up.
Because UTGen conditions on the faulty code rather than sampling inputs independently, its tests localize errors better than CodeT-style generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the same bootstrapping recipe—corrupt a reference, sample inputs that expose the corruption, relabel outputs with rationales—were applied to other programming languages or to non-functional properties such as performance or memory, the attack-rate/output-accuracy trade-off would likely re-emerge and training would be expected to resolve it there as well.
Once decontamination is confirmed, UTGen-style models could serve as a cheap 'test oracle' for reinforcement-learning loops that train code models on execution feedback, potentially replacing much of the cost of sampling gold solutions during training.
The trade-off may generalize beyond code: in any specification-to-behavior task, the hardest distinguishing examples are precisely the cases where current models are least able to predict the expected behavior, so the same training recipe may apply to data transformations or API usage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper is worth a look for anyone working on LLM code repair or test generation. The core idea is clean: train a model to generate unit tests that both reveal a bug and carry the correct expected output, then feed those tests into a debugging loop. The trade-off between attack rate and output accuracy is a genuinely useful framing.

What the paper does well: the UTGen recipe (perturb gold code, execute to filter failing tests, add rationales) is a practical way to bootstrap training data without needing gold tests at inference. The UTDebug pipeline, with self-consistency for output prediction and backtracking based on a test-suite pass rate, is sensible and ablated carefully. Results are consistent across three 7-8B models, a 32B model, and multiple benchmarks; the public artifact is a real plus. Comparing against CodeT and a trained reward model for best-of-N selection makes the contribution concrete.

The soft spots are real but manageable. First, the abstract says the 12.35% gain on MBPP+Fix (Hard) is "over other LLM-based UT generation baselines," but that number is versus the No-UT baseline; against the prompted baseline it is about 4.7%. The 7.59% intrinsic improvement is also not directly traceable to Table 1—per-model gains range from roughly 5% to 7%. These are presentational errors, not necessarily fatal, but the headline numbers should be fixed. The more substantive concern is the lack of any decontamination check. Tulu-3 is a broad SFT mixture, and HumanEval/MBPP+ are common in such mixtures; without an overlap analysis, the intrinsic and debugging numbers on those benchmarks could be optimistic. This is a missing robustness check, not evidence of dodgy behavior, but it needs to be addressed before I'd trust the claim of generalization.

Minor point: the "hard" split (keeping faulty code that passes 50-95% of tests) is reasonable but arbitrary; a sensitivity analysis on that threshold would strengthen it.

Audience: this is for researchers and engineers working on automated debugging, feedback generation, and unit-test synthesis. It is an empirical systems paper, not a theoretical breakthrough, and it claims appropriate scope.

Recommendation: yes, this deserves peer review—a serious referee can verify the reproducibility and push for the needed decontamination analysis. Fix the abstract numbers and add a training/eval overlap check, and this would be a solid contribution.

Referee Report

3 major / 4 minor

Summary. The paper proposes UTGen, a supervised fine-tuning recipe that teaches LLMs to generate unit-test input-output pairs conditioned on a task description and a buggy program, aiming for both high attack rate and correct expected outputs without access to a gold solution. Training data is bootstrapped from the Tulu-3 SFT mixture by corrupting gold code, sampling failing unit-test inputs, and adding post-hoc rationales whose outputs are grounded by executing the gold code. The authors then introduce UTDebug, a multi-turn debugging pipeline that uses self-consistency over UT outputs and pass-rate-based backtracking to handle noisy generated tests. Experiments cover HumanEvalFix, MBPP+Fix, and a new harder MBPP+ split; the paper reports improved debugging pass@1 across several 7-32B models, best-of-N code selection gains over a reward-model reranker, and improved GPT-4o debugging when UTGen-generated tests from a smaller model are used as feedback.

Significance. If the claims withstand scrutiny, this is a valuable contribution: automated, reference-free unit-test generation with error-revealing inputs and correct outputs is a real bottleneck for LLM debugging and verification. The paper's strengths include its public release of code and datasets, evaluation across multiple model families and datasets, the use of gold-code execution to ground training outputs, and a well-motivated intrinsic metric suite (attack rate, output accuracy, Accuracy∩Attack) whose downstream correlation is explicitly studied in Table 7. The UTDebug mechanisms—self-consistency for output prediction and validation/backtracking against a suite of generated tests—are sensible and are shown to be important in ablation. The consistent direction of results across models and datasets gives the central thesis initial plausibility. However, the headline numbers need correction, and the training-data overlap question must be resolved before the generalization claims can be accepted.

major comments (3)

[Abstract; §4; §5.2, Table 2] The abstract attributes a 12.35% improvement on MBPP+Fix (Hard) to "other LLM-based UT generation baselines," but Table 2 shows that for Qwen2.5 32B the differences are 4.70% over Prompted (40.59 to 45.29) and 22.35% over Random (22.94 to 45.29); 12.35% is the gain over the No-UT baseline (32.94 to 45.29). The abstract's 7.59% Accuracy∩Attack figure is also not directly traceable to any single model-baseline pair in Table 1 or Table 6; the closest text statement in §5.1 reports 7.65% for Llama3.1, which is the gain over Random, not Prompted. Please state the exact comparison being reported or correct the numbers.
[Appendix B; §5, Tables 1, 2, 5, 6] The UTGen training pipeline filters the Tulu-3 SFT mixture for Python functions but reports no decontamination check against HumanEval, MBPP+, or EvalPlus, which are exactly the evaluation benchmarks used in the paper. These benchmarks are widely distributed in web corpora and open-source repositories, so exact or near-exact overlap with a general SFT mixture is plausible. If the training data contains the evaluation problems or their gold solutions, the model can memorize expected outputs and edge-case inputs, inflating Attack Rate, Output Accuracy, Accuracy∩Attack, and downstream debugging pass@1 without demonstrating generalization to unseen programs. Please add a decontamination analysis for the Tulu-3 subset used in training, report the overlap statistics, and rerun the headline evaluations on a non-overlapping subset if any overlap is found.
[§4; Appendix A.1] The MBPP+Fix (Hard) split, which carries much of the paper's headline debugging claims, is defined by an arbitrary 50%-95% pass-rate filter on sampled faulty solutions. No sensitivity analysis is provided for these thresholds, and the split is self-constructed. Since the abstract and §5.2 emphasize results on this split, please report how the conclusions vary with the threshold choice or provide a principled justification for the selected range.

minor comments (4)

[Table 2] The column header "MPBB+Fix" appears to be a typo for "MBPP+Fix."
[Table 1; §5.1] Table 1 reports averages over 3 runs but does not report standard deviations or confidence intervals; given the small absolute gaps between some conditions, please include variance information or specify which differences are stable across runs.
[Appendix E, Table 8] The ablation results in Table 8 appear to be single-run numbers; given the large drops when removing test-time scaling, it would be helpful to report significance or repeated-run variance.
[References] There are duplicate reference entries for Guo et al. (2024) and for Muennighoff et al. (2023/2024); please consolidate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: UTGen's central training signal is external gold-code execution, and every downstream claim is evaluated against private EvalPlus/human-written unit tests.

full rationale

Walking the claimed derivation chain, UTGen's supervision is anchored externally rather than in the model's own predictions. Section 3.2 states that a unit test is kept as failing when fr(x) != bfb(x), using the reference (gold) code, and that 'to ensure output accuracy, we use the output of the reference code during training.' The intrinsic metrics in Section 4 compare generated inputs and outputs against fr and bfb, and downstream debugging in Appendix A.2 is measured with 'private human-annotated UTs,' not with UTGen's own tests. UTDebug's test-time scaling and backtracking components are ablated in Appendix E (Table 8), so even though the 50%-vote filtering is attributed to the authors' prior work (Prasad et al., 2024), the component's contribution is independently demonstrated by removal, not assumed from the citation. The abstract's headline improvements are evaluated on held-out external benchmarks (HumanEvalFix, MBPP+, EvalPlus), not on the training mixture. The one legitimate validity concern is contamination: Appendix B filters Tulu-3 SFT data for Python functions but reports no explicit decontamination against HumanEval or MBPP+. If overlap existed, results could be inflated, but that would be data leakage or benchmark contamination, not circularity under the stated standard; the paper exhibits no equation, definition, or self-citation chain in which a claimed prediction reduces to its own input by construction. Therefore no significant circularity is present in the derivation chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The main free parameters are pipeline hyperparameters rather than fitted constants; the most consequential ad hoc choice is the pass-rate filter for the Hard split and the lack of decontamination, which could affect the validity of the evaluation.

free parameters (5)

Number of generated UTs (n) = 3 (scaled to 9,15 in Appendix F)
Chosen for the main experiments; Appendix F shows performance varies with n.
Self-consistency sample count (k) = 8
Number of output completions sampled for majority vote; a fixed choice.
SC vote threshold = >50% (at least 4 of 8 votes)
Unit tests below this confidence are discarded; threshold selected without sensitivity analysis.
Debugging rounds (m) = 3
Number of UTDebug iterations; fixed across experiments.
Hard split pass-rate filter = 50%-95% of private UTs
Used to construct MBPP+Fix (Hard); arbitrary selection criterion that defines the benchmark difficulty.

assumptions (3)

domain assumption Executing the gold reference code provides the correct expected output for any unit test input.
Used throughout Sec 3.2 to label training data and to compute attack rate and output accuracy.
ad hoc to paper The Tulu-3 SFT mixture is a safe source of training problems that does not overlap with the evaluation benchmarks (HumanEval, MBPP+) in a way that inflates results.
Appendix B filters Tulu-3 but reports no decontamination check against HumanEval/MBPP+; if overlap exists, the evaluation is optimistic.
ad hoc to paper Self-consistency with a 50% vote threshold yields UT outputs accurate enough for reliable debugging feedback.
UTDebug relies on this; the ablation in Table 8 supports it, but the threshold is not studied.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning to Generate Unit Tests for Automated Debugging." pith.science (2026). https://pith.science/paper/TNIOBM2G

@misc{pith2026250201619,
  author       = {Pith},
  title        = {Pith review of: Learning to Generate Unit Tests for Automated Debugging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNIOBM2G}},
  note         = {Machine review of arXiv:2502.01619}
}

read the original abstract

Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs), motivating automated test generation. However, we uncover a trade-off between generating unit test inputs that reveal errors when given a faulty code and correctly predicting the unit test output without access to the gold solution. To address this trade-off, we propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs based on task descriptions. Since model-generated tests can provide noisy signals (e.g., from incorrectly predicted outputs), we propose UTDebug that (i) scales UTGen via test-time compute to improve UT output prediction, and (ii) validates and backtracks edits based on multiple generated UTs to avoid overfitting, and helps LLMs debug effectively. We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs. When used with UTDebug, we find that feedback from UTGen's unit tests improves pass@1 accuracy of Qwen2.5 32B on HumanEvalFix and our own harder debugging split of MBPP+ by over 3.17% and 12.35% (respectively) over other LLM-based UT generation baselines. Moreover, we observe that feedback from Qwen2.5 32B-based UTGen model can enhance debugging with frontier LLMs like GPT-4o by 13.8%. Lastly, we demonstrate that UTGen is a better judge for code correctness, outperforming a state-of-the-art trained 8B reward model by 4.43% on HumanEval+ with best-of-10 sampling using Qwen2.5 7B.

Figures

Figures reproduced from arXiv: 2502.01619 by the authors.

**Figure 1.** We propose UTGEN, which automatically generates failing unit tests (UTs) for a faulty code (triggering errors) without access to the gold solution. The generated UTs can in turn be used for LLM debugging via UTDEBUG, improving code accuracy. With rapid advancements in training large language models (LLMs; Achiam et al., 2023; Anthropic, 2024; Gemini et al., 2023), enhancing their coding abilities has garnered sign… view at source ↗

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Left: We highlight potential issues with debugging a faulty code using generated UTs: (I) non-failing UTs misclassify faulty code as correct; (II) UTs with incorrect outputs produce incorrect feedback and consequently, unsuccessful debugging. Right: We introduce UTDEBUG which (a) uses inference-time scaling to select better UT outputs based on a majority vote, and (b) generates multiple UTs for validation, discardin… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Increasing number of UTs across MBPP+Fix and MBPP+Fix (Hard) using UTs [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 15 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[4]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024
[5]

Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

2024
[6]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021
[7]

Kent Beck. 2022. Test driven development: By example. Addison-Wesley Professional

work page 2022
[8]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209--224

work page 2008

Show all 56 references

[9]

Sang Kil Cha, Maverick Woo, and David Brumley. 2015. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725--741. IEEE

2015
[10]

Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Tzu-iunn Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang, and Jinyoung Yeo. 2024. Coffee-gym: An environment for evaluating and improving natural language feedback on erroneous code. arXiv preprint...

2024 arXiv
[11]

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023 a . Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations

2023
[12]

Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2024. https://arxiv.org/abs/2409.12147 Magicore: Multi-agent, iterative, coarse-to-fine refinement for reasoning . arXiv preprint arXiv:2409.12147

2024
[13]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

2021 arXiv
[14]

Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. 2023 b . Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128

2023 arXiv
[15]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

2023
[16]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[17]

Massimo Ficco, Roberto Pietrantuono, and Stefano Russo. 2011. Bug localization in test-driven development. Advances in Software Engineering, 2011(1):492757

2011
[18]

Team Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

2023 arXiv
[19]

Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065

2024 arXiv
[21]

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024 b . Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

2024 arXiv
[22]

Christian Holler, Kim Herzig, and Andreas Zeller. 2012. Fuzzing with code fragments. In 21st USENIX Security Symposium (USENIX Security 12), pages 445--458

2012
[23]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[24]

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

2024
[25]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

2024 arXiv
[26]

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

2024 arXiv
[27]

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

2024
[28]

James C King. 1976. Symbolic execution and program testing. Communications of the ACM, 19(7):385--394

1976
[29]

Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

2024
[30]

Smith, and Hannaneh Hajishirzi

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024 b . https://huggingface.co/spaces/allenai/reward-bench Rewardbench: Evaluating reward m...

2024
[31]

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

2023 arXiv
[32]

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

2023 arXiv
[33]

Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024 a . Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

2024 arXiv
[34]

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024 b . Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36

2024
[35]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

2024
[36]

Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fr \"a nken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. arXiv preprint arXiv:2410.12832

2024 arXiv
[37]

Maximilien and L

E.M. Maximilien and L. Williams. 2003. https://doi.org/10.1109/ICSE.2003.1201238 Assessing test-driven development at ibm . In 25th International Conference on Software Engineering, 2003. Proceedings., pages 564--569

2003 arXiv
[38]

Seungjun Moon, Hyungjoo Chae, Yongho Song, Taeyoon Kwon, Dongjin Kang, Kai Tzu-iunn Ong, Seung-won Hwang, and Jinyoung Yeo. 2023. Coffee: Boost your code llms by fixing bugs with feedback. arXiv preprint arXiv:2311.07215

2023 arXiv
[39]

Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124

2023 arXiv
[40]

Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024. https://openreview.net/forum?id=mw1PWNSWZP Octopack: Instruction tuning code large language models . In The Twe...

2024
[41]

Michael Maximilien, Thirumalesh Bhat, and Laurie A

Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie A. Williams. 2008. https://doi.org/10.1007/S10664-008-9062-Z Realizing quality improvement through test driven development: results and experiences of four industrial teams . Empir. Softw. Eng., 13(3):289--302

2008 doi
[42]

Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: Teaching large language models to reason about code execution. In Forty-first International Conference on Machine Learning

2024
[43]

Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations

2023
[44]

Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. 2024. Self-consistency preference optimization. arXiv preprint arXiv:2411.04109

2024 arXiv
[45]

Henry Gordon Rice. 1953. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2):358--366

1953
[46]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

2023 arXiv
[47]

Max Sch \"a fer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering

2023
[48]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

2024
[49]

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . arXiv preprint arXiv:2408.03314

2024 arXiv
[50]

Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. https://arxiv.org/abs/2402.08115 On the self-verification limitations of large language models on reasoning and planning tasks . arXiv preprint arXiv:2402.08115

2024 arXiv
[51]

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

2024 arXiv
[52]

Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. 2010. https://doi.org/10.1145/1810295.1810441 Parameterized unit testing: theory and practice . In 2010 ACM/IEEE 32nd International Conference on Software Engineering, volume 2, pages 483--484

2010
[53]

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

2022
[54]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[55]

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STaR : Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

2022
[56]

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240

2024 arXiv
[57]

Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906

2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[4] [4]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024

[5] [5]

Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

2024

[6] [6]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021

[7] [7]

Kent Beck. 2022. Test driven development: By example. Addison-Wesley Professional

work page 2022

[8] [8]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209--224

work page 2008