Pith. sign in

REVIEW 5 major objections 6 minor 33 references

Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ThinkCoder claims that a single LLM exploration agent plus a non-LLM verifier can match or outperform multi-agent code-generation systems while spending only a small fraction of the compute.

desk verdict A cost-aware single-agent code generation framework with real results but an overstated efficiency headline. read the letter →

arxiv 2502.17442 v2 pith:2TKXRBF5 submitted 2024-12-30 cs.SE cs.AI

Xiaoqing Zhang , Yuhan Liu , Flood Sung , Xiuying Chen , Shuo Shang , Rui Yan This is my paper · ORCID
classification cs.SEcs.AI
keywords ThinkCodercodegenerationtest-timecomputeself-refinementtestingpoolReinforcedSelf-TrainingPass@1LLMverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that expensive multi-agent test-time compute is avoidable: one LLM can explore many candidate solutions and their tests, and a deterministic code runner can pick the best candidate by execution pass rate. The ThinkCoder framework interleaves this thorough exploration with optimal refinement, carrying only the best candidate and a concrete failed test into the next round. It then distills successful exploration trajectories back into the model with Reinforced Self-Training, so future searches start from a better policy. In the reported comparisons, this raises Pass@1 by 3.0% over MapCoder while using only 6.4% of the computation cost, and it reaches a 0.5% higher Pass@1 than AgentCoder after two refinement rounds instead of five.

What carries the argument

The load-bearing object is the Testing Pool, a growing set of LLM-generated regular, boundary, and performance tests. The CodeVerifier executes every candidate solution against this pool and ranks solutions by the pass rate $r_g = \#\text{tests passed} / |TP|$, choosing $g_s = \arg\max_i r_{g_i}$; this ranking requires no LLM calls. The pool is self-evolving: after each iteration, new tests are AST-deduplicated and are added to the permanent pool only when the local best solution beats the global best, so the selection signal improves as refinement proceeds. The other half of the machinery is Reinforced Self-Training (ReST), which takes successful exploration trajectories -- those that reached full pass rate -- and uses them as preference data to fine-tune the base model, so the model becomes better at finding good candidates and tests in later rounds.

What would settle it

Run ThinkCoder on a benchmark with official hidden tests and, for each problem, compare the Testing Pool pass rate of every candidate with its hidden-test pass rate; if the CodeVerifier's chosen best candidate is frequently not the candidate with the highest hidden-test pass rate, the pool is not a faithful proxy and the optimal-refinement claim fails for those problems.

Watch

Extended reading notes

Core claim

ThinkCoder's central discovery is that a single LLM can carry out the whole exploration-and-refinement loop if it is paired with a non-LLM CodeVerifier and a self-evolving Testing Pool. In each iteration, the exploration agent generates k candidate solutions and m test cases per solution; the tests are aggregated into a pool, deduplicated by AST, and used to compute a pass rate for every candidate. The candidate with the highest pool pass rate is selected as the current best, and the next instruction is built from the problem, that best code, a randomly selected failed test, and the failure feedback. The paper reports that this loop improves Pass@1 by 3.0% over MapCoder with only 6.4% of the computation cost, reaches a 0.5% higher Pass@1 than AgentCoder in two refinement rounds rather than five, and, when combined with ReST fine-tuning on successful trajectories, lets LLaMA2-7B reach competitive results with roughly 20% of the compute.

Load-bearing premise

The loop stands on the assumption that tests generated by the model are a fair stand-in for the hidden tests that define correctness, because the CodeVerifier can only rank solutions by those generated tests.

Editorial extensions

If this is right

  • A single model plus deterministic execution can replace a four-agent pipeline like MapCoder, cutting reported token usage to about 6.4% of the multi-agent cost at comparable or better Pass@1.
  • On benchmarks with sparse official tests, adding ThinkCoder's self-generated tests is enough to lift Pass@1, so the method is most effective where small test sets under-specify the problem.
  • Distilling successful trajectories into the model shifts part of the test-time compute into offline training, letting a 7B model compete with much larger systems at roughly 20% of the reported computational resources.
  • The joint improvement of generated code and generated tests, measured against ground-truth code and tests in the paper, suggests the testing pool can be used as a training signal rather than only as an evaluator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same select-by-execution loop could transfer to any task with a cheap deterministic correctness check, such as SQL query generation, proof checking, or tool-use planning, though the paper only evaluates Python code.
  • Because the Testing Pool is built by the same LLM that writes the code, the pass rate it measures can drift from true correctness if the model's generated tests share its blind spots; a re-run with human-authored or adversarial tests would test how far the selection signal generalizes.
  • ReST's rapid convergence, acknowledged in the paper's Limitations section, suggests the distillation mostly improves local search toward known-good answers rather than expanding the global solution space; an on-policy update that rewrites the pool during training would be a natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes ThinkCoder, a test-time computation framework for code generation that uses a single LLM-based Exploration Agent to produce candidate solutions and test cases, and a non-LLM CodeVerifier that selects the best candidate by pass rate on a self-evolving Testing Pool. The framework is applied to MBPP, HumanEval, extended/extrapolation benchmarks, LiveCodeBench, and CodeContests, and the paper also introduces a Reinforced Self-Training (ReST) phase that fine-tunes smaller models on successful exploration trajectories. The main claims are that ThinkCoder improves Pass@1 over strong multi-agent baselines (MapCoder, AgentCoder, LDB, LPW) while substantially reducing token usage, and that ReST lets models like LLaMA2-7B reach competitive performance with fewer test-time compute resources.

Significance. If the empirical results hold, the contribution is genuine and practically relevant: replacing multi-agent LLM pipelines with a single LLM plus a deterministic verifier could substantially reduce the cost of high-quality code generation, and the ReST phase shows a plausible path to amortizing exploration cost into the model weights. The paper ships a detailed algorithm (Algorithm 1), a code repository link, and a dedicated transfer analysis (Figure 4), which are real strengths. However, the central quantitative claims are not yet fully supported: the headline cost figure is not reproducible from the tables, hyperparameters are selected on the evaluation benchmark, the selection signal for the best solution is defined inconsistently and may be biased toward self-generated tests, and no uncertainty quantification is provided for any Pass@1 number. These issues are fixable but require substantive revision.

major comments (5)
  1. [Section 6.6, Table 5] The abstract's claim that ThinkCoder uses 'just 6.4% of the computation cost' of MapCoder is not reproducible from Table 5. From the token counts listed, the ThinkCoder/MapCoder ratios are approximately 0.26/21.3 = 1.2% for HumanEval, 0.24/5.58 = 4.3% for MBPP, and 2.94/18.3 = 16.1% for CodeContests. No single number equals 6.4%, and the paper does not explain how 6.4% is derived. Moreover, the cost metric counts only agent response tokens and omits prompt tokens, test-generation tokens, and CodeVerifier execution time. Please define the cost metric explicitly and recompute all reported efficiency claims accordingly.
  2. [Section 6.5, Figure 3] Hyperparameters are selected on the MBPP evaluation set, which is also a primary benchmark in Tables 1-4. Section 6.5 says 'We conducted hyperparameter selection experiments on the MBPP dataset,' and Figure 3(a) shows Pass@1 as a function of temperature on MBPP. If these curves were used to pick t, k, n, or θ for the main results, then the reported MBPP numbers are optimistically biased and cannot be treated as unbiased test-set estimates. Please instead tune on a held-out split (e.g., the MBPP validation portion described in Section 5.1) or fix a single default configuration and report sensitivity separately.
  3. [Algorithm 1, lines 13-21; Section 3.3] There is an inconsistency between the text and the algorithm regarding the selection signal. Section 3.3 states that the CodeVerifier computes the pass rate by running candidate code against 'all test cases,' but Algorithm 1 line 14 computes r_gi 'with the input TP'' (only the test cases generated in the current iteration), and TP is expanded only when the local pass rate improves the global best (lines 19-21). Consequently, the best solution is selected based on its pass rate on a newly generated, self-curated test set, not on the accumulated Testing Pool. This is load-bearing because it determines both the refinement direction and the final output. Please clarify which signal is actually used and justify why a pass rate on self-generated tests transfers to hidden tests; Figure 4 provides transfer evidence only for MBPP with CodeQwen1.5-7B-Chat, and the Limitations section itself concedes a gap between generated and annotated cases.
  4. [Tables 1-4, 6, 7] No error bars, confidence intervals, or repeated-seed results are reported for any Pass@1 number, despite the stochastic generation process (temperature t in {0.5, 1.0}, k samples per problem, random test selection in Algorithm 1 line 6). Some comparisons hinge on small differences, e.g., Table 2 shows average Pass@1 of 87.7 vs 87.2 for ThinkCoder versus AgentCoder with GPT-4-Turbo. Without variance estimates, these differences may be well within run-to-run noise. Please report means and standard deviations or bootstrap intervals over at least three to five independent runs.
  5. [Section 4.1; Figure 2 caption] The data collection procedure for ReST is described ambiguously. Section 4.1 says samples are added to D_M0 when the CodeVerifier's pass_rate equals 1, while Figure 2's caption says success trajectories are collected 'based on the verification with ground truth tests.' These are different filtering criteria. If ground-truth tests are used, the text in Section 4.1 should say so explicitly; if generated tests are used, then the ReST training signal inherits the same self-generated-test proxy risk identified in the Testing Pool, and the claim that ReST aligns with 'human preferences' needs additional support.
minor comments (6)
  1. [Section 3.1, Section 6.6, Conclusion] There are several typos: 'paiblue' should be 'paired' in Section 3.1, 'Compablue' should be 'Compared' in Section 6.6, 'ThinkeCoder' should be 'ThinkCoder' in the Conclusion, and 'Artifcial' should be 'Artificial' in the author affiliations.
  2. [Algorithm 1, line 6] The algorithm samples 'm successful test cases from TP' in every iteration, but TP is initially empty; please specify how the first iteration handles missing tests and feedback (the later 'if ft and f is not None' guard implies a special case, but it is not stated explicitly).
  3. [Table 5] The column header 'Tokens (k)' is confusing; it presumably means thousands of tokens. Please rename it to 'Tokens (thousands)' or define the unit in the caption.
  4. [Figure 4] The figure plots trendlines for generated code and generated tests on MBPP, but it does not report per-iteration sample sizes, confidence intervals, or the number of problems, and the trendlines appear to have few underlying points. Please provide these details.
  5. [Table 7] Table 7 reports results on 'randomly 60 MBPP samples' with GPT-4, but the random selection process (seed, sampling method, whether these overlap with the evaluation set) is not described. This subset is small and the reported Pass@1 values may be unstable.
  6. [References] The paper cites Backus et al. (1960) for Abstract Syntax Trees, but that reference is the Algol 60 report; a more standard reference on syntax trees in program analysis would be more appropriate, though this is of course a minor point.

Circularity Check

1 steps flagged · score 3.0 of 10

Internal Testing-Pool pass rate is self-confirming by construction, but headline benchmark Pass@1 results are externally evaluated, so circularity is partial.

  1. self definitional [Section 3.5 (Testing Pool); Algorithm 1, lines 6 and 19-22; Section 4 opening]
    "after each iteration, the testing pool only selects test cases that can improve the code's pass rate, thus enhancing the diversity and accuracy of the pool. ... if lp > gp then Set gp = lp, gs = ls; Add TP' to TP ... Randomly select m successful test cases from TP as tests ... Each increase in the pass rate signifies the development of better codes."

    The pass-rate signal that selects the 'best' code is computed on a pool that is itself generated by the same exploration agent, and the pool is retained only when a new code's pass rate improves (lp > gp). The next iteration then samples 'successful test cases' from that pool. Therefore the Testing Pool is curated by the very quantity it is used to measure, and gp is non-decreasing by the update rule rather than by code quality alone. Section 3.5 says the pool 'only selects test cases that can improve the code's pass rate,' and Section 4 interprets the resulting monotone TP pass rate as proof of better code. That internal inference is circular by construction.

full rationale

The central Pass@1 comparisons (Tables 1-4) are measured on held-out HumanEval, MBPP, ET variants, LiveCodeBench, and CodeContests, so the headline claim does not reduce by definition to the self-generated Testing Pool; there is no equation-level identity between TP pass rate and benchmark pass rate. I therefore do not assign a high circularity score. However, the paper's internal verification signal is self-confirming by construction: the Testing Pool is LLM-generated, Algorithm 1 adds TP' to TP only when a new code's local pass rate improves (lp > gp), and the next round samples 'successful test cases' from TP. Section 3.5 states that the pool 'only selects test cases that can improve the code's pass rate,' and Section 4 then reads the resulting monotone TP pass rate as evidence that 'each increase... signifies the development of better codes.' That inference is circular for the internal metric. Figure 4 provides some independent ground-truth support, and the Limitations section explicitly concedes 'the gap between generated and annotated cases,' which is why this is a partial internal circularity rather than a full invalidation of the external results. ReST data collection uses ground-truth verification for DM0 (Figure 2), but the reflection trajectories RM0 are selected using LLM-generated tests, so part of the training signal inherits the same proxy. Net: one concrete self-definitional step in the Testing Pool/CodeVerifier loop; external benchmarks keep the central claim independently testable.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The only postulates are empirical assumptions about self-generated tests serving as a correctness proxy and about benchmark cleanliness, plus the hand-set hyperparameters listed above.

free parameters (5)
  • Exploration budget k = 5 for base-model runs, 20 for SOTA comparison
    Hand-set; Pass@k saturates with k, and k=20 is described as inefficient in Section 6.2.
  • Refinement budget n = 5 for base runs, 2 for SOTA comparison
    Hand-set; more rounds help but saturate, as shown in Figure 3c.
  • Sampling temperature t = 0.5 for base runs, 1.0 for SOTA comparison
    Chosen per experiment; Figure 3a shows the optimal temperature differs across models.
  • Budget control threshold theta = 0.8 for base runs, 1.0 for SOTA comparison
    Controls early stopping; higher theta reduces task count but risks leaving errors uncorrected, as discussed in Appendix A.2.
  • Number of generated tests per exploration m = 3
    Set to balance Pass@1 and runtime; Appendix A.3 shows m=10 gives 86.7 versus 81.7 for m=3.
assumptions (3)
  • domain assumption Pass rate on the LLM-generated testing pool predicts performance on the hidden test set.
    ThinkCoder's selection and early stopping in Algorithm 1 rely on this, and the authors acknowledge in the Limitations section that there is a gap between generated and annotated tests.
  • standard math The local Python execution environment correctly evaluates generated code against the testing pool.
    The CodeVerifier runs code against tests; this is the standard assumption for benchmark harnesses and is not explicitly proven in the paper.
  • domain assumption The HumanEval, MBPP, and extended test sets are not contaminated in the base or fine-tuned models.
    Not discussed in the paper; this matters because the exploration trajectories come from code LLMs that may have memorized popular benchmark problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement." pith.science (2026). https://pith.science/paper/2TKXRBF5

@misc{pith2026250217442,
  author       = {Pith},
  title        = {Pith review of: Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TKXRBF5}},
  note         = {Machine review of arXiv:2502.17442}
}
read the original abstract

Code generation is crucial in software engineering for automating the coding process efficiently. While test-time computation methods show promise, they suffer from high latency due to multiple computation rounds. To overcome this, we introduce \textbf{ThinkCoder}, a framework that combines thorough exploration with optimal refinement. The exploration phase diversifies the solution space by searching for potential solutions, followed by a refinement phase that enhances precision. This approach allows us to select the best solution through careful consideration before taking action, avoiding excessive trial and error. To further minimize test-time computation overhead, we introduce preference-driven optimization with Reinforced Self-Training (ReST), which uses exploration trajectories from ThinkCoder to guide LLM's evolution. This approach enhances LLM's exploration efficiency via preference learning, cutting costs while maintaining accuracy. ThinkCoder boosts the performance with a single LLM, excelling on benchmarks like HumanEval and MBPP. Compared to SOTA models, it improves Pass@1 by 3.0\% over MapCoder with just 6.4\% of the computation cost. Against AgentCoder, ThinkCoder achieves a 0.5\% higher Pass@1 after 2 rounds, outperforming AgentCoder's 5 rounds. Additionally, ReST with success trajectories enhances efficiency, allowing models like LLaMA2-7B to achieve competitive results using only 20\% of the computational resources. These results highlight the framework's effectiveness and scalability.

Figures

Figures reproduced from arXiv: 2502.17442 by the authors.

Figure 1
Figure 1. The end-to-end process of ThinkCoder involves [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Trajectory collection with ThinkCoder and its application in ReST training for LLMs. We collect [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) The Pass@1 metric of the baseline models under different temperature [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The variation in Pass@1 performance of the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The relationship between the computational [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: ThinkCoder exploration agent prompt example. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: ThinkCoder CodeVerifier input example [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: ThinkCoder exploration examples. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: ThinkCoder refinement examples [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: ThinkCoder testing pool examples. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 9 canonical work pages

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  2. [2]

    John W Backus, Friedrich L Bauer, Julien Green, Charles Katz, John McCarthy, Alan J Perlis, Heinz Rutishauser, Klaus Samelson, Bernard Vauquois, Joseph Henry Wegstein, et al. 1960. Report on the algorithmic language algol 60. Communications of the ACM, 3(5):299--311

    work page 1960

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. GitHub repository

    2023

  5. [5]

    Angelica Chen, J \'e r \'e my Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. 2023. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749

    arXiv 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  7. [7]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. arXiv preprint arXiv:2301.09043

    arXiv 2023

  8. [8]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology, 33(7):1--38

    2024

Show all 33 references
  1. [9]

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998

    2023 arXiv

  2. [10]

    Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2024. Exploring the potential of chatgpt in automated code refinement: An empirical study. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1--13

    2024

  3. [11]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  4. [12]

    Dong Huang, Qingwen Bu, and Heming Cui. 2023 a . Codecot and beyond: Learning to program and test like a developer. arXiv preprint arXiv:2308.08784

    2023 arXiv

  5. [13]

    Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023 b . Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010

    2023 arXiv

  6. [14]

    Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403

    2024 arXiv

  7. [15]

    Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023. Chatgpt and software testing education: Promises & perils. In 2023 IEEE international conference on software testing, verification and validation workshops (ICSTW), pages 4130--4137. IEEE

    2023

  8. [16]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology, 33(7):1--30

    2024

  9. [17]

    Bryan F Jones, HH Sthamer X Yang, and DE Eyres. 2024. The automatic generation of software test data sets using adaptive search techniques. WIT Transactions on Information and Communication Technologies, 14

    2024

  10. [18]

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2024 a . Structured chain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology

    2024

  11. [19]

    Qingyao Li, Wei Xia, Kounianhua Du, Xinyi Dai, Ruiming Tang, Yasheng Wang, Yong Yu, and Weinan Zhang. 2024 b . Rethinkmcts: Refining erroneous thoughts in monte carlo tree search for code generation. arXiv preprint arXiv:2409.09584

    2024

  12. [20]

    Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Finding failure-inducing test cases with chatgpt. arXiv preprint arXiv:2304.11686

    2023 arXiv

  13. [21]

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology, 33(5):1--26

    2024

  14. [22]

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124

    2023 arXiv

  15. [23]

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286

    2022 arXiv

  16. [24]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  17. [25]

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500

    2024 arXiv

  18. [26]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  19. [27]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  20. [28]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering

    2024

  21. [29]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A pair programming framework for code generation via multi-plan exploration and feedback-driven refinement. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1319--1331

    2024

  22. [30]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658

    2024 arXiv

  23. [31]

    Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. 2023. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372

    2023 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.