Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Analyzing the Instability of Large Language Models in Automated Bug Injection and Correction

T0 review · 4 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that ChatGPT's bug-fix outputs become both structurally and functionally less stable as temperature rises, and that setting temperature to zero does not remove nondeterminism.

desk verdict Right question, small sample, and a few numbers that don't add up — fixable, but not ready as-is. read the letter →

arxiv 2509.06429 v1 pith:AHH6TNYM submitted 2025-09-08 cs.SE

Mehmet Bilal Er , Nagehan İlhan , Umut Kuran This is my paper
classification cs.SE
keywords LLMinstabilitybugfixingautomatedprogramrepairtemperatureparameterLevenshteinsimilarityoutputequivalenceratenondeterminismQuixBugs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to measure whether a large language model gives stable, functionally correct fixes when asked repeatedly to repair the same buggy code. Using 20 buggy Python algorithm problems from the QuixBugs benchmark and the ChatGPT (GPT-4) model, the authors compare nine outputs per problem across temperatures 0, 0.5, and 1, measuring structural similarity with normalized Levenshtein distance and functional equivalence with a test-based Output Equivalence Rate. They find that output variability grows as temperature rises: average syntactic similarity between repeated fixes falls, and functional success drops from 70% at temperature 0 to 62% at temperature 1. They also report that temperature 0 does not make the model deterministic, because repeated runs still differ and some fixes still fail tests. If this holds, automated bug-fixing tools need variance-aware validation—multiple samples, test-based selection, and output-equivalence thresholds—rather than single-run acceptance.

What carries the argument

The measuring pair is normalized Levenshtein similarity, which scores character-level syntactic distance between two generated fixes, and the Output Equivalence Rate (OER), which scores how often two programs produce identical outputs on a shared test set. These are applied over a repeated-generation design: 20 QuixBugs problems, three temperatures (0, 0.5, 1), and three generations per temperature, giving nine fixes per problem and 540 outputs total. The combination is what lets the paper separate 'looks different' from 'behaves differently' for the same bug.

What would settle it

Run the same 20 QuixBugs problems with many more generations per temperature, for example 30 or 100, logging API version and sampling flags; if the per-problem categories and the 0.70-to-0.62 OER trend remain stable across repeated replications, the central claim holds, and if they swing, the instability estimates themselves are unstable. Separately, issue repeated identical requests at temperature 0 and check whether any two outputs differ; if they never differ in a controlled setting, the paper's residual-nondeterminism claim would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that the same buggy code, given to the same model with the same prompt, produces a distribution of fixes rather than one fix, and that this distribution widens as the sampling temperature rises. Across 20 buggy Python algorithms from QuixBugs, three generations at each of temperatures 0, 0.5, and 1 show average Levenshtein similarity between repeated fixes falling and functional agreement with test oracles (OER) dropping from 0.70 at temperature 0 to 0.62 at temperature 1. A second claim is that temperature 0 does not restore determinism: repeated runs at temperature 0 still differ syntactically and some still fail tests, so residual randomness is attributed to s

Load-bearing premise

The study assumes that three generations per problem per temperature are enough to estimate a problem's success rate and output variance; with only three binary trials, one different run changes an entire problem's category, so the reported stability differences rest on that small-sample premise.

Editorial extensions

If this is right

  • Single-shot fixes from an LLM should not be treated as dependable; a CI/CD admission gate should require multiple generated fixes and test-based verification before accepting a patch.
  • Lowering temperature improves stability but does not guarantee determinism, so production pipelines that assume deterministic output at temperature 0 are unsound.
  • A combined view of structural similarity and functional equivalence is a workable way to quantify how variable an LLM repair pipeline is.
  • Automation of bug fixing is safer for problems with standard, template-like solutions, while graph and state-heavy tasks need stronger guardrails or human review.
  • Multi-sample ensembling with test-aware selection and clustering of similar patches can reduce variance-induced failures in LLM-based repair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether sampling more fixes at higher temperature and then selecting by test results would recover the lost reliability; that is a natural next experiment implied by its protocol.
  • With only three runs per condition, the exact stability rankings of individual problems are fragile; a larger replication would be needed to know which problems truly sit on the stable or unstable side.
  • The protocol could be applied to other models and decoding settings beyond temperature; if the same pattern holds, variance reporting would become a standard part of LLM-repair evaluation.
  • One could also test whether API version changes or non-temperature sampling flags account for the residual nondeterminism seen at temperature 0.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper reports an empirical study of GPT-4's consistency in repairing 20 Python bugs selected from QuixBugs. For each problem, three fixes are generated at each of three temperatures (0, 0.5, 1.0). The authors compute normalized Levenshtein similarity between generated fixes and an Output Equivalence Rate (OER) intended to capture functional consistency. They report that mean OER declines from 0.70 to 0.62 as temperature increases, that syntactic similarity becomes more dispersed, and that temperature 0 does not guarantee determinism. They conclude by recommending variance-aware, multi-sample validation protocols before LLM-generated patches are deployed.

Significance. The topic is timely and the study design has good intentions: it uses a public program-repair benchmark, varies temperature systematically, and combines structural and functional measures rather than reporting best-of-N accuracy. If the empirical results were statistically solid, the paper would provide a useful cautionary result for LLM-based repair tooling. However, the current evidence is not solid: the OER metric is not computed as defined, the sample size per condition (n=3) cannot support the variance and trend claims, and several tables contain internal numerical contradictions. The paper also fails to provide replication data, exact model version, or inferential statistics. With revision, the central idea could become a contribution; in its current form it is not established.

major comments (4)
  1. [§3.3, Eq. (3); §4.2 Tables 5-7] The OER is defined as the proportion of test inputs for which two programs produce identical outputs, but the experimental tables report only binary pass/fail per generated fix against the reference test suite (successes/failures out of 3 trials). These are different quantities. As reported, 'OER=0.70' is actually the mean per-problem success rate, not an output-equivalence rate between two programs. This mismatch is load-bearing because the paper's central functional-instability measure is built on it. Please either redefine the metric as a pass@k-style success rate or compute true pairwise OER among the three generations per (problem, temperature).
  2. [§4.2 Tables 2-4] Several descriptive statistics are internally inconsistent. For example, Table 2 'breadth_first_search' reports Average=0.40, Maximum=0.40, Minimum=0.39, yet Variance=0 and StdDev=0; 'bucketsort' reports Variance=0 but StdDev=0.02; 'rpn_eval' reports Variance=0 but StdDev=0.02. Similar issues occur in Tables 3 and 4. Because these tables support the syntactic-instability claim, they must be recomputed and verified. Provide raw pairwise similarity values or code/scripts so readers can check the computations.
  3. [§4.2 Tables 5-7, Figure 5] With n=3 per (problem, temperature), the aggregate OER difference between T=0 (42/60) and T=1 (37/60) has a standard error of approximately 0.086; the observed drop is 0.083 (z≈0.97), so the headline 'OER decreases with temperature' is not distinguishable from sampling noise. For individual problems, any 1/3 or 2/3 success count yields a sample standard deviation of 0.58—the maximum possible—so the reported variance bars in Figure 5 are largely artifacts of the tiny sample. The manuscript reports no confidence intervals, hypothesis tests, or effect-size measures. The claims 'much more unstable and variable' and 'high temperatures showing especially high rates of functional failure' require either substantially more generations per cell or appropriate statistical modeling.
  4. [§3.1; §4.2; Abstract; Conclusion] The experimental design is described inconsistently. §3.1 states two temperatures (1.0 and 0.0), while the abstract and §4.2 use three (0, 0.5, 1.0). The abstract says nine outputs per problem (20×9=180 total), while the conclusion says 'the model was run nine times for each problem at three different temperature values' and reports 540 outputs (20×9×3=540). This discrepancy affects the interpretation of every table. Specify the exact number of generations per (problem, temperature, run) and correct the totals.
minor comments (7)
  1. [Introduction] The sentence 'The major contributions of this study are summarized as follows:' is duplicated immediately after 'The main contributions of this study are summarized as follows:'.
  2. [References] QuixBugs is cited twice as [7] and [40]; please merge or distinguish them if they refer to different versions.
  3. [Figures/Tables] There are typos in figure/table labels: Figure 2 caption 'Tempreature', Table 7 header 'genel', and the text near Figure 4 refers to 'breadth basis search'.
  4. [§3.1] It is not stated whether Levenshtein similarity is computed pairwise among the three generated fixes or between each fix and a golden reference fix. This must be clarified because it changes the meaning of the variance and low-similarity ratios.
  5. [§4.2 Tables 2-4] The threshold for the 'Low Similarity Ratio (<0.7)' column is arbitrary and not justified. Consider reporting the full distribution or a sensitivity analysis around this threshold.
  6. [Reproducibility] The paper does not state the exact GPT-4 model version, API access date, or decoding parameters beyond temperature. These details are necessary for replication given known model-version drift.
  7. [Section 6] The limitations paragraph mentions AST-based metrics and model-version drift, but it does not acknowledge the small number of repetitions (n=3) as a limitation. This should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's instability findings are direct empirical measurements, with no fitted parameter relabeled as a prediction and no load-bearing self-citation.

full rationale

The paper reports an empirical study: it runs GPT-4 on 20 QuixBugs problems at three temperature settings, with three generations per condition, and computes Levenshtein similarity and Output Equivalence Rate (OER) from the raw outputs. The central claims—that outputs become more variable as temperature rises and that temperature=0 does not guarantee full determinism—are summaries of the measured tables and figures, not derived quantities that coincide with an input by construction. No parameter is fitted to a subset of the data and then 'predicted' on a closely related quantity; the metrics in Equations (1)-(3) are standard external definitions applied to the generated code and test outputs. The reference list contains no self-citations by the present authors that carry a load-bearing premise. The proposed stabilization protocol is presented as a methodological recommendation motivated by the observed variance, not as a separate predictive result claimed to be derived from first principles. Concerns about the small number of repeated runs (n=3) or possible benchmark leakage in the training data are validity/threats-to-generalization issues, not circularity in the derivation sense. Therefore, no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The main assumptions are about metric validity, oracle completeness, small-sample representativeness, and the role of temperature. The only hand-chosen number is the 0.7 low-similarity threshold.

free parameters (1)
  • Low-similarity threshold = 0.7
    Hand-picked cutoff defining 'Low Similarity Ratio' in Tables 2-4; no justification or sensitivity analysis is provided.
assumptions (4)
  • domain assumption Character-level Levenshtein similarity is a valid measure of syntactic similarity for code
    Used throughout Section 3.2 and all similarity tables; no justification that character-level distance captures meaningful syntactic structure of code.
  • domain assumption QuixBugs test oracles are correct and sufficient to determine functional correctness
    OER (as pass rate) relies on the oracle being complete and correct; no coverage or oracle quality analysis is provided.
  • domain assumption Three repeated generations per (problem, temperature) adequately characterize output variance
    Tables 5-7 and Figures 3-5 compute variance from n=3 binary trials; no power analysis or justification for such a small sample is given.
  • domain assumption GPT-4's temperature parameter is the primary controllable source of output variability
    The paper attributes trends to temperature but also finds residual non-determinism at T=0, indicating unmeasured sampling and API factors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing the Instability of Large Language Models in Automated Bug Injection and Correction." pith.science (2026). https://pith.science/paper/AHH6TNYM

@misc{pith2026250906429,
  author       = {Pith},
  title        = {Pith review of: Analyzing the Instability of Large Language Models in Automated Bug Injection and Correction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHH6TNYM}},
  note         = {Machine review of arXiv:2509.06429}
}
read the original abstract

The use of Large Language Models (LLMs) in software engineering tasks is growing, especially in the areas of bug fixing and code generation. Nevertheless, these models often yield unstable results; when executed at different times with the same input, they can generate radically different code. The consistency of LLMs in bug-fixing tasks has not yet been thoroughly assessed, despite the fact that this instability has typically been discussed in the literature in relation to code generation. The purpose of this study is to look into how unstable an LLM like ChatGPT is when it comes to fixing code bugs. We examine the structural, syntactic, and functional variations among several fix recommendations made in response to the same prompt using code samples with various error types. Additionally, we assess how instability is affected by the temperature settings (0, 0.5, and 1) used for the model's deterministic operation. For a total of 20 problems in the experimental analysis, the model produced three fix suggestions at each temperature value, comparing nine distinct outputs for each problem. The Syntax Similarity and Output Equivalence Rate (OER) metrics were used to assess the outputs' structural and functional consistency. The results demonstrate that the model's outputs become much more unstable and variable as the temperature rises, with high temperatures showing especially high rates of functional failure. According to syntax similarity analyses, the suggested fixes show notable structural differences at high temperatures but are fairly similar at low temperatures. The purpose of this study is to provide important methodological insights into how LLM-based error correction systems can be applied more consistently in software development processes while also casting doubt on their dependability.

Figures

Figures reproduced from arXiv: 2509.06429 by the authors.

Figure 1
Figure 1. Syntactic Similarity (Levenshtein) – Temperature Comparison As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Average Similarity by Problem and Tempreature There are also task nuances in [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Success Category Distribution by Temperature [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Heatmap of Success Rate per Problem and Temperature [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Success Rate with StdDev [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is Model Instability just Noise to be Tolerated or a Property that can be Managed?

    cs.SE 2026-07 accept novelty 6.0 of 10

    Performance instability in SBSE is pervasive yet partially manageable by labeling, acquisition, complexity and Gini splits; a data-inherent residual floor remains.

Reference graph

Works this paper leans on

38 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jiang, F

    J. Jiang, F. Wang, J. Shen, S. Kim, S. Kim, A Survey on Large Language Models for Code Generation, ACM Trans. Softw. Eng. Methodol. (2025). https://doi.org/10.1145/3747588

    doi:10.1145/3747588 2025

  2. [2]

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, Competition-level code generation with alphacode, Science (1979) 378 (2022) 1092–1097

    work page 1979

  3. [3]

    C.S. Xia, Y. Wei, L. Zhang, Automated program repair in the era of large pre-trained language models, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023: pp. 1482–1494

    work page 2023

  4. [5]

    K. Liu, L. Li, A. Koyuncu, D. Kim, Z. Liu, J. Klein, T.F. Bissyandé, A critical review on the evaluation of automated program repair systems, Journal of Systems and Software 171 (2021) 110817

    work page 2021

  5. [6]

    Gazzola, D

    L. Gazzola, D. Micucci, L. Mariani, Automatic software repair: A survey, in: Proceedings of the 40th International Conference on Software Engineering, 2018: p. 1219

    work page 2018

  6. [8]

    H. Ye, M. Martinez, T. Durieux, M. Monperrus, A comprehensive study of automatic program repair on the QuixBugs benchmark, Journal of Systems and Software 171 (2021) 110825

    work page 2021

  7. [9]

    Ouyang, J.M

    S. Ouyang, J.M. Zhang, M. Harman, M. Wang, An empirical study of the non- determinism of chatgpt in code generation, ACM Transactions on Software Engineering and Methodology 34 (2025) 1–28

    work page 2025

  8. [10]

    Thakur, B

    S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, S. Garg, Verigen: A large language model for verilog code generation, ACM Transact Des Autom Electron Syst 29 (2024) 1–31

    work page 2024

Show all 38 references
  1. [11]

    Y. Liu, C. Tantithamthavorn, Y. Liu, L. Li, On the reliability and explainability of language models for program generation, ACM Transactions on Software Engineering and Methodology 33 (2024) 1–26

    2024

  2. [12]

    Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.-B.D. Le, D. Lo, Refining chatgpt-generated code: Characterizing and mitigating code quality issues, ACM Transactions on Software Engineering and Methodology 33 (2024) 1–26

    2024

  3. [13]

    Y. Tang, Z. Liu, Z. Zhou, X. Luo, Chatgpt vs sbst: A comparative assessment of unit test suite generation, IEEE Transactions on Software Engineering 50 (2024) 1340–1359

    2024

  4. [14]

    Z. Liu, Y. Tang, X. Luo, Y. Zhou, L.F. Zhang, No need to lift a finger anymore? assessing the quality of code generation by chatgpt, IEEE Transactions on Software Engineering 50 (2024) 1548–1584

    2024

  5. [15]

    S. Kang, J. Yoon, N. Askarbekkyzy, S. Yoo, Evaluating diverse large language models for automatic and general bug reproduction, IEEE Transactions on Software Engineering (2024)

    2024

  6. [16]

    Y. Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y. Xu, J. Wang, Fine-tuning large language models to improve accuracy and comprehensibility of automated code review, ACM Transactions on Software Engineering and Methodology 34 (2024) 1–26

    2024

  7. [17]

    Elazar, N

    Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, Y. Goldberg, Measuring and improving consistency in pretrained language models, Trans Assoc Comput Linguist 9 (2021) 1012–1031

    2021

  8. [18]

    L. Wang, X. Chen, X. Deng, H. Wen, M. You, W. Liu, Q. Li, J. Li, Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs, NPJ Digit Med 7 (2024) 41

    2024

  9. [19]

    L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, S. Li, Exploring the Capabilities of LLMs for Code- Change-Related Tasks, ACM Transactions on Software Engineering and Methodology 34 (2025) 1–36

    2025

  10. [20]

    Mizrahi, G

    M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, G. Stanovsky, State of what art? a call for multi-prompt llm evaluation, Trans Assoc Comput Linguist 12 (2024) 933–949

    2024

  11. [21]

    X. Zhou, S. Cao, X. Sun, D. Lo, Large language model for vulnerability detection and repair: literature review and the road ahead (2024), ArXiv Preprint ArXiv:2404.02525 (2024)

    2024 arXiv

  12. [22]

    J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, Q. Wang, Software testing with large language models: Survey, landscape, and vision, IEEE Transactions on Software Engineering 50 (2024) 911–936

    2024

  13. [23]

    Tamberg, H

    K. Tamberg, H. Bahsi, Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study. arXiv 2024, ArXiv Preprint ArXiv:2405.15614 (n.d.)

    2024 arXiv

  14. [24]

    Marques, R.R

    N. Marques, R.R. Silva, J. Bernardino, Using ChatGPT in software requirements engineering: a comprehensive review. Future Internet 16 (6)(2024), (2024)

    2024

  15. [25]

    X. Zhu, W. Zhou, Q.-L. Han, W. Ma, S. Wen, Y. Xiang, When software security meets large language models: A survey, IEEE/CAA Journal of Automatica Sinica 12 (2025) 317–334

    2025

  16. [26]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineering: A systematic literature review, ACM Transactions on Software Engineering and Methodology 33 (2024) 1–79

    2024

  17. [27]

    Ahmad, S

    B. Ahmad, S. Thakur, B. Tan, R. Karri, H. Pearce, On hardware security bug code fixes by prompting large language models, IEEE Transactions on Information Forensics and Security 19 (2024) 4043–4057

    2024

  18. [28]

    Y. Dong, X. Jiang, Z. Jin, G. Li, Self-collaboration code generation via chatgpt, ACM Transactions on Software Engineering and Methodology 33 (2024) 1–38

    2024

  19. [29]

    Nashaat, J

    M. Nashaat, J. Miller, Towards efficient fine-tuning of language models with organizational data for automated software review, IEEE Transactions on Software Engineering (2024)

    2024

  20. [30]

    H. Li, Y. Hao, Y. Zhai, Z. Qian, Enhancing static analysis for practical bug detection: An llm-integrated approach, Proceedings of the ACM on Programming Languages 8 (2024) 474–499

    2024

  21. [31]

    L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, W.Y. Wang, Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies, Trans Assoc Comput Linguist 12 (2024) 484–506

    2024

  22. [32]

    Chelli, J

    M. Chelli, J. Descamps, V. Lavoué, C. Trojani, M. Azar, M. Deckert, J.-L. Raynier, G. Clowez, P. Boileau, C. Ruetsch-Chelli, Hallucination rates and reference accuracy of ChatGPT and bard for systematic reviews: comparative analysis, J Med Internet Res 26 (2024) e53164

    2024

  23. [33]

    Z. Lin, S. Guan, W. Zhang, H. Zhang, Y. Li, H. Zhang, Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models, Artif Intell Rev 57 (2024) 243

    2024

  24. [34]

    Levenshtein, Binary codes capable of correcting deletions, insertions, and reversals, Soviet Physics

    V.I. Levenshtein, Binary codes capable of correcting deletions, insertions, and reversals, Soviet Physics. Doklady 10 (1965) 707–710. https://api.semanticscholar.org/CorpusID:60827152

    1965

  25. [35]

    J. Li, G. Li, Z. Li, Z. Jin, X. Hu, K. Zhang, Z. Fu, Codeeditor: Learning to edit source code with pre-trained models, ACM Transactions on Software Engineering and Methodology 32 (2023) 1–22

    2023

  26. [36]

    J. Li, Y. Li, G. Li, Z. Jin, Y. Hao, X. Hu, Skcoder: A sketch-based approach for automatic code generation, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023: pp. 2124–2135

    2023

  27. [37]

    Mastropaolo, L

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, G. Bavota, On the robustness of code generation techniques: An empirical study on github copilot, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023: p...

    2023

  28. [38]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, J. Steinhardt, Measuring Coding Challenge Competence With APPS, (2021). https://arxiv.org/abs/2105.09938

    2021 arXiv

  29. [39]

    D. Zan, B. Chen, D. Yang, Z. Lin, M. Kim, B. Guan, Y. Wang, W. Chen, J.-G. Lou, CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation, (2022). https://arxiv.org/abs/2206.06888

    2022 arXiv

  30. [40]

    D. Lin, J. Koppel, A. Chen, A. Solar-Lezama, QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge, in: Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Huma...

    2017

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.