Pith. sign in

REVIEW 3 major objections 6 minor 39 references

LLM-generated code fails on the same tests far more often than independence would allow, so majority-vote ensembles capture under half the reliability gain N-version programming promises.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:54 UTC pith:E3ZVZFBG

load-bearing objection First solid methodology for LLM failure independence; the correlation result holds directionally even if the exact Z/eff numbers are a bit soft on exchangeability. the 3 major comments →

arxiv 2607.02808 v1 pith:E3ZVZFBG submitted 2026-07-02 cs.SE

A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code

Rodrigo Pato Nogueira , Karthik Pattabiraman , Marco Vieira , Jo\~ao R. Campos This is my paper
classification cs.SE
keywords N-Version ProgrammingLarge Language ModelsCode GenerationFailure IndependenceBehavioral DiversitySoftware ReliabilityMajority VotingFault Correlation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

N-version programming tries to raise reliability by running several independently written versions of the same program and taking a majority vote. Its classical bottleneck was cost; large language models now make generating many versions cheap, so the open question is whether those versions actually fail independently. This paper supplies the first systematic way to measure that independence for LLM-generated code and then applies it at scale: twelve models, five languages, three prompting styles, and 224 programming problems with carefully built test suites. The measurements show that solutions from the same model are nearly structural clones, that even solutions from different models still fail the same tests far more often than a hypergeometric independence baseline predicts, and that three- and five-version majority ensembles realize only about 0.43–0.44 of the reliability improvement independence would have delivered. Manual inspection further reveals that many seemingly different failure patterns share the same root cause. Heterogeneous models help a little; language and prompting barely matter. The result is both a concrete negative finding for today’s models and a reusable yardstick for checking whether future models ever become diverse enough for true N-version fault tolerance.

Core claim

Across 224 problems, LLM-generated implementations fail on the same test cases far more often than expected under statistical independence (mean Z ≈ 2.04; 65.9 % of pairs significantly correlated). Consequently, three- and five-version majority-vote ensembles realize only 0.43 and 0.44 of the reliability gain that would be available if failures were independent, and the fraction falls below 0.3 when all versions come from the same model. Even when failure signatures look different, manual fault analysis shows they frequently share the same underlying root cause.

What carries the argument

A three-layer measurement stack: (1) CodeBLEU structural similarity, (2) a hypergeometric Z-score that compares observed co-failures against the overlap expected under independence given each version’s failure count, and (3) redundancy effectiveness—the fraction of the theoretical majority-vote reliability gain actually realized. Manual root-cause inspection closes the loop by checking whether distinct failure patterns still share the same fault.

Load-bearing premise

That short, unambiguous contest-style programming problems with high-coverage unit tests are a fair proxy for the well-specified software components where N-version programming is actually used, so the measured failure correlation will transfer beyond competitive programming.

What would settle it

Repeat the same protocol on a new set of models or on real library/system components whose specifications admit multiple correct behaviors; if three- and five-version ensembles then realize redundancy effectiveness near 1.0 (or at least well above 0.5) while the hypergeometric Z-scores drop to near zero, the independence deficit disappears and the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes the first systematic methodology for assessing failure independence of LLM-generated code for N-Version Programming, combining structural diversity (CodeBLEU), behavioral diversity (pairwise co-failure Z-scores against a hypergeometric independence baseline), N-version reliability under majority voting with a redundancy-effectiveness metric, and manual root-cause inspection. Instantiated on 224 curated contest problems, twelve models, five languages, and three prompting strategies, the study finds that same-model implementations are highly similar and that failures co-occur far more often than independence predicts (mean Z ≈ 2.04 on non-edge tests; 65.9% of pairs significant). Three- and five-version ensembles realize only 0.43 and 0.44 of the reliability gain available under independence (below 0.3 for same-model ensembles). Manual analysis indicates that even distinct failure patterns often share root causes. The authors conclude that current LLM-generated solutions do not satisfy NVP’s independence assumption, though heterogeneous models help partially, and that the methodology can track this property as models evolve.

Significance. The work addresses a timely and practically important question: whether cheap multi-version generation with LLMs can revive NVP. Its main strengths are (i) a multi-dimensional methodology that goes beyond correctness and surface diversity to failure correlation and reliability gains, (ii) a large, carefully filtered empirical study with sensitivity checks on filtering thresholds and a public dataset of prompts, code, tests, and metrics, and (iii) triangulation via structural metrics, behavioral Z-scores, voting simulations, and manual fault inspection. If the independence deficit holds, the paper usefully constrains expectations for LLM-based redundancy and supplies a reusable evaluation tool. The redundancy-effectiveness metric is a clear conceptual contribution relative to reporting absolute reliability alone.

major comments (3)
  1. [§III-B2, Eqs. (1)–(2); §V] §III-B2, Eqs. (1)–(2) and §V Construct Validity: The headline behavioral claim (mean Z ≈ 2.04; 65.9% of pairs significant) rests on a hypergeometric null that assumes test exchangeability. Residual difficulty variation within the non-edge and edge strata can inflate co-failure counts relative to the uniform null and produce positive Z even under conditional independence. The paper acknowledges the issue but does not quantify its magnitude. A load-bearing revision is needed: e.g., difficulty-stratified or per-test nulls, a simulation under known difficulty heterogeneity, or bounds showing how large residual variation would have to be to erase the reported deviation. Without this, the precise numerical claim “far more often than expected under independence” is not fully secured.
  2. [§III-B3, Eqs. (3)–(4); Table II] §III-B3, Eqs. (3)–(4) and Table II: Redundancy effectiveness uses a single average majority-pass probability raised to the power T under the same exchangeability assumption. Heterogeneous per-test pass rates make [Pmaj_avg]^T an overestimate of the product of true per-test majority probabilities, which inflates headroom and systematically lowers reported eff (0.43/0.44). Because these fractions are the abstract’s central quantitative result, the paper should either recompute headroom with a product over heterogeneous per-test majority probabilities (or a difficulty-aware model) or demonstrate that the bias is small relative to the gap from 1.0. The qualitative conclusion that gains are modest may survive; the specific 0.43/0.44 figures currently do not.
  3. [§III-C; §V External Validity] §III-C and §V External Validity: The transfer argument—that unambiguous contest problems with single correct behavior are a best-case proxy for discrete NVP components—is stated clearly but remains an untested axiom. Shared libraries, multi-correct behaviors, and specification ambiguity in real components could change correlation structure in either direction. At minimum, the paper should either (a) add a small non-contest case study (e.g., library-style or systems components with executable oracles) or (b) more sharply bound the claim to “well-specified single-behavior components” and avoid language that generalizes to NVP deployment without qualification. This is load-bearing for the practical implication that LLM-NVP is limited in the settings where NVP is used.
minor comments (6)
  1. [§III-E] §III-E Procedure step 4: typo “remaning solutions” → “remaining solutions”.
  2. [Fig. 3; Fig. 4; §III-B1] Figure 3 and Figure 4 captions and body text would benefit from explicit statement of the CodeBLEU component weights (0.1, 0.1, 0.4, 0.4) near the figures, not only in §III-B1, so readers can interpret n-gram vs AST/dataflow contributions without flipping back.
  3. [Table II] Table II: clarify in the caption that absolute reliability for homogeneous model ensembles is influenced by the filtering step (strong models contribute more fully correct solutions), so hom-vs-het comparison should be read primarily through redundancy effectiveness—as noted in the text but easy to miss when reading the table alone.
  4. [§IV-D] §IV-D: the manual analysis covers ~650 implementations from 15 problems selected for extreme/intermediate behavioral similarity. State more explicitly how many distinct root-cause categories were coded and whether coding was dual-reviewed, to strengthen reproducibility of the fault-level claims.
  5. [§II] Related work (§II): briefly contrast the proposed Z-score and redundancy-effectiveness metrics with Zheng et al.’s AST-similarity N-version performance study and with Galápagos, so the novelty claim is sharper for readers familiar with those lines.
  6. [Abstract; §I] Dataset URL is given as a tinyurl; prefer a stable archival link (Zenodo/DOI or institutional repo) in the camera-ready version for long-term reproducibility.

Circularity Check

0 steps flagged

No circularity: empirical co-failure and voting metrics are compared to independent hypergeometric/independence nulls, not defined to equal them.

full rationale

This is a self-contained empirical measurement study. Structural diversity is CodeBLEU similarity among generated solutions; behavioral diversity is the pairwise Z-score of observed co-failures against a hypergeometric independence baseline E_ij = F_i F_j / T (Eqs. 1–2) conditioned only on each implementation’s failure count; redundancy effectiveness is the fraction of the independence-headroom reliability gain that majority voting actually realizes (Eqs. 3–4). None of these quantities is fitted to force the conclusion, defined in terms of the target claim, or justified by a load-bearing self-citation uniqueness theorem. The independence baseline is a standard null, not a parameter tuned so that Z or eff must be large; observing Z ≈ 2.04 and eff ≈ 0.43–0.44 is therefore a genuine empirical deviation, not a tautology. Manual fault inspection is qualitative corroboration. Author-overlapping citation [15] concerns functional-correctness evaluation and is not load-bearing for the independence claims. Residual test-exchangeability concerns affect validity of the null, not circularity of the derivation. Score 0; steps empty.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claim is empirical and rests on standard statistical and NVP assumptions plus several study-design choices (problem filters, temperature, held-out filter tests, CodeBLEU weights). No new physical entities are postulated. Free parameters are experimental knobs that affect which solutions enter the analysis; sensitivity checks reduce but do not eliminate dependence on them.

free parameters (5)
  • problem_selection_thresholds
    Exclude if <50% of implementations pass ≥33% of tests, or >50% pass ≥95%, or reference solutions average <50 LOC; these cutoffs define the 224-problem set and are chosen by hand (sensitivity analysis reported).
  • generation_temperature
    Fixed at 0.6 with five samples per model–language–prompt cell, following prior practice rather than optimized for diversity.
  • solution_filter_held_out_tests
    Three basic unit tests used to admit solutions to diversity analysis; alternatives (1–3 tests, 10–90% pass rates) change counts but not main conclusions per authors.
  • CodeBLEU_component_weights
    Weights (0.1, 0.1, 0.4, 0.4) taken from CodeBLEU authors; structural-diversity conclusions depend on this metric choice.
  • N_version_sizes
    Reliability analysis restricted to N∈{3,5} majority voting; other N or voting rules not explored.
axioms (5)
  • domain assumption Under independence, shared failures of two implementations with F_i and F_j failures on T tests follow a hypergeometric expectation E_ij = F_i F_j / T (test exchangeability).
    Core null model for behavioral diversity and for P_maj under independence (§III-B2, III-B3); authors partially mitigate by splitting edge vs non-edge tests.
  • domain assumption Majority voting over N functionally equivalent versions is the operational definition of NVP reliability gain in this study.
    Standard NVP decision rule used to define correct(S) and redundancy effectiveness.
  • ad hoc to paper Contest problems with single correct behavior and strong unit tests are a reasonable proxy for discrete NVP components.
    Explicit external-validity premise in §III-C and §V; load-bearing for transfer of conclusions.
  • domain assumption CodeBLEU (n-gram, weighted n-gram, AST, dataflow) is an adequate measure of structural diversity within a language.
    Used for RQ1; language-specific, so cross-language structural comparison is not claimed.
  • domain assumption Models are black boxes accessed only via prompts; observed correlation reflects model behavior under fixed temperature/defaults.
    Problem framing §III-A; excludes fine-tuning or decoding interventions that might change independence.
invented entities (1)
  • redundancy_effectiveness (eff(S) = gain / headroom under independence) independent evidence
    purpose: Normalize realized majority-vote reliability gain by the gain that would be available if failures were independent, isolating diversity benefit from raw correctness.
    Defined in §III-B3 as a derived metric; not a physical entity, but a paper-specific construct central to RQ3 claims. Independent evidence is the metric's operational definition on observed pass/fail matrices.

pith-pipeline@v1.1.0-grok45 · 22032 in / 3628 out tokens · 30963 ms · 2026-07-12T06:54:22.936695+00:00 · methodology

0 comments
read the original abstract

N-Version Programming (NVP) improves software reliability by executing multiple independent implementations and combining outputs, but its adoption is limited by high cost and the assumption of failure independence, which empirical studies have challenged. Recent advances in Large Language Models (LLMs) reduce the cost of generating multiple implementations, shifting focus to whether their failures are independent. We propose the first systematic methodology to assess failure independence in LLM-generated code and apply it to 224 problems across twelve models, five languages, and three prompting strategies. We analyze both structural and behavioral diversity (i.e., whether implementations fail on the same test cases), complemented by N-version reliability analysis under majority voting and manual inspection of the generated code. Structural diversity analysis shows that implementations from the same model are highly similar, while different models produce more distinct solutions. The same trend appears in behavioral diversity, with implementations from different models showing higher diversity yet still failing on the same tests far more often than expected under independence. N-version reliability analysis reinforces this: three- and five-version ensembles realize only 0.43 and 0.44 of the reliability gain achievable under independence, dropping below 0.3 when ensembles are built from the same model. Manual fault analysis shows that even different failure patterns often share root causes. Overall, these results suggest LLM-generated solutions do not satisfy NVP's failure independence assumption, though heterogeneous models help partially. They also validate our methodology as a tool for systematically evaluating failure independence as models evolve.

Figures

Figures reproduced from arXiv: 2607.02808 by Jo\~ao R. Campos, Karthik Pattabiraman, Marco Vieira, Rodrigo Pato Nogueira.

Figure 1
Figure 1. Figure 1: Evaluation Methodology and behavioral diversity metrics, reliability gains, and explicit treatment of the different LLM-based generated configurations axes (model, language, prompt). III. METHODOLOGY [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dataset construction steps 1) Problem Selection: The first step filters out problems that are too simple or too difficult, ensuring the selected set supports meaningful analysis of failure diversity. Since the original test suites lack sufficient coverage, we first augment them using coverage-guided fuzzing with AFL++ [34], mu￾tating existing test cases and retaining those exploring new execution paths in … view at source ↗
Figure 3
Figure 3. Figure 3: Average CodeBLEU similarity across languages. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of model/prompting on CodeBLEU score. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 2 canonical work pages

  1. [1]

    Pilkington and L

    E. Pilkington and L. Aratani. (2024, Jul.) Us transportation, police and hospital systems stricken by global crowdstrike it outage. The Guardian. Accessed: 2025-11-18. [Online]. Available: https://www.theguardian.co m/technology/article/2024/jul/19/crowdstrike-microsoft-outage

    2024

  2. [2]

    L. K. Wee. (2024, Jul.) Here comes the wave of insurance claims for the crowdstrike outage. Business Insider. Accessed: 2025-11-18. [Online]. Available: https://www.businessinsider.com/businesses-claiming-losse s-crowdstrike-outage-insurance-billions-losses-cyber-policies-2024-7

    2024

  3. [3]

    D. Perell. (2020) The boeing 737 max. Accessed: 2025-11-18. [Online]. Available: https://perell.com/essay/boeing-737-max/

    2020

  4. [4]

    Brooks and H

    F. Brooks and H. Kugler,No silver bullet. April, 1987

    1987

  5. [5]

    Optimizing preventive service of software products,

    E. N. Adams, “Optimizing preventive service of software products,”IBM Journal of Research and Development, vol. 28, no. 1, pp. 2–14, 1984

    1984

  6. [6]

    Predicting the location and number of faults in large software systems,

    T. J. Ostrand, E. J. Weyuker, and R. M. Bell, “Predicting the location and number of faults in large software systems,”IEEE Transactions on Software Engineering, vol. 31, no. 4, pp. 340–355, 2005

    2005

  7. [7]

    N-version programming: A fault-tolerance approach to reliability of software operation,

    L. Chen and A. Avizienis, “N-version programming: A fault-tolerance approach to reliability of software operation,” inProc. 8th IEEE Int. Symp. on Fault-Tolerant Computing (FTCS-8), vol. 1, 1978, pp. 3–9

    1978

  8. [8]

    Dependable computing and fault-tolerance,

    J.-C. Laprie, “Dependable computing and fault-tolerance,”Digest of Papers FTCS-15, vol. 10, no. 2, p. 124, 1985

    1985

  9. [9]

    An experimental evaluation of the assumption of independence in multiversion programming,

    J. C. Knight and N. G. Leveson, “An experimental evaluation of the assumption of independence in multiversion programming,”IEEE Transactions on Software Engineering, vol. SE-12, no. 1, pp. 96–109, 1986

    1986

  10. [10]

    N-version programming for railway interlocking systems: Synchronization and voting strategy,

    O. Eris ¸, U. Yıldırım, M. S. Durmus ¸, M. T. S ¨oylemez, and S. Kurtulan, “N-version programming for railway interlocking systems: Synchronization and voting strategy,”IFAC Proceedings Volumes, vol. 45, no. 24, pp. 177–180, 2012, 13th IFAC Symposium on Control in Transportation Systems. [Online]. Available: https: //www.sciencedirect.com/science/article/...

    2012

  11. [11]

    Highly available blockchain nodes with n-version design,

    J. Ron, C. Soto-Valero, L. Zhang, B. Baudry, and M. Monperrus, “Highly available blockchain nodes with n-version design,”IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 4, pp. 4084–4097, 2024

    2024

  12. [12]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    Pith/arXiv arXiv 2021

  13. [13]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Biet al., “Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,”Nature, vol. 645, no. 8081, pp. 633–638, 2025

    2025

  15. [15]

    Beyond functional correctness: An empirical evaluation of large language models for text- to-code generation,

    R. P. Nogueira, M. Vieira, and J. R. Campos, “Beyond functional correctness: An empirical evaluation of large language models for text- to-code generation,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE), 2025, pp. 264–275

    2025

  16. [16]

    Emergent abilities of large language models,

    J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzleret al., “Emergent abilities of large language models,”arXiv preprint arXiv:2206.07682, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  18. [18]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lagoet al., “Competition- level code generation with alphacode,”Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  19. [19]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    Pith/arXiv arXiv 2021

  20. [20]

    Galapagos: Automated n-version programming with llms,

    J. Ron, D. Gaspar, J. Cabrera-Arteaga, B. Baudry, and M. Monper- rus, “Galapagos: Automated n-version programming with llms,”arXiv preprint arXiv:2408.09536, 2024

    arXiv 2024

  21. [21]

    Evaluating the diversity and quality of llm generated content,

    A. Shypula, S. Li, B. Zhang, V . Padmakumar, K. Yin, and O. Bastani, “Evaluating the diversity and quality of llm generated content,”arXiv preprint arXiv:2504.12522, 2025

    arXiv 2025

  22. [22]

    Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,

    D. Cotroneo, C. Improta, and P. Liguori, “Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE), 2025, pp. 252–263

    2025

  23. [23]

    System structure for software fault tolerance,

    B. Randell, “System structure for software fault tolerance,”SIGPLAN Not., vol. 10, no. 6, p. 437–449, Apr. 1975. [Online]. Available: https://doi.org/10.1145/390016.808467

    work page doi:10.1145/390016.808467 1975

  24. [24]

    A survey of rollback-recovery protocols in message-passing systems,

    E. N. M. Elnozahy, L. Alvisi, Y .-M. Wang, and D. B. Johnson, “A survey of rollback-recovery protocols in message-passing systems,” ACM Comput. Surv., vol. 34, no. 3, p. 375–408, Sep. 2002. [Online]. Available: https://doi.org/10.1145/568522.568525

    work page doi:10.1145/568522.568525 2002

  25. [25]

    Exception handling and tolerance of software faults,

    F. Cristian, “Exception handling and tolerance of software faults,” Software Fault Tolerance, vol. 4, 1995

    1995

  26. [26]

    Analysis of faults in an n- version software experiment,

    S. Brilliant, J. Knight, and N. Leveson, “Analysis of faults in an n- version software experiment,”IEEE Transactions on Software Engineer- ing, vol. 16, no. 2, pp. 238–247, 1990

    1990

  27. [27]

    How diversely can language models solve problems? exploring the algorithmic diversity of model-generated code,

    S. Lee, H. Chon, J. Jang, D. Lee, and H. Yu, “How diversely can language models solve problems? exploring the algorithmic diversity of model-generated code,”arXiv preprint arXiv:2503.00691, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Multi-programming language ensemble for code generation in large language model,

    T. Xue, X. Li, T. Azim, R. Smirnov, J. Yu, A. Sadrieh, and B. Pahlavan, “Multi-programming language ensemble for code generation in large language model,”arXiv preprint arXiv:2409.04114, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Can generative ai enhance the effectiveness of n-version programming?

    J. Zheng, H. Okamura, and T. Dohi, “Can generative ai enhance the effectiveness of n-version programming?” inInternational Symposium on Software Fault Prevention, Verification, and Validation. Springer, 2025, pp. 1–16

    2025

  30. [30]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...

  31. [31]

    Available: https://arxiv.org/abs/2005.14165

    [Online]. Available: https://arxiv.org/abs/2005.14165

    Pith/arXiv arXiv 2005

  32. [32]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,”arXiv preprint arXiv:2009.10297, 2020

    Pith/arXiv arXiv 2009

  33. [33]

    N. L. Johnson, A. W. Kemp, and S. Kotz,Univariate discrete distribu- tions. John Wiley & Sons, 2005

    2005

  34. [34]

    Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, V . Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss, “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,”

  35. [35]

    Available: https://arxiv.org/abs/2105.12655

    [Online]. Available: https://arxiv.org/abs/2105.12655

    Pith/arXiv arXiv

  36. [36]

    AFL++ : Combining incremental steps of fuzzing research,

    A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse, “AFL++ : Combining incremental steps of fuzzing research,” in14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association, Aug. 2020. [Online]. Available: https://www.usenix.org/conference/woot20/present ation/fioraldi

    2020

  37. [37]

    CodeContests+: High-quality test case generation for competitive programming,

    Z. Wang, S. Liu, Y . Sun, M. Ding, and H. Li, “CodeContests+: High-quality test case generation for competitive programming,” inFindings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 5576–5600. [Online...

    2025

  38. [38]

    Cosmic ray: Mutation testing tool for python,

    S. N. AS, “Cosmic ray: Mutation testing tool for python,” https://gith ub.com/sixty-north/cosmic-ray, accessed: 2026-03-24

    2026

  39. [39]

    Python documentation,

    Python Software Foundation, “Python documentation,” https://docs.pyt hon.org/3/, 2024, accessed: 2026-03-24

    2024