Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Collating ten studies, the paper concludes that ChatGPT (GPT-4-Turbo-0125) is the most accurate coding assistant among those compared, with 87.2% pass@1 on HumanEval.

desk verdict A transcription exercise that crowns a winner its own table can't support; too many internal errors for the synthesis to be trusted. read the letter →

arxiv 2411.09224 v1 pith:4KGW7KEV submitted 2024-11-14 cs.SE cs.AI

Md Kamrul Siam , Huanying Gu , Jerry Q. Cheng This is my paper · ORCID
classification cs.SEcs.AI
keywords AIcodegenerationChatGPTGeminiGitHubCopilotAlphapass@kHumanEvalcode-generationbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish which of four AI programming assistants — ChatGPT, Gemini, GitHub Copilot, and AlphaCode — gives programmers the most accurate code, based on ten studies published between 2022 and 2024. The authors combine reported pass@k and test-case pass-rate scores across benchmarks such as HumanEval, ODEX, NaturalCodeBench, Natural2Code, LeetCode, and Codeforces into a single comparison table. From that table they conclude that ChatGPT (GPT-4-Turbo-0125) is the most accurate, with an 87.2% pass@1 on HumanEval, followed by other GPT-4-Turbo versions and Gemini-1.5-Pro. They also identify pass@k and test-case pass rate as the dominant evaluation metrics, and HumanEval as the dominant benchmark. The broader point is that these tools are close enough in capability that programmers, researchers, and regulators should focus on reliability, verification, and ethics rather than on a single winner.

What carries the argument

The argument rides on a cross-study comparison table that maps each model to a metric, a benchmark, a programming language, and an accuracy score, with pass@k (the chance that at least one of k generated samples passes all tests) and test-case pass rate as the two evaluation lenses. HumanEval, a widely used benchmark of hand-written programming problems, supplies the decisive pass@1 numbers, while NaturalCodeBench, Natural2Code, ODEX, LeetCode, and Codeforces provide the supporting comparisons. The table is the central object: it converts ten independent studies into a single ordering from which the paper reads its winner.

What would settle it

Run the same held-out set of programming problems through ChatGPT (GPT-4-Turbo-0125), Gemini-1.5-Pro, AlphaCode, and GitHub Copilot under identical sampling and test conditions; if GPT-4-Turbo-0125's pass@1 is not the highest, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that ChatGPT (GPT-4-Turbo-0125) is the model that provides the most accurate code for programmers, as measured by pass@1 on HumanEval, where it reports 87.2%, with GPT-4-Turbo-1106 next at 81.7% and Gemini-1.5-Pro also performing strongly. It further claims that when multiple attempts are allowed, Gemini-Ultra performs very well on Natural2Code, and that AlphaCode and GitHub Copilot are best assessed by test-case pass rates on competitive programming platforms. From the assembled table, the paper concludes that ChatGPT (GPT-4-Turbo-0125) stands out as the most accurate model for generating code across different benchmarks. The study does not run new code-generation experiments; it synthesizes reported scores from the surveyed papers.

Load-bearing premise

The ranking assumes that a pass@1 score from one benchmark and a test-case pass rate from another measure the same underlying skill, so the largest number in the mixed table identifies the single most accurate model.

Editorial extensions

If this is right

  • Programmers who need correct code on the first try get the best reported odds from ChatGPT (GPT-4-Turbo-0125), at 87.2% pass@1 on HumanEval, according to the paper's collated numbers.
  • The close scores of GPT-4-Turbo-1106 and Gemini-1.5-Pro mean the top tier is tight, so model choice can be driven by cost, latency, or tool integration without a large accuracy penalty.
  • For assistants like AlphaCode and GitHub Copilot, test-case pass rate is the informative metric, so evaluations that report only pass@k may understate their usefulness in iterative coding workflows.
  • Because HumanEval dominates the surveyed literature, benchmark choice shapes which model looks best, and the paper's ranking is conditional on that distribution.
  • Future coding-assistant comparisons will likely need to report both pass@k and test-case pass rate, since the two metrics capture different strengths.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: HumanEval scores have been rising quickly, so an 87.2% pass@1 may not separate models on harder or more varied problem sets; on NaturalCodeBench the paper's own table shows ChatGPT (GPT-4-Turbo-0125) at 52.5% versus Gemini-1.5-Pro at 42.3%, a narrower margin than on HumanEval.
  • Editorial inference: The headline ranking mixes pass@1 scores from one benchmark with test-case pass rates from different contest platforms, so a fair head-to-head that runs all four assistants on identical problems and sampling settings could reorder them.
  • Editorial inference: A direct extension would be to report per-language rankings; the table contains Java, Python, and C++ rows, and a programmer working primarily in C++ may not see the same ordering as one working in Python.
  • Editorial inference: Since the surveyed papers use different sampling temperatures, numbers of generated samples, and test harnesses, the practical takeaway may be that prompt engineering and test-driven validation matter as much as the benchmark ranking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper is a secondary analysis that compiles published accuracy numbers for ChatGPT, Gemini, AlphaCode, and GitHub Copilot from ten recent papers. It defines three research questions: which model produces the most accurate code (RQ1), which evaluation metrics are used (RQ2), and which benchmarks are used (RQ3). The empirical core is Table 1, a heterogeneous collection of pass@1, pass@100, and test-case pass-rate scores across HumanEval, ODEX, NaturalCodeBench, Natural2Code, LeetCode, and Codeforces. Section 5.1 answers RQ1 by declaring ChatGPT (GPT-4-Turbo-0125) the 'most accurate model for generating code across different benchmarks' based on 87.2% pass@1 on HumanEval. The remaining sections discuss transformer architecture, context, response accuracy, ethics, fairness, limitations, and future work.

Significance. If the cross-benchmark ranking were valid, this would be a useful practitioner-facing snapshot of coding assistant performance. The paper's strengths are its explicit research questions, a consolidated table of recently published scores, and a discussion of ethical and fairness considerations that is often absent from benchmark reports. However, the paper is a literature synthesis rather than a new evaluation, and it provides no reproducible pipeline, raw data, or uncertainty quantification. The headline result is not significant in the statistical sense: no error bars, confidence intervals, or significance tests are reported, and the ranking is an artifact of selecting a single row from a table of incomparable metrics. The practical value is therefore limited until the comparison is placed on a protocol-matched footing.

major comments (4)
  1. [Section 5.1 and Table 1] The claim that GPT-4-Turbo-0125 is 'the most accurate model for generating code across different benchmarks' is not supported by the data as presented. The table mixes pass@1 scores (e.g., ChatGPT rows, Gemini-Pro rows) with pass@100 scores (Gemini-Ultra, Gemini-Pro from [58,66]) and test-case pass rates (AlphaCode from [48], GitHub Copilot from [55,65]). These metrics are not commensurable: pass@k changes with k by construction, and test-case pass rate measures partial credit over test cases rather than whole-problem correctness. Benchmarks also differ in difficulty and distribution. Without normalization, an explicit comparability argument, or a matched-protocol comparison, the 'winner' is simply the largest number in the table.
  2. [Section 5.1 vs. Table 1] Section 5.1 conflicts with Table 1 on two Gemini numbers. Table 1 reports Gemini-1.5-Pro HumanEval pass@1 as 0.719, but Section 5.1 reports 74.9% for the same model and benchmark. Table 1 reports Gemini-Ultra pass@100 as 0.747 on HumanEval and 0.749 on Natural2Code, but Section 5.1 states 74.7% on Natural2Code. These transcription errors mean the table cannot currently be used as the basis for RQ1; the authors must reconcile the text with the table and verify every quoted number against the cited sources.
  3. [Section 5.1 and references [57,73]] The attribution for the winning score is dubious. Section 5.1 grounds GPT-4-Turbo-0125's 87.2% HumanEval score in references [57,73]. Reference [57] is the GPT-4 Technical Report (March 2023), which predates the GPT-4-Turbo-0125 checkpoint and does not report this model or this number. Reference [73] is NaturalCodeBench; its abstract and title indicate a study of HumanEval mismatch, and it is not obvious that it contains the 87.2% pass@1 figure. The authors need to cite the exact source for each row in Table 1, with page or table numbers; otherwise the headline result cannot be verified.
  4. [Section 5.2 and Table 2] The counts '7 out of 10 papers' and '6 out of 10' papers are not traceable to the cited references. For example, the HumanEval row in Table 2 cites [22], [54], [68], [21], [55], and [63], but [54] is a Harvard Business Review commentary on ChatGPT and [55] is a GitHub Copilot empirical study; these do not appear to be papers that use HumanEval as their evaluation benchmark. The reference list also contains [n.d.] entries and duplicate citations, so the reader cannot reconstruct which papers were actually surveyed. This undermines RQ2 and RQ3 as quantitative claims.
minor comments (6)
  1. [Section 2] The paper contains several typos and inconsistent names, e.g., 'Goolgle', 'Genimi', and 'Alpha code' for AlphaCode; these should be corrected.
  2. [Section 3.3] The text equates AlphaCode with AlphaCodium, but AlphaCodium is a distinct prompt/flow-engineering method proposed by Tal Ridnik; the section heading should describe AlphaCode, and the AlphaCodium content should be separately identified.
  3. [Figure 2] Figure 2 is not self-contained: 'Code for checking fact using ChatGPT [56]' gives no explanation of what the code does or what it demonstrates.
  4. [Abstract] The abstract promises evaluation across 'Java, Python and C++', but Table 1 contains no C++ rows for ChatGPT or Gemini; the scope statement should match the data.
  5. [References] Reference list formatting is incomplete: many entries have '[n.d.]' instead of a year, one entry ([10]) has no title, and [48] duplicates [49]; the citations do not meet ACM style requirements.
  6. [Section 4] No search or inclusion criteria are given for selecting the '10 latest research papers'; a reader cannot assess selection bias or reproduce the literature search.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's conclusions are summaries of cited external benchmark results, not derivations from its own definitions or fits.

full rationale

None of the seven circularity patterns is present. The paper is a secondary evaluation: Section 4 states it analyzed 10 recent research papers, and Table 1 transcribes pass@k and test-case-pass-rate values from those external sources, with the GPT-4-Turbo-0125 HumanEval score of 87.2% attributed to [57,73]. The RQ1 answer in Section 5.1 is a restatement of the highest HumanEval number in that table; it is an evidentiary summary rather than a prediction obtained by fitting a parameter or by defining the outcome into existence. There are no fitted parameters renamed as predictions, no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The internal inconsistency between Section 5.1's 74.9% and Table 1's 0.719 for Gemini-1.5-Pro on HumanEval, and the mixing of pass@1, pass@100, and test-case pass rates across benchmarks of different difficulty, are correctness and validity flaws in the comparison; they do not make the inference circular. Because the cited external benchmarks are independent of this paper's assumptions and the paper makes no original fitted claim, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theory, parameters, or entities. It depends on the accuracy of cited benchmark numbers and on the comparability of different benchmarks, neither of which is defended or tested.

assumptions (3)
  • domain assumption Pass@1 scores from different benchmarks (HumanEval, ODEX, Natural Code Bench, Natural2Code) are comparable for ranking model accuracy
    Section 5.1 ranks models by taking the highest pass@1 across rows in Table 1 without normalizing for benchmark difficulty or evaluation protocol.
  • domain assumption The ten cited papers' reported benchmark numbers are accurate and reproducible
    Section 4 states the paper relies on ten papers, but the text contradicts Table 1 for Gemini-1.5-Pro, showing that transcription errors are present and that the underlying data were not independently verified.
  • domain assumption HumanEval pass@1 is a sufficient basis for declaring a model 'most accurate' for programmers
    Section 5.1's conclusion extrapolates from a single benchmark to overall coding accuracy, ignoring the same model's lower scores (e.g., 52.5% on Natural Code Bench).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers." pith.science (2026). https://pith.science/paper/4KGW7KEV

@misc{pith2026241109224,
  author       = {Pith},
  title        = {Pith review of: Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KGW7KEV}},
  note         = {Machine review of arXiv:2411.09224}
}
read the original abstract

Our everyday lives now heavily rely on artificial intelligence (AI) powered large language models (LLMs). Like regular users, programmers are also benefiting from the newest large language models. In response to the critical role that AI models play in modern software development, this study presents a thorough evaluation of leading programming assistants, including ChatGPT, Gemini(Bard AI), AlphaCode, and GitHub Copilot. The evaluation is based on tasks like natural language processing and code generation accuracy in different programming languages like Java, Python and C++. Based on the results, it has emphasized their strengths and weaknesses and the importance of further modifications to increase the reliability and accuracy of the latest popular models. Although these AI assistants illustrate a high level of progress in language understanding and code generation, along with ethical considerations and responsible usage, they provoke a necessity for discussion. With time, developing more refined AI technology is essential for achieving advanced solutions in various fields, especially with the knowledge of the feature intricacies of these models and their implications. This study offers a comparison of different LLMs and provides essential feedback on the rapidly changing area of AI models. It also emphasizes the need for ethical developmental practices to actualize AI models' full potential.

Figures

Figures reproduced from arXiv: 2411.09224 by the authors.

Figure 1
Figure 1. ChatGPT Architecture [11] 3.2 Gemini Gemini is a multimodal AI meaning it has the ability to compre￾hend and write content using various data modalities, such as text, images, audio, video, and code. Perhaps the most spectacular thing is that it suits almost any feature. It is worth talking about the work of Gemini on the MMLU task. MMLU is a short test that seeks to determine an AI system’s language processing comp… view at source ↗
Figure 2
Figure 2. Code for checking fact using ChatGPT [56] [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection

    cs.CR 2025-08 conditional novelty 5.0 of 10

    RTST, a two-agent moderator with an explainable Behavior ledger and per-prompt weight updates, reduced attack success rate from 12-63% to 0-17% on three jailbreak benchmarks with Gemini 2.5 Flash.

  2. Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation

    cs.SE 2025-06 conditional novelty 5.0 of 10

    Nine LLMs generated 3,373 UML methods across 90 diagrams from waste-management use cases, producing mostly valid, structurally faithful output with notable cross-model naming consensus.

  3. Can Large Language Models Improve SE Active Learning via Warm-Starts?

    cs.SE 2024-12 conditional novelty 5.0 of 10

    LLM-generated warm starts improve active learning on low- and medium-dimensional software engineering tasks but underperform Gaussian process methods on high-dimensional tasks.

  4. Software Reuse in the Generative AI Era: From Cargo Cult Towards AI Native Software Engineering

    cs.SE 2025-06 unverdicted novelty 4.0 of 10

    AI-generated code reuse is framed as a modern cargo cult practice, and the paper calls for a research agenda to make it systematic.

Reference graph

Works this paper leans on

74 extracted references · 52 canonical work pages · cited by 4 Pith papers

  1. [8]

    Gemini: All About This Zodiac Sign’s Personality Traits, Compatibility and More

    [n.d.]. Gemini: All About This Zodiac Sign’s Personality Traits, Compatibility and More. https://astrostyle .com/astrology/zodiac-signs/gemini/

    work page

  2. [48]

    Yujia Li et al. 2022. Competition-level code generation with AlphaCode. Science 378, 6624 (Dec. 2022), 1092–1097. https://doi .org/10.1126/science.abq1158

    doi:10.1126/science.abq1158 2022

  3. [57]

    OpenAI, Josh Achiam, et al . 2023. GPT-4 Technical Report. (2023). https: //doi.org/10.48550/ARXIV.2303.08774

    arXiv doi:10.48550/arxiv.2303.08774 2023

  4. [73]

    Shudan Zhang et al. 2024. NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts. (2024). https://doi .org/ 10.48550/ARXIV.2405.04520

    arXiv doi:10.48550/arxiv.2405.04520 2024

  5. [22]

    Syeda Nahida Akter et al. 2023. An In-depth Look at Gemini’s Language Abilities. (2023). https://doi .org/10.48550/ARXIV.2312.11444

    arXiv doi:10.48550/arxiv.2312.11444 2023

  6. [54]

    Ethan Mollick. [n.d.]. ChatGPT Is a Tipping Point for AI. ([n. d.]). https: //hbr.org/2022/12/chatgpt-is-a-tipping-point-for-ai

    work page 2022

  7. [68]

    Yuqing Wang and Yun Zhao. [n.d.]. Gemini in Reasoning: Unveiling Com- monsense in Multimodal Large Language Models. arXiv:2312.17661 [cs] http://arxiv.org/abs/2312.17661

    arXiv

  8. [21]

    https://blog .google/technology/ai/google-gemini-next-generation-model- february-2024/

    2024. https://blog .google/technology/ai/google-gemini-next-generation-model- february-2024/

    work page 2024

  9. [55]

    Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories . ACM, Pittsburgh Pennsylvania, 1–5. https: //doi.org/10.1145/3524842.3528470

    arXiv 2022

  10. [63]

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. [n.d.]. Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. https://doi .org/ 10.48550/arXiv.2401.08500 arXiv:2401.08500 [cs]

    arXiv doi:10.48550/arxiv.2401.08500

Show all 74 references
  1. [1]

    About GitHub Copilot Individual

    [n.d.]. About GitHub Copilot Individual. https://docs .github.com/en/copilot/ copilot-individual/about-github-copilot-individual

  2. [2]

    Build with the Gemini API

    [n.d.]. Build with the Gemini API. https://ai .google.dev/

  3. [3]

    ChatGPT vs

    [n.d.]. ChatGPT vs. Microsoft Copilot vs. Gemini: Which is the best AI chat- bot? https://www .zdnet.com/article/chatgpt-vs-microsoft-copilot-vs-gemini- which-is-the-best-ai-chatbot/

  4. [4]

    [n.d.]. Chord. https://chord .pub/article/39449/how-to-use-alphacode

  5. [5]

    Code and debug with Bard

    [n.d.]. Code and debug with Bard. https://blog .google/technology/ai/code-with- bard/

  6. [6]

    Competitive programming with AlphaCode

    [n.d.]. Competitive programming with AlphaCode. https://deepmind .google/ discover/blog/competitive-programming-with-alphacode/

  7. [7]

    Gemini - Google DeepMind

    [n.d.]. Gemini - Google DeepMind. https://deepmind .google/technologies/ gemini/

  8. [9]

    Getting started with GitHub Copilot

    [n.d.]. Getting started with GitHub Copilot. https://docs .github.com/en/copilot/ using-github-copilot/getting-started-with-github-copilot

  9. [10]

    GitHub Copilot vs

    [n.d.]. GitHub Copilot vs. ChatGPT: Which is Better for Coding in 2024? ([n. d.])

    2024

  10. [11]

    GPT-3.5 model architecture

    [n.d.]. GPT-3.5 model architecture. https://iq .opengenus.org/gpt-3-5-model/

  11. [12]

    Models comparison: OpenAI documentation

    [n.d.]. Models comparison: OpenAI documentation. https:// platform.openai.com/docs/models/overview

  12. [13]

    PLM, ChatGPT, and Large Language Model Thoughts

    [n.d.]. PLM, ChatGPT, and Large Language Model Thoughts. https: //beyondplm.com/2023/01/28/plm-chatgpt-and-large-language-model- thoughts/

    2023

  13. [14]

    The purpose, benefits, and downsides of GitHub Copilot | Proxify.io

    [n.d.]. The purpose, benefits, and downsides of GitHub Copilot | Proxify.io. https://proxify.io/articles/what-is-github-copilot

  14. [15]

    Quickstart for GitHub Copilot

    [n.d.]. Quickstart for GitHub Copilot. https://docs .github.com/en/copilot/ quickstart

  15. [16]

    Safeguarding Data Integrity and Privacy in the Age of LLMs | Sentra Blog

    [n.d.]. Safeguarding Data Integrity and Privacy in the Age of LLMs | Sentra Blog. https://www .sentra.io/blog/safeguarding-data-integrity-and-privacy-in- the-age-of-ai-powered-large-language-models-llms

  16. [17]

    The transformer architecture | Python

    [n.d.]. The transformer architecture | Python. https://campus .datacamp.com/ courses/introduction-to-llms-in-python/the-large-language-models-llms- landscape?ex=7

  17. [18]

    Understanding Transformers & the Architecture of LLMs

    [n.d.]. Understanding Transformers & the Architecture of LLMs. https:// www.mlq.ai/llm-transformer-architecture/

  18. [19]

    What Are Large Language Models (LLMs)? | IBM

    [n.d.]. What Are Large Language Models (LLMs)? | IBM. https://www .ibm.com/ topics/large-language-models

  19. [20]

    What is ChatGPT and why does it matter? Here’s what you need to know

    [n.d.]. What is ChatGPT and why does it matter? Here’s what you need to know. https://www .zdnet.com/article/what-is-chatgpt-and-why-does-it- matter-heres-everything-you-need-to-know/

  20. [23]

    K. C. Sabreena Basheer. [n.d.]. Unleashing the Power of DeepMind’s AlphaCode: Revolutionizing Code Writing. https://www .analyticsvidhya.com/blog/2023/ 12/unleashing-the-power-of-deepminds-alphacode-revolutionizing-code- writing/

    2023

  21. [24]

    Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A Neu- ral Probabilistic Language Model. In Advances in Neural Information Pro- cessing Systems , T. Leen, T. Dietterich, and V. Tresp (Eds.), Vol. 13. MIT Press. https://proceedings .neurips.cc/paper_files/paper/2000/f...

    2000

  22. [25]

    Davide Castelvecchi. 2022. Are ChatGPT and AlphaCode going to replace pro- grammers? Nature (Dec. 2022). https://doi .org/10.1038/d41586-022-04383-z

    2022 doi

  23. [26]

    Sam McKay CFA. [n.d.]. How to Use Chat GPT: A Simple Guide for Beginners | Master Data Skills + AI. https://blog .enterprisedna.co/how-to-use-chat-gpt/

  24. [27]

    Abel Chandra, Laura Tünnermann, Tommy Löfstedt, and Regina Gratz. [n.d.]. Transformer-based deep learning for predicting protein properties in the life sciences. 12 ([n. d.]), e82819. https://doi .org/10.7554/eLife.82819 Conference’17, July 2017, Washington, DC, USA Manuscript...

    doi

  25. [28]

    Jeffrey Dastin and Jeffrey Dastin. [n.d.]. Google rebrands Bard chatbot as Gemini, rolls out paid subscription. ([n. d.]). https: //www.reuters.com/technology/google-rebrands-bard-chatbot-gemini- rolls-out-paid-subscription-2024-02-08/

    2024

  26. [29]

    Jeffrey Dastin and Jeffrey Dastin. [n.d.]. Google unveils Chat- GPT rival Bard, AI search plans in battle with Microsoft. ([n. d.]). https://www.reuters.com/technology/google-opens-bard-chatbot-test- users-plans-more-ai-search-2023-02-06/

    2023

  27. [30]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2019 doi

  28. [31]

    Victor Dibia. [n.d.]. AlphaCode: Competition-Level Code Generation with Trans- former Based Architectures | Paper Review. https://victordibia .com

  29. [32]

    Thomas Dohmke. [n.d.]. GitHub Copilot is generally available to all develop- ers. https://github .blog/2022-06-21-github-copilot-is-generally-available-to- all-developers/

    2022

  30. [33]

    So what if ChatGPT wrote it?

    Yogesh K. Dwivedi et al. [n.d.]. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. 71 ([n. d.]), 102642. https://doi.org/10.1016/j.ijinfo...

  31. [34]

    Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xi- aohang Zhao. 2024. Bias of AI-generated content: an examination of news produced by large language models. Scientific Reports 14, 1 (March 2024), 5224. https://doi.org/10.1038/s41598-024-55686-2

    2024 doi

  32. [35]

    Nat Friedman. [n.d.]. Introducing GitHub Copilot: your AI pair program- mer. https://github .blog/2021-06-29-introducing-github-copilot-ai-pair- programmer/

    2021

  33. [36]

    Bill Gates. [n.d.]. The Age of AI has begun. https://www .gatesnotes.com/The- Age-of-AI-Has-Begun

  34. [37]

    Dave Gershgorn. [n.d.]. GitHub and OpenAI launch a new AI tool that generates its own code. https://www .theverge.com/2021/6/29/22555777/github-openai-ai- tool-autocomplete-code

    2021

  35. [38]

    Kristi Hines. [n.d.]. History Of ChatGPT: A Timeline Of The Meteoric Rise Of Generative AI Chatbots. https://www .searchenginejournal.com/history-of- chatgpt-timeline/488370/

  36. [39]

    Xu Huajie. [n.d.]. Github Copilot - A Groundbreaking Code Autocomplete Tool. ([n. d.]). https://doi .org/10.13140/RG.2.2.29962.24002

  37. [40]

    Senol Isci. [n.d.]. Comprehensive Guide on Evaluation of Response Generation and Retrieval in LLMs. https://medium .com/@senol.isci/comprehensive-guide- on-evaluation-of-response-generation-and-retrieval-with-llms-0cbc2adb3ae6

  38. [41]

    Frederick Jelinek. [n.d.]. Statistical methods for speech recognition . MIT Press

  39. [42]

    Ahmed Naseer Kaftan, Majid Kadhum Hussain, and Farah Hasson Naser. [n.d.]. Response accuracy of ChatGPT 3.5 Copilot and Gemini in interpreting biochemi- cal laboratory data a pilot study. 14, 1 ([n. d.]), 8233. https://doi .org/10.1038/ s41598-024-58964-1

  40. [43]

    Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. [n.d.]. Copy- right Violations and Large Language Models. https://doi .org/10.48550/ arXiv.2310.13771 arXiv:2310.13771 [cs]

    doi arXiv

  41. [44]

    Tahsin Alamgir Kheya, Mohamed Reda Bouadjenek, and Sunil Aryal. [n.d.]. The Pursuit of Fairness in Artificial Intelligence Models: A Survey. arXiv:2403.17333 [cs] http://arxiv .org/abs/2403.17333

    arXiv

  42. [45]

    Stefan Kombrink, Tomáš Mikolov, Martin Karafiát, and Lukáš Burget. 2011. Re- current neural network based language modeling in meeting recognition. InProc. Interspeech 2011. 2877–2880. https://doi .org/10.21437/Interspeech.2011-720

    2011 doi

  43. [46]

    Akshay Kulkarni, Adarsha Shivananda, Anoosh Kulkarni, and Dilip Gudivada. [n.d.]. Google Bard and Beyond . Apress, 79–99. https://doi .org/10.1007/978-1- 4842-9994-4_5

    doi

  44. [47]

    Vimal Kumar, Priyam Srivastava, Ashay Dwivedi, Ishan Budhiraja, Debjani Ghosh, Vikas Goyal, and Ruchika Arora. [n.d.]. Large-Language-Models (LLM)- Based AI Chatbots: Architecture, In-Depth Analysis and Their Performance Evaluation. In Recent Trends in Image Processing and Pat...

    2027

  45. [49]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando De Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien De Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

  46. [50]

    Louis Lippens. 2024. Computer says ‘no’: Exploring systemic bias in ChatGPT using an audit approach. Computers in Human Behavior: Artificial Humans 2, 1 (Jan. 2024), 100054. https://doi .org/10.1016/j.chbah.2024.100054

    2024

  47. [51]

    Yue Liu et al. [n.d.]. Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues. 33, 5 ([n. d.]), 1–26. https://doi .org/10.1145/ 3643674

  48. [52]

    Wei Ma, Shangqing Liu, Zhihao Lin, Wenhan Wang, Qiang Hu, Ye Liu, Cen Zhang, Liming Nie, Li Li, and Yang Liu. [n.d.]. LMs: Understanding Code Syntax and Se- mantics for Code Analysis. ([n. d.]). https://doi .org/10.48550/ARXIV.2305.12138

    doi arXiv

  49. [53]

    Dan Milmo. [n.d.]. ChatGPT reaches 100 million users two months after launch. ([n. d.]). https://www .theguardian.com/technology/2023/feb/02/chatgpt-100- million-users-open-ai-fastest-growing-app

    2023

  50. [56]

    OpenAI. [n.d.]. Code generated by ChatGPT. https://chat .openai.com Generated by ChatGPT

  51. [58]

    Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2024. Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review. (2024). https://doi .org/ 10.48550/ARXIV.2406.12655

    2024 doi arXiv

  52. [59]

    Leone Perdigão. [n.d.]. ChatGPT: a deep dive. https:// leoneperdigao.medium.com/chatgpt-a-deep-dive-1feade9c4d77

  53. [60]

    Wolfe Ph.D

    Cameron R. Wolfe Ph.D. [n.d.]. Decoder-Only Transformers: The Workhorse of Generative LLMs. https://cameronrwolfe .substack.com/p/decoder-only- transformers-the-workhorse

  54. [61]

    Salimans T

    Narasimhan K. Salimans T. & Sutskever I. Radford, A. [n.d.]. Improv- ing language understanding by generative pre-training. ([n. d.]). https: //www.mikecaptain.com/resources/pdf/GPT-1.pdf

  55. [62]

    Tal Ridnik. [n.d.]. State-of-the-art Code Generation with AlphaCodium - From Prompt Engineering to Flow Engineering. https://www .codium.ai/blog/ alphacodium-state-of-the-art-code-generation-for-code-contests/

  56. [64]

    Roumeliotis and Nikolaos D

    Konstantinos I. Roumeliotis and Nikolaos D. Tselikas. [n.d.]. ChatGPT and Open- AI Models: A Preliminary Review. 15, 6 ([n. d.]), 192. https://doi .org/10.3390/ fi15060192

  57. [65]

    Ilja Siroš, Dave Singelée, and Bart Preneel. 2024. GitHub Copilot: the perfect Code compLeeter? (2024). https://doi .org/10.48550/ARXIV.2406.11326

    2024 doi arXiv

  58. [66]

    Gemini Team, Rohan Anil, et al . 2023. Gemini: A Family of Highly Capable Multimodal Models. (2023). https://doi .org/10.48550/ARXIV.2312.11805

    2023 doi arXiv

  59. [67]

    James Vincent. [n.d.]. DeepMind says its new AI coding engine is as good as an average human programmer. https://www .theverge.com/2022/2/2/22914085/ alphacode-ai-coding-program-automatic-deepmind-codeforce

    2022

  60. [69]

    Kyle Wiggers. 2023. Google unveils AlphaCode 2, powered by Gem- ini. https://techcrunch .com/2023/12/06/deepmind-unveils-alphacode-2- powered-by-gemini/

    2023

  61. [70]

    Tong Xiao and Jingbo Zhu. [n.d.]. Introduction to Transformers: an NLP Per- spective. arXiv:2311.17633 [cs] http://arxiv .org/abs/2311.17633

    arXiv

  62. [71]

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. [n.d.]. Practical and ethical challenges of large language models in education: A systematic scoping review. 55, 1 ([n. d.]), 90–112. https://...

    doi

  63. [72]

    Abdelrahman Zayed, Gonçalo Mordido, Samira Shabanian, Ioana Baldini, and Sarath Chandar. 2024. Fairness-Aware Structured Pruning in Transformers. Proceedings of the AAAI Conference on Artificial Intelligence 38, 20 (March 2024), 22484–22492. https://doi .org/10.1609/aaai.v38i20.30256

    2024 doi

  64. [74]

    Shuyin Zhao. [n.d.]. GitHub Copilot now has a better AI model and new capa- bilities. https://github .blog/2023-02-14-github-copilot-now-has-a-better-ai- model-and-new-capabilities/

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.