Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems

Asish Nelapati; Chandan Shivalingaiah; Everton Guimaraes; Nathalia Nascimento

REVIEW 5 major objections 5 minor 1 cited by

Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In a controlled benchmark of four LLMs on 150 LeetCode problems, ChatGPT's generated code is the fastest and most memory-efficient at every difficulty level, while Copilot and DeepSeek degrade as problems get harder.

desk verdict The paper collects a real dataset but the central rankings do not survive contact with its own tables, and the complexity analysis is partially self-referential. read the letter →

arxiv 2508.03931 v1 pith:4PMOIQP7 submitted 2025-08-05 cs.SE

Everton Guimaraes , Nathalia Nascimento , Chandan Shivalingaiah , Asish Nelapati This is my paper

classification cs.SE

keywords LLMcodegenerationLeetbenchmarkexecutiontimememoryusagealgorithmiccomplexityChatGPTGeminiDeepSeek

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish how four widely used coding assistants—ChatGPT, GitHub Copilot, Google Gemini, and DeepSeek—compare when given the same 150 LeetCode problems, the same structured prompt, and the same two target languages, Java and Python. Its central claim is that ChatGPT is the consistent winner: its generated solutions show the lowest execution times and memory usage at easy, medium, and hard levels, and its performance stays flat as difficulty rises, whereas Copilot and DeepSeek slow down and spread out on hard problems and Gemini needs more attempts. The study also finds a language trade-off that holds across every model: Java solutions run faster, Python solutions use less memory. If the comparison holds, developers choosing an LLM for performance-sensitive coding get head-to-head evidence instead of single-model benchmarks.

What carries the argument

The experimental pipeline: a fixed role-based prompt ('act as a software developer') given to each model for the same 150 problems, submission of the generated code to LeetCode's online judge for execution-time and memory readings, and a complexity-ranking table (Table IV) that converts each LLM's self-reported Big-O notation into a numbered scale—$\mathcal{O}(1) = 1$ up to $\mathcal{O}(n!) = 9$—so that optimization can be averaged and significance-tested. The ranking table is the device that turns subjective complexity labels into comparable scores, and the online-judge readings are the device that turns 'efficiency' into numbers.

What would settle it

Re-run the same 150 problems with identical prompts and fixed model versions, submitting each solution multiple times to LeetCode: if ChatGPT's margin over the runner-up (e.g., 29.43 ms versus 102.23 ms on medium problems) falls within the run-to-run jitter of identical code, the speed ranking is measurement noise, not model capability. Independently, verify each LLM's self-reported Big-O with a static complexity analyzer to confirm or overturn the RQ3 optimization ordering.

Watch

Extended reading notes

Core claim

The paper's central claim: on 150 LeetCode problems with an identical prompt, ChatGPT generates the most time- and memory-efficient code at every difficulty—13.47 ms easy, 29.43 ms medium, 47.56 ms hard, versus Copilot's 342.98 ms at hard—and stays stable with minimal outliers. Copilot and DeepSeek grow more variable as problems get harder; Gemini's success rate drops to 68% with more attempts. Across all models, Java runs faster while Python uses less memory, and ChatGPT's solutions cluster in the lowest time-complexity ranks, confirmed by a Friedman test (8.20, $p = 0.042$). The authors position this as the first comparison of LLM-generated code performance across models on LeetCode.

Load-bearing premise

The rankings rest on two unverified assumptions: LeetCode's per-submission runtime and memory readings are stable enough to compare across models, and each LLM's own stated efficiency grade (its Big-O label) accurately describes the code it generated.

Editorial extensions

If this is right

Developers choosing an LLM for performance-sensitive code get a direct signal: ChatGPT-generated solutions hold their speed and memory on hard problems, while Copilot and DeepSeek degrade as difficulty rises.
The Java-faster, Python-leaner trade-off holds for every model tested, so language choice can be made independently of model choice when one resource matters more than the other.
Because the accuracy of any given LLM's self-reported Big-O varies with difficulty and language (RQ4), generated complexity annotations should be verified rather than trusted.
The methodology—structured prompt, LeetCode judge, ranked complexity scale—can be re-applied to new models, languages, and problem sets, so future LLMs can be slotted into the same comparison.
Gemini's rising attempt count on hard problems means interactive cost, not just solution quality, should factor into model selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

LeetCode's per-submission runtime readings are noisy; since the paper does not report run-to-run variance for identical code, the large ChatGPT-versus-Copilot gap on hard problems is the trustworthy signal, while small gaps (e.g., Gemini versus DeepSeek) may be noise.
The ranking table flattens real distinctions—$\mathcal{O}(n^2)$ and $\mathcal{O}(n^2 \log n)$ share rank 5, and all exponential classes share rank 8—so a finer scale could reorder the models on the hardest problems.
The paper ranks each LLM's self-reported Big-O without independent verification; checking those labels against a static complexity analyzer would test whether the 'most optimized' claim survives scrutiny.
The replication package is promised only upon acceptance, so the headline numbers are not yet independently checkable by other researchers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: this is a straightforward benchmark of four LLMs on 150 LeetCode problems, in Java and Python, with runtime/memory from LeetCode's judge and Big-O labels taken from the models' own responses. The idea is fine and the topic is timely, but the paper as written has a load-bearing internal inconsistency in its descriptive statistics. The efficiency rankings that form its headline result are not supported by the numbers as reported.

What it does well: the experiment design is understandable, the prompt is described, and the dual-language comparison is a sensible angle. The qualitative finding that Java solutions run faster while Python solutions use less memory is consistent with what one would expect, and the attempt-count data on Gemini is a small useful addition. There is real work here.

The soft spots are not minor. First, Tables II and III describe the same 300 submissions per model (150 problems × two languages), but they imply incompatible overall means for ChatGPT. Weighting the difficulty table gives ~30 ms and ~44 MB; weighting the language table gives ~18 ms and ~31 MB. The gap is far beyond rounding, and the method section does not say that one table covers a different subset. At least one table is wrong or mislabeled. That alone undermines RQ1 and RQ2.

Second, RQ3's optimization ranking depends on the models' self-reported Big-O complexity with no independent check, and RQ4's "accuracy" has no defined ground truth. The paper acknowledges Big-O is approximate but doesn't address the circularity. The novelty claim over Coignion et al. is also overstated; they cite that study as inspiration but don't specify what this paper adds beyond a different problem subset and more models.

The statistical reporting also contradicts itself: memory differences are called non-significant at all levels and then significant at easy/medium in consecutive sentences. A careful copy-edit won't fix that; the analysis needs to be redone and the tables reconciled.

Who is this for: someone tracking LLM code-generation benchmarks might glance at it, but I wouldn't rely on the rankings. It deserves a serious referee in principle, because the data is real and the flaws are fixable, but only if the authors can reconcile the tables and define a non-circular complexity evaluation. As is, I would not cite it.

My advice: send it to review only if you expect the authors to do a major revision; otherwise desk reject with detailed feedback. I'd lean toward giving them the chance to fix it, but not accepting the current version.

Referee Report

5 major / 5 minor

Summary. The paper reports a controlled benchmark of four LLMs (ChatGPT/GPT-4o, Copilot/GPT-4, Gemini 1.0, DeepSeek Coder 2.5) on 150 LeetCode problems, with equal numbers of easy, medium, and hard problems, generating solutions in both Java and Python. The authors evaluate execution time, memory usage, and Big-O complexity, and claim that ChatGPT is consistently the most time- and memory-efficient model, that Java solutions are faster while Python solutions are more memory-efficient, and that complexity rankings identify ChatGPT as producing the most optimized solutions. The paper also claims to be the first to compare LLM-generated code performance across models on LeetCode problems.

Significance. A comparative benchmark of this size—150 problems, two languages, four models—would be a useful contribution to the empirical literature on LLM code generation, if the measurements were reliable. The experimental design has reasonable elements: a standardized prompt, a three-attempt limit, and use of LeetCode's judge. However, the manuscript currently contains mutually inconsistent summary statistics in its central tables, relies on self-reported Big-O labels without independent verification, and reports contradictory statistical results. Because the headline conclusions rest directly on these elements, the paper's current contribution is not supported by the evidence it presents. No replication package or raw data is supplied at review time; the package is only promised upon acceptance.

major comments (5)

[Section IV-A/IV-B (Tables II and III)] The design stated in Sections III-B1 and III-B3 is that each model solves the same 150 problems in both Java and Python. Under that design, the difficulty-stratified means in Table II and the language-stratified means in Table III must imply the same overall mean. They do not. For ChatGPT, Table II implies an overall mean execution time of (13.47+29.43+47.56)/3 = 30.15 ms and memory of (40.78+43.02+46.89)/3 = 43.56 MB, while Table III implies (6.85+29.63)/2 = 18.24 ms and (46.76+15.87)/2 = 31.32 MB. The gaps are ~12 ms and ~12 MB. For Gemini the discrepancy is far larger: 125.67 ms vs 15.80 ms. Even if Table II were meant to be Java-only, the ChatGPT Java mean of 6.85 ms in Table III is below the Easy-level mean of 13.47 ms in Table II, which is impossible for a weighted average of the same data. Since RQ1 and RQ2 rank models directly from these tables, the reported efficiency rankings a
[Section III-B4, Table IV, RQ3/RQ4] The complexity analysis is circular. The prompt in Section III-B3 instructs the LLM to calculate the Big-O complexity, and Section III-B4 says complexity is 'derived from analyzing the generated code' without any independent computation. RQ3's optimization ranking therefore uses each model's self-reported complexity label as both the input and the outcome: a model that labels its own solution O(n) is ranked better even if the code is actually O(n^2). RQ4 calls this 'accuracy' but provides no ground-truth complexity definition. Table IV's rank mapping is also arbitrary: O(n^2 log n) is ranked the same as O(n^2), O(4^n) is ranked the same as O(2^n) and O(3^n), and O(n81) (presumably O(n^81)) is ranked below O(2^n). These choices directly drive the conclusion that ChatGPT is the 'most time-optimized' model.
[Section IV-A2] The memory-usage paragraph contradicts itself in consecutive sentences: first it says 'ANOVA and Kruskal-Wallis tests indicated no significant differences at any difficulty level (p>0.05 across all tests)', then it says 'Kruskal-Wallis tests identified notable differences at the easy (H=20.50, p<0.001) and medium (H=17.63, p<0.01) levels.' These cannot both be true. The execution-time results are also mixed—ANOVA is significant only at hard level while Kruskal-Wallis is significant only at easy level—yet the text uses them to support a uniform ChatGPT advantage without reconciling the inconsistency. The statistical analysis needs to be redone and reported coherently.
[Section IV-D] RQ4 promises to analyze the 'accuracy of LLM-generated complexity metrics' using mixed-effects models, but no model coefficients, p-values, or accuracy definitions are reported. Table V counts submissions in complexity classes, and Figures 8-11 show distributions, but there is no reference accuracy measure (e.g., expert labels or a deterministic complexity analyzer). The claim that the paper evaluates accuracy is therefore unsupported.
[Section III-B3 / Section VI-A] The internal-validity section states that 'we perform multiple test iterations per problem, averaging the results to lessen server fluctuations on LeetCode,' but the experimental procedure in Section III-B3 describes only up to three attempts per problem/language to obtain a correct solution and does not state how many accepted submissions were used for each execution-time and memory reading. Because LeetCode judge measurements are known to be noisy, the number of submissions per data point is load-bearing for RQ1 and RQ2. Please clarify whether each reported mean is based on one submission, multiple submissions, or multiple accepted attempts.

minor comments (5)

[Section II-B] Typo: 'LeetCod' should be 'LeetCode'. Also, Section II-C's claim to be 'the first to investigate the differences in the performance of LLM-generated code across different models on LeetCode problems' is difficult to square with the cited reference [7], which is titled 'A performance study of LLM-generated code on LeetCode'; the novelty claim should be tempered or justified.
[Table IV] The notation 'O(n81)' is unclear; if O(n^81) is intended, format it accordingly. The rank ordering should be justified with a standard growth-rate hierarchy, since several adjacent entries are grouped at the same rank despite being asymptotically very different.
[References] Reference [23] is malformed: 'Q IHAO ZHU ET AL ., YEAR =2024, E. A. P. U.' should be a complete citation for DeepSeek-Coder-V2.
[Throughout] The paper alternates between 'GPT' and 'ChatGPT' (e.g., Section V), which is confusing because Copilot also uses a GPT architecture. Use one consistent label for each model.
[Section III-C] The replication package is promised only upon acceptance. Given the empirical nature of the study and the need to verify the inconsistent tables, making the raw data and analysis scripts available at review time would materially improve confidence.

Circularity Check

2 steps flagged · score 6.0 of 10

RQ3/RQ4 complexity findings are self-referential: the LLMs' own Big-O labels are used both as the data and as the measure of 'optimization'/'accuracy'; runtime/memory parts are external but Tables II/III are internally inconsistent.

self definitional [Section III-B3 / III-B4 and Section IV-C (RQ3)]
"the LLM is instructed to calculate the Big O complexity, which prompts it to assess the efficiency of its solution—an essential factor in our performance evaluation. ... First, we standardized the time complexities generated by the LLMs and ranked them giving a numerical value to each complexity."

The optimization ranking is built from the same LLMs' self-reported Big-O labels rather than from an independent complexity analysis of the generated code. The conclusion that ChatGPT is 'the most time-optimized model' therefore reduces to the observation that ChatGPT tends to assign low complexity ranks to its own solutions. The ranking is defined by the models' outputs, so the result is guaranteed by construction to measure self-labeling, not actual asymptotic efficiency.
self definitional [Section III-A (RQ4), III-B4, IV-D]
"RQ4: How does problem difficulty impact the accuracy of LLM-generated complexity metrics in Java and Python? ... We analyzed the accuracy of LLM-generated complexity metrics by problem difficulty and programming language using mixed-effects models."

RQ4's 'accuracy' is analyzed without any independent ground-truth complexity labels. The data used (e.g., Table V) are exactly the LLM-generated Big-O metrics the question is about, so the analysis is a re-tabulation of self-reported labels. Nothing in the design compares the models' labels to a reference complexity computed from the code, so 'accuracy' is not an external measure; it is the paper's name for the distribution of the models' own outputs.

full rationale

The execution-time and memory measurements (RQ1/RQ2) are taken from LeetCode's online judge, an external oracle, so those empirical comparisons are not circular by themselves. The self-referential part is the complexity contribution: Section III-B3 instructs each LLM to generate its own Big-O label, and Section IV-C then ranks models using 'the time complexities generated by the LLMs,' so the RQ3 finding that ChatGPT is the most time-optimized model is equivalent to saying ChatGPT assigns itself low complexity ranks. RQ4 is similarly circular because 'accuracy of LLM-generated complexity metrics' is never checked against an independent ground truth; Table V merely tabulates the same self-reported labels. No load-bearing self-citations are present, and the runtime/memory data are externally sourced, though the promised replication package is not yet public. Separately, the reported numbers in Table II and Table III imply mutually inconsistent overall means for ChatGPT (30.15 ms / 43.56 MB vs 18.24 ms / 31.32 MB), which undermines the reliability of the external efficiency ranking, but that is an internal-validity problem rather than a circularity. Overall, the central complexity claims reduce to the models' own self-assessments, giving a partial but real circularity score of 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No formal model; the contribution rests on measurement assumptions and a hand-assigned complexity ranking. The complexity ranking and attempt limit are the main hand-chosen degrees of freedom.

free parameters (2)

Complexity rank mapping (Table IV)
Hand-assigned ordinal ranks to Big-O classes (e.g., O(log n2) rank 2, O(n81) rank 7); arbitrary ordering of exponential classes affects RQ3 averages and Friedman test.
Attempt limit of 3 per problem/language
Caps measured success rates and may censor slow/incorrect attempts, affecting runtime means and attempt statistics.

assumptions (4)

domain assumption LeetCode online judge runtime/memory measurements are reliable and comparable across submissions
Used as ground truth for RQ1/RQ2; no control for server load or repeated-run variance.
domain assumption LLM-self-reported Big-O complexity is accurate
RQ3/RQ4 treat generated complexity labels as data without external verification.
domain assumption The four models' outputs are directly comparable despite different versions and dates
Table I lists GPT-4o, GPT-4, Gemini 1.0, DeepSeek Coder 2.5 with different release timing and parameter counts; no timestamp of queries.
domain assumption The curated 150 problems are representative of software engineering tasks
External validity claim based on LeetCode's structure; but proprietary dataset not shared.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems." pith.science (2026). https://pith.science/paper/4PMOIQP7

@misc{pith2026250803931,
  author       = {Pith},
  title        = {Pith review of: Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PMOIQP7}},
  note         = {Machine review of arXiv:2508.03931}
}

read the original abstract

Large Language Models (LLMs) like ChatGPT, Copilot, Gemini, and DeepSeek are transforming software engineering by automating key tasks, including code generation, testing, and debugging. As these models become integral to development workflows, a systematic comparison of their performance is essential for optimizing their use in real world applications. This study benchmarks these four prominent LLMs on one hundred and fifty LeetCode problems across easy, medium, and hard difficulties, generating solutions in Java and Python. We evaluate each model based on execution time, memory usage, and algorithmic complexity, revealing significant performance differences. ChatGPT demonstrates consistent efficiency in execution time and memory usage, while Copilot and DeepSeek show variability as task complexity increases. Gemini, although effective on simpler tasks, requires more attempts as problem difficulty rises. Our findings provide actionable insights into each model's strengths and limitations, offering guidance for developers selecting LLMs for specific coding tasks and providing insights on the performance and complexity of GPT-like generated solutions.

Figures

Figures reproduced from arXiv: 2508.03931 by the authors.

**Figure 1.** Prompt Design Overview online judge system, where automated tests evaluated their correctness, execution time, and memory usage. This standardized evaluation ensured consistency in assessing each solution’s performance metrics. We collected the following metrics for each generated solution: • Execution Time: The time taken by the solution to run, as reported by LeetCode’s automated testing system. • Memory Usage: T… view at source ↗

**Figure 2.** Runtime Distribution by Difficulty and LLM [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Memory Distribution by Difficulty and LLM [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Comparison of Execution Time by Language for Each LLM [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Comparison of Memory Usage by Language for Each LLM [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Time Complexity Distribution by LLM and Difficulty [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Space Complexity Distribution by LLM and Difficulty [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Standardized Time Complexity for ChatGPT [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Standardized Time Complexity for Gemini lems also contribute to O(1), though less prominently. The model exhibits a balanced spread across O(n) space complexity, spanning all difficulty levels and languages, especially medium-difficulty problems in Java. This pattern …

**Figure 10.** Figure 10: Standardized Space Complexity for Copilot [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

**Figure 12.** Figure 12: Success Rate of LLMs by Problem Difficulty Level [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

**Figure 13.** Figure 13: Trend of Average Number of Attempts per Difficulty Level [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Unlocking LLM Code Correction with Iterative Feedback Loops
cs.SE 2026-06 unverdicted novelty 6.0 of 10

Empirical evaluation finds reasoning LLMs improve code correction across iterations using execution feedback and outperform non-reasoning models, with syntactic and runtime errors easier to fix than logical ones.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages · cited by 1 Pith paper

[1]

Learning to represent edits

A LLAMANIS , M., B ROCKSCHMIDT , M., AND KHADEMI , M. Learning to represent edits. In International Conference on Learning Represen- tations

work page
[2]

Recode: Benchmarking language models for code repair and generation

A NONYMOUS . Recode: Benchmarking language models for code repair and generation. https://github.com/example/recode, 2023. ReCode Benchmark

work page 2023
[3]

B., AND POLIKARPOVA , N

B ARKE , S., J AMES , M. B., AND POLIKARPOVA , N. Grounded copilot: How programmers interact with code-generating models, 2022

work page 2022
[4]

M., G EBRU , T., M CMILLAN -MAJOR , A., AND SHMITCHELL , S

B ENDER , E. M., G EBRU , T., M CMILLAN -MAJOR , A., AND SHMITCHELL , S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610–623

work page 2021
[5]

Supersonic: Learning to generate source code optimizations in c/c++

C HEN , Z., F ANG , S., AND MONPERRUS , M. Supersonic: Learning to generate source code optimizations in c/c++. IEEE Transactions on Software Engineering (2024), 1–17

work page 2024
[6]

Codereval: A benchmark for evaluating code generation models, 2022

C ODER EVAL. Codereval: A benchmark for evaluating code generation models, 2022. Benchmark for evaluating code generation models

work page 2022
[7]

A performance study of llm-generated code on leetcode

C OIGNION , T., Q UINTON , C., AND ROUVOY, R. A performance study of llm-generated code on leetcode. In 28th International Conference on Evaluation and Assessment in Software Engineering (EASE) (Salerno, Italy, 2024), ACM

work page 2024
[8]

E., AND COMBE - MALE , B

D ¨ODERLEIN , J.-B., A CHER , M., K HELLADI , D. E., AND COMBE - MALE , B. Piloting copilot and codex: Hot temperature, cold prompts, or black magic? SSRN Electronic Journal (2023)

work page 2023

Show all 30 references

[9]

ET., M. C. Evaluating large language models trained on code. https://github.com/openai/human-eval, 2021. OpenAI HumanEval Benchmark

2021
[10]

Evaluating large language models trained on code, 2021

ET AL ., C. Evaluating large language models trained on code, 2021. OpenAI Codex

2021
[11]

Codebert: A pre-trained model for programming and natural languages

F ENG , Z., G UO, D., TANG , D., D UAN, N., L IU, X., G ONG , M., S HOU , L., Z HOU , M., Y IN, J., AND JIANG , D. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EM...

2020
[12]

Z., C LEMENT , C

G ARG , S., M OGHADDAM , R. Z., C LEMENT , C. B., S UNDARESAN , N., AND WU, C. Deepperf: A deep learning-based approach for improving software performance, 2022

2022
[13]

T., AND MORGAN , E

J ESSE , K., A HMED , T., D EVANBU , P. T., AND MORGAN , E. Large language models and simple, stupid bugs, 2023

2023
[14]

Beyond code generation: An observational study of chatgpt usage in software engineering practice

K HOJAH , R., M OHAMAD , M., L EITNER , P., AND GOMES DE OLIVEIRA NETO , F. Beyond code generation: An observational study of chatgpt usage in software engineering practice. Proceedings of the ACM on Software Engineering 1 , FSE (2024), Article 81

2024
[15]

Leetcode: The world’s leading online programming learn- ing platform

L EET CODE. Leetcode: The world’s leading online programming learn- ing platform. https://leetcode.com/, 2024. Accessed: 2024-11-06

2024
[16]

S., W ANG , Y., AND ZHANG , L

L IU, J., X IA, C. S., W ANG , Y., AND ZHANG , L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

2023
[17]

Ai software reliability: Concepts and related domains

L IU, Y., ET AL . Ai software reliability: Concepts and related domains. In 2023 IEEE International Conference on Artificial Intelligence and Industrial Applications (AIIIP) (2023), pp. 287–292

2023
[18]

B., D RAIN , D., B ING , L., Z HOU , M., AND SUNDARE - SAN , N

L U, S., G UO, D., R EN, S., S VYATKOVSKIY , A., B LANCO , A., CLEMENT , C. B., D RAIN , D., B ING , L., Z HOU , M., AND SUNDARE - SAN , N. Codexglue: A benchmark dataset and open challenge for code intelligence. Proceedings of the 2021 Conference on Empirical Methods in Natur...

2021
[19]

Artificial in- telligence versus software engineers: An evidence-based assessment focusing on non-functional requirements

N ASCIMENTO , N., A LENCAR , P., AND COWAN, D. Artificial in- telligence versus software engineers: An evidence-based assessment focusing on non-functional requirements. University of Waterloo (2023)

2023
[20]

H., S HAIKH , A., M UMTAZ , A., A SLAM , A., AND ILYAS, A

N AVEED , H., A HMAD , A., A HMAD , M., T ANVEER , W., J AVED, S. H., S HAIKH , A., M UMTAZ , A., A SLAM , A., AND ILYAS, A. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 (2023)

2023 arXiv
[21]

An empirical evaluation of github copilot’s code suggestions

N GUYEN , N., AND NADI , S. An empirical evaluation of github copilot’s code suggestions. In 19th International Conference on Mining Software Repositories (MSR) (Pittsburgh, PA, USA, 2022), ACM

2022
[22]

Gpt-4 technical report, 2024

O PENAI. Gpt-4 technical report, 2024

2024
[23]

Q IHAO ZHU ET AL ., YEAR =2024, E. A. P. U. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

2024
[24]

Choose your pro- gramming copilot: A comparison of the program synthesis performance of github copilot and genetic programming

S OBANIA , D., B RIESCH , M., AND ROTHLAUF , F. Choose your pro- gramming copilot: A comparison of the program synthesis performance of github copilot and genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference (2022), pp. 1019– 1027

2022
[25]

Stratascratch: Real-world data science coding chal- lenges

S TRATASCRATCH . Stratascratch: Real-world data science coding chal- lenges. https://www.stratascratch.com/, 2022. Benchmark platform for data science and coding challenges

2022
[26]

K., F U, S., AND SUNDARESAN , N

S VYATKOVSKIY , A., D ENG , S. K., F U, S., AND SUNDARESAN , N. Intellicode compose: Code generation using transformer models. Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022), 3036–3046

2022
[27]

Expectation vs

V AITHILINGAM , P., Y ANG , G., AND ZHU, H. Expectation vs. expe- rience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New York, NY , USA, 2022), Association...

2022
[28]

V., AND VAUGHAN , J

V ASCONCELOS , H., B ANSAL , G., F OURNEY , A., L IAO, Q. V., AND VAUGHAN , J. W. Generation probabilities are not enough: Uncertainty highlighting in ai code completions. ACM Transactions on Computer- Human Interaction (Oct. 2024)

2024
[29]

W HITE , J., F U, Q., H AYS, S., S ANDBORN , M., S PENCER -SMITH , J., AND SCHMIDT , D. C. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2302.11382 (2023)

2023 arXiv
[30]

F., A LON , U., AND NEUBIG , G

X U, F. F., A LON , U., AND NEUBIG , G. Polycoder: A language model for code completion. arXiv preprint arXiv:2202.13169 (2022)

2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Learning to represent edits

A LLAMANIS , M., B ROCKSCHMIDT , M., AND KHADEMI , M. Learning to represent edits. In International Conference on Learning Represen- tations

work page

[2] [2]

Recode: Benchmarking language models for code repair and generation

A NONYMOUS . Recode: Benchmarking language models for code repair and generation. https://github.com/example/recode, 2023. ReCode Benchmark

work page 2023

[3] [3]

B., AND POLIKARPOVA , N

B ARKE , S., J AMES , M. B., AND POLIKARPOVA , N. Grounded copilot: How programmers interact with code-generating models, 2022

work page 2022

[4] [4]

M., G EBRU , T., M CMILLAN -MAJOR , A., AND SHMITCHELL , S

B ENDER , E. M., G EBRU , T., M CMILLAN -MAJOR , A., AND SHMITCHELL , S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610–623

work page 2021

[5] [5]

Supersonic: Learning to generate source code optimizations in c/c++

C HEN , Z., F ANG , S., AND MONPERRUS , M. Supersonic: Learning to generate source code optimizations in c/c++. IEEE Transactions on Software Engineering (2024), 1–17

work page 2024

[6] [6]

Codereval: A benchmark for evaluating code generation models, 2022

C ODER EVAL. Codereval: A benchmark for evaluating code generation models, 2022. Benchmark for evaluating code generation models

work page 2022

[7] [7]

A performance study of llm-generated code on leetcode

C OIGNION , T., Q UINTON , C., AND ROUVOY, R. A performance study of llm-generated code on leetcode. In 28th International Conference on Evaluation and Assessment in Software Engineering (EASE) (Salerno, Italy, 2024), ACM

work page 2024

[8] [8]

E., AND COMBE - MALE , B

D ¨ODERLEIN , J.-B., A CHER , M., K HELLADI , D. E., AND COMBE - MALE , B. Piloting copilot and codex: Hot temperature, cold prompts, or black magic? SSRN Electronic Journal (2023)

work page 2023