Measuring the Runtime Performance of C++ Code Written by Humans using GitHub Copilot

Daniel Erhabor; Meiyappan Nagappan; Samer Al-Kiswany; Sreeharsha Udayashankar

Measuring the Runtime Performance of C++ Code Written by Humans using GitHub Copilot

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.06439 v2 pith:VJJ6EQCR submitted 2023-05-10 cs.SE

Daniel Erhabor , Sreeharsha Udayashankar , Meiyappan Nagappan , Samer Al-Kiswany This is my paper · ORCID

classification cs.SE

keywords copilotperformanceruntimecodedevelopersgithubparticipantsprogramming

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

GitHub Copilot is an artificially intelligent programming assistant used by many developers. While a few studies have evaluated the security risks of using Copilot, there has not been any study to show if it aids developers in producing code with better runtime performance. We evaluate the runtime performance of C++ code produced when developers use GitHub Copilot versus when they do not. To this end, we conducted a user study with 32 participants where each participant solved two C++ programming problems, one with Copilot and the other without it and measured the runtime performance of the participants' solutions on our test data. Our results suggest that using Copilot may produce C++ code with (statistically significant) slower runtime performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation
cs.SE 2025-01 conditional novelty 5.0 of 10

An LLM code optimizer using control-flow-graph differences and retrieved examples reports 7.3% average runtime reduction on 116 C++ programs versus zero-shot GPT-4o.

Pith tools