REVIEW 7 cited by
GitHub Copilot AI pair programmer: Asset or Liability?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Automatic program synthesis is a long-lasting dream in software engineering. Recently, a promising Deep Learning (DL) based solution, called Copilot, has been proposed by OpenAI and Microsoft as an industrial product. Although some studies evaluate the correctness of Copilot solutions and report its issues, more empirical evaluations are necessary to understand how developers can benefit from it effectively. In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks. For the former, we assess the performance and functionality of Copilot in solving selected fundamental problems in computer science, like sorting and implementing data structures. In the latter, a dataset of programming problems with human-provided solutions is used. The results show that Copilot is capable of providing solutions for almost all fundamental algorithmic problems, however, some solutions are buggy and non-reproducible. Moreover, Copilot has some difficulties in combining multiple methods to generate a solution. Comparing Copilot to humans, our results show that the correct ratio of humans' solutions is greater than Copilot's suggestions, while the buggy solutions generated by Copilot require less effort to be repaired.
Forward citations
Cited by 7 Pith papers
-
CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models
A new 12,662-prompt cybersecurity jailbreak benchmark and an obfuscation-based jailbreak method that reportedly outperforms prior methods on AdvBench.
-
GHIssuemarket: A Sandbox Environment for SWE-Agents Economic Experimentation
The authors present GHIssueMarket, an open-source sandbox that simulates a peer-to-peer auction marketplace for SWE-agents, and propose a new research field called Intelligent Software Engineering Economics.
-
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
ChatGPT helps students pass programming tests but not understand concepts; both heavy reliance and minimal use lead to weaker learning.
-
Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights
SBC is a hybrid metric that reverse-generates requirements from LLM-written code and scores their semantic, lexical, and completeness match to the original requirement, but its validity is not established.
-
On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code
Appending LLM-generated misleading code snippets to coding prompts lowers pass@1 by 12-34% for open-source and 3-24% for commercial code LLMs; guided prompting restores only part of the loss.
-
Experience with GitHub Copilot for Developer Productivity at Zoominfo
At ZoomInfo, GitHub Copilot suggestions were accepted 33% of the time with 20% of lines accepted, alongside 72% developer satisfaction, in line with earlier vendor and industry reports.
-
Feature-Factory: Automating Software Feature Integration Using Generative AI
The paper describes an LLM-driven pipeline for automated feature integration and demonstrates it with a single toy 'add logging' example, without rigorous evaluation.
Discussion (0). Continue with ORCID to comment.