A new code-writing data analysis benchmark shows human experts outperforming a frontier LLM on average with lower performance variance.
hub Contested
ArXiv , year=
Contested. 1 Pith paper cite this work to dispute or refute its claims.
38
Pith papers citing it
6
external citations · Pith
Contested
1 dispute or refute
abstract
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
hub tools
citation-role summary
background 4
contradiction 1
citation-polarity summary
representative citing papers
TDDev automates the full TDD loop for web app generation from requirements, delivering 34-48 percentage point quality gains and zero manual intervention in user studies.
Developers use LLMs like ChatGPT mainly for knowledge acquisition and code generation at the detailed design level, reporting benefits such as better technology selection and early flaw detection alongside limitations like lengthy outputs, incorrect code, and hallucinations.
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
Coding agents reached 22-29% adoption in GitHub projects within months of release, with agent-assisted commits larger and focused on features and bug fixes.
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
Longitudinal panel study of 802 developers shows an enterprise AI coding mandate doubled per-capita merged pull requests to 2.09x baseline, with gains associated with AI adoption and accumulated use while review processes automated.
Study of 930k+ agent PRs shows repository explains ~50% of integration friction variance, with agents concentrating it twice as much as humans (ICC 0.30 vs 0.16) after controls.
Using CORE-Bench as a case study, the paper shows that saturated benchmarks can still deliver insights on efficiency, reliability, model-scaffold differences, and human collaboration even after accuracy plateaus, and introduces improved benchmark versions plus a small randomized experiment demonstra
Mixed-methods study creates taxonomy of AI IDE rules from 7310 instances, analyzes evolution drivers, and reports that rule updates raise average artifact compliance from 49.14% to 72.13%.
The web's anti-bot regime should be replaced by a framework that presumptively lets user-authorized AI agents act for their principals, requires platforms to disclose access policies, and permits agent blocking only when proportionate to concrete harms.
AI-assisted requirements engineering in industry is constrained more by tool integration and organisational readiness than by the capabilities of the AI tools themselves.
Survey of 162 vibe coders finds perceptions of AI code quality similar across experience levels but motivations, interaction styles, and quality assurance practices diverge, revealing a perception-action gap.
Proposes a three-step benchmark design method (define work activity, specify tested setting, score work product) derived from work studies and O*NET, demonstrated via three case analyses.
Case study reports one staff engineer with four AI agents delivering a four-person-scoped brownfield project in half the planned time under Spec-Driven Development, with high code acceptance and major cost savings.
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
A game-theoretic model shows that individually rational adoption of generative AI causes model collapse that reduces collective social welfare for important tasks, with habit formation creating spillovers from low-stakes to high-value domains.
Claude Code answers recurring agent design questions with a thin model loop wrapped in dense safety, context, extensibility, and persistence harnesses, and those same questions get different answers in OpenClaw and Hermes.
EditFlow reconstructs temporal developer editing flows from code changes to benchmark and optimize AI code edit recommenders so they align with natural incremental reasoning rather than static snapshots.
Observational study of Claude Code and GitHub Copilot CLI at Microsoft finds social-network-driven adoption, activity-linked retention, and a persistent 24% lift in merged pull requests among adopters.
Experiments on 250 participants show LLM-assisted survey responses range from under 10% on Prolific to over 80% on Mechanical Turk, with identifiable characteristics and partial mitigation effects.
A multisite biometric study finds lower cognitive engagement under AI assistance via EEG and blink rate, with physiological-performance links present only in the non-AI condition.
Systematic review of agentic AI in the SDLC finds output verifiability drives industrial adoption in later phases, with Planner-Executor-Reviewer as the dominant pattern, plus a new multi-agent LLM screening pipeline for high-volume SLRs.
Meta-analysis of 23 studies shows moderate productivity gains from GenAI coding assistants (Hedges' g=0.33) but no significant effect on learning (g=0.14).
citing papers explorer
-
Flaws in the LLM Automation Narrative
A new code-writing data analysis benchmark shows human experts outperforming a frontier LLM on average with lower performance variance.
-
From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements
TDDev automates the full TDD loop for web app generation from requirements, delivering 34-48 percentage point quality gains and zero manual intervention in user studies.
-
Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey
Developers use LLMs like ChatGPT mainly for knowledge acquisition and code generation at the detailed design level, reporting benefits such as better technology selection and early flaw detection alongside limitations like lengthy outputs, incorrect code, and hallucinations.
-
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
-
Agentic Much? Adoption of Coding Agents on GitHub
Coding agents reached 22-29% adoption in GitHub projects within months of release, with agent-assisted commits larger and focused on features and bug fixes.
-
From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
-
AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
Longitudinal panel study of 802 developers shows an enterprise AI coding mandate doubled per-capita merged pull requests to 2.09x baseline, with gains associated with AI adoption and accumulated use while review processes automated.
-
Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software
Study of 930k+ agent PRs shows repository explains ~50% of integration friction variance, with agents concentrating it twice as much as humans (ICC 0.30 vs 0.16) after controls.
-
Life After Benchmark Saturation: A Case Study of CORE-Bench
Using CORE-Bench as a case study, the paper shows that saturated benchmarks can still deliver insights on efficiency, reliability, model-scaffold differences, and human collaboration even after accuracy plateaus, and introduces improved benchmark versions plus a small randomized experiment demonstra
-
Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study
Mixed-methods study creates taxonomy of AI IDE rules from 7310 instances, analyzes evolution drivers, and reports that rule updates raise average artifact compliance from 49.14% to 72.13%.
-
The Agentic Web Requires New Normative Infrastructure
The web's anti-bot regime should be replaced by a framework that presumptively lets user-authorized AI agents act for their principals, requires platforms to disclose access policies, and permits agent blocking only when proportionate to concrete harms.
-
Faster than the Team, Faster than the Customer: Tool Integration, Collaboration, and Organisational Lag in AI-assisted RE
AI-assisted requirements engineering in industry is constrained more by tool integration and organisational readiness than by the capabilities of the AI tools themselves.
-
From Prompting to Verification: How Experience Shapes Vibe Coding Practices
Survey of 162 vibe coders finds perceptions of AI code quality similar across experience levels but motivations, interaction styles, and quality assurance practices diverge, revealing a perception-action gap.
-
Design and Report Benchmarks for Knowledge Work
Proposes a three-step benchmark design method (define work activity, specify tested setting, score work product) derived from work studies and O*NET, demonstrated via three case analyses.
-
One Developer Is All You Need: A Case Study of an AI-Augmented One-Person Squad in a Brownfield Enterprise
Case study reports one staff engineer with four AI agents delivering a four-person-scoped brownfield project in half the planned time under Spec-Driven Development, with high code acceptance and major cost savings.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
-
Generative artificial intelligence reduces social welfare through model collapse
A game-theoretic model shows that individually rational adoption of generative AI causes model collapse that reduces collective social welfare for important tasks, with habit formation creating spillovers from low-stakes to high-value domains.
-
Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
Claude Code answers recurring agent design questions with a thin model loop wrapped in dense safety, context, extensibility, and persistence harnesses, and those same questions get different answers in OpenClaw and Hermes.
-
EditFlow: Benchmarking and Optimizing Code Edit Recommendation Systems via Reconstruction of Developer Flows
EditFlow reconstructs temporal developer editing flows from code changes to benchmark and optimize AI code edit recommenders so they align with natural incremental reasoning rather than static snapshots.
-
Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
Observational study of Claude Code and GitHub Copilot CLI at Microsoft finds social-network-driven adoption, activity-linked retention, and a persistent 24% lift in merged pull requests among adopters.
-
A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents
Experiments on 250 participants show LLM-assisted survey responses range from under 10% on Prolific to over 80% on Mechanical Turk, with identifiable characteristics and partial mitigation effects.
-
Using Biometrics to Understand AI-Assisted Coding Performance and its Perception
A multisite biometric study finds lower cognitive engagement under AI assistance via EEG and blink rate, with physiological-performance links present only in the non-AI condition.
-
Assistance to Autonomy: A Systematic Literature Review of Agentic AI across the Software Development Life Cycle
Systematic review of agentic AI in the SDLC finds output verifiability drives industrial adoption in later phases, with Planner-Executor-Reviewer as the dominant pattern, plus a new multi-agent LLM screening pipeline for high-volume SLRs.
-
A meta-analysis of the effect of generative AI on productivity and learning in programming
Meta-analysis of 23 studies shows moderate productivity gains from GenAI coding assistants (Hedges' g=0.33) but no significant effect on learning (g=0.14).
-
The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development
The Productivity-Reliability Paradox arises because AI code generators produce variable output while developers lack sufficient specification discipline, making governance models focused on specifications the binding constraint rather than model improvements.
-
More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems
AI-native software ecosystems exhibit emergent behaviors best explained by complex adaptive systems theory, requiring new ecosystem-level monitoring and seven testable propositions that may extend or replace Lehman's laws.
-
Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects
AI IDEs with structured guidance can produce functional large-scale code but frequently introduce design flaws such as duplication, complexity, and principle violations that risk long-term maintainability.
-
The Fast and Spurious: Developer Productivity with GenAI
Survey of 415 developers finds GenAI accelerates coding output but redistributes effort into review and verification, making net productivity gains appear spurious at current adoption levels.
-
Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration
Interviews reveal a four-stage vibe coding workflow that accelerates prototyping while introducing tensions between quick efficiency and reflective design intention, plus asymmetries in trust and ownership.
-
Pomona: Continuous Code Quality Improvement via Small, Agentic Pull Requests at Bloomberg
A two-skill agentic tool (Pomona) that fixes code-quality issues via approximately 10-line pull requests saw 15 of 17 PRs merged in a one-month Bloomberg team pilot.
-
How to Compare the Security of Code Written by Humans to LLM-generated Code
Proposes and validates via feasibility study an open-source automated framework for reproducible, species-fair security comparisons of human-written, LLM-generated, and hybrid code.
-
At What Cost? Software Developers' Well-Being in the Age of GenAI
The paper proposes shifting GenAI research in software engineering from narrow performance metrics to also include developer well-being, social context, and sustainable productivity.
-
Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development
Agentic Agile-V uses Agile-V as backbone and a Specify-Constrain-Orchestrate-Prove-Evolve-Verify loop to convert AI agent conversations into traceable engineering artifacts with acceptance evidence.
-
LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering
Structured specifications as LLM inputs can make high-quality repository-level code generation feasible with better verifiability than natural language prompts.
-
From Determinism to Delegation: AI-Native Software Engineering and the Evolution of the Agentic Engineer
Software engineering is undergoing a paradigm shift to AI-native practices centered on agentic systems rather than traditional code.
-
The Rise of AI-Native Software Engineering: Implications for Practice, Education, and the Future Workforce
A systematic review of 48 papers on AI in software engineering synthesizes evidence into frameworks for AI-native practice, a nine-dimension competency model, a four-phase curriculum roadmap, and an agenda of research gaps, while noting contradictory productivity findings.
-
Reshaping Undergraduate Computer Science Education in the Generative AI Era
The paper recommends shifting CS education from implementation skills to AI-native competencies, fundamental concepts, and critical evaluation based on workshop consensus.
- IACDM: Interactive Adversarial Convergence Development Methodology -- A Structured Framework for AI-Assisted Software Development