Pith. sign in

REVIEW 4 cited by

How much does AI impact development speed? An enterprise-based randomized controlled trial

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.12944 v3 pith:IHYJDSQZ submitted 2024-10-16 cs.SE cs.HC

Elise Paradis , Kate Grey , Quinn Madison , Daye Nam , Andrew Macvean , Vahid Meimand , Nan Zhang , Ben Ferrari-Church
show 1 more author
Satish Chandra
This is my paper · ORCID
classification cs.SEcs.HC
keywords effectimpacttasktimedeveloperdevelopersfoundacross
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

    cs.AI 2025-07 conditional novelty 7.0 of 10

    In a randomized trial of 246 real open-source tasks, experienced developers took 19% longer when AI tools were allowed, despite forecasting 24% faster completion.

  2. AI-Assisted Fixes to Code Review Comments at Scale

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Fine-tuned Llama models generate exact-match patches for 68% of internal code review comments, and a safety trial shows AI suggestions slow reviewers unless hidden from them.

  3. ACE: Automated Technical Debt Remediation with Validated Large Language Model Refactorings

    cs.SE 2025-07 conditional novelty 5.0 of 10

    ACE automates code refactoring by validating LLM-generated suggestions with syntax, CodeHealth, and semantic checks, and by discarding suggestions that fail those checks.

  4. Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A fine-tuned classifier and question generator let a small coding assistant detect under-specified prompts and ask for clarification, which users rated better than a baseline in a small study.

Pith tools