Pith

open record

sign in
Browse

arxiv: 2606.00438 · v1 · pith:QBIYO4BL · submitted 2026-05-30 · cs.SE

GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis

Alex Heilman , Alex Kyllo , Emerson Murphy-Hill This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-06-28 18:46 UTCgrok-4.3pith:QBIYO4BLrecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: Average PRs vs. active coding hours, with propor [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] reproduced from arXiv: 2606.00438
classification cs.SE
keywords GitHub Copilotdeveloper productivityobservational studyfixed effectspull requestsdose-response analysissoftware engineering
0
0 comments X

The pith

Engineers complete 40.5 percent more pull requests in high Copilot weeks than zero-usage weeks at the same measured effort.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper uses 43 weeks of data from 16,223 engineers to test whether GitHub Copilot increases the number of pull requests completed. Engineer fixed effects remove time-invariant differences in skill or role, while controls for active coding time and browser time isolate usage effects from overall effort levels. The resulting estimate shows higher Copilot usage weeks produce more output at equivalent effort, with a monotonic pattern and diminishing returns at the top end. Seven robustness checks address alternative explanations such as task reallocation or easier work during high-usage periods.

Core claim

Under an explicitly stated conditional-independence assumption, the within-engineer design estimates a tool-specific efficiency effect: engineers are estimated to complete 40.5% more PRs in their highest GHCP usage weeks relative to their zero-usage weeks, holding measured development effort constant. The gradient is monotonic with diminishing returns at high intensity.

What carries the argument

Engineer fixed effects plus active coding time and browser time inside a Poisson Pseudo-Maximum Likelihood model with two-way fixed effects, which defines the estimand as an efficiency effect rather than a selection or effort effect.

If this is right

  • The productivity gain remains after tests for non-coding AI use, team shocks, within-week reallocation, cross-week contamination, PR slicing, easier tasks, and alternative treatment measures.
  • Output rises steadily with usage intensity but flattens at the highest levels.
  • The design separates a tool-specific efficiency gain from general differences in engineer skill or week-to-week busyness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the conditional independence assumption holds, organizations could measure returns by tracking PR output per coding hour before and after wider Copilot rollout.
  • The dose-response shape implies that moderate usage may capture most gains without requiring maximum intensity.
  • The same fixed-effects approach could be applied to other AI coding assistants to compare efficiency effects across tools.

Load-bearing premise

After conditioning on engineer fixed effects and measured effort, variation in Copilot usage across weeks is independent of other time-varying factors that affect pull request output.

What would settle it

A randomized trial that assigns different Copilot access levels to matched engineers, records their active coding time, and counts completed pull requests would directly test the 40.5 percent estimate.

Figures

Figures reproduced from arXiv: 2606.00438 by Alex Heilman, Alex Kyllo, Emerson Murphy-Hill.

Figure 2
Figure 2. Figure 2: Weekly GHCP usage trajectories for three sampled [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Usage depth dose-response [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Placebo outcome: own PRs vs. teammates’ PRs by [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Placebo treatment: non-coding M365 Copilot [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Timing tests: current (𝑡), lagged (𝑡−1), and leading (𝑡+1) GHCP usage coefficients. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Task-mix test: GHCP gradient on PRs authored vs. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: PR size decomposition: dose-response by files [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Alternative treatment operationalization: usage [PITH_FULL_IMAGE:figures/full_fig_p008_10.png] view at source ↗
read the original abstract

Does GitHub Copilot (GHCP) make engineers more productive, or do the engineers who use it more differ from those who use it less? And even within a single engineer, are GHCP-heavy weeks just busy weeks in which more of everything gets done? We study these questions using 43 weeks of data from 16,223 software engineers across Microsoft's Cloud+AI organization. Engineer fixed effects address the first concern by comparing each engineer against themselves rather than against other engineers, eliminating time-invariant differences in skill, role, and team. Active coding time and browser time then enter a Poisson Pseudo-Maximum Likelihood model with two-way fixed effects to address the harder, within-engineer confound: that GHCP-heavy weeks coincide with high-effort weeks. This defines our estimand as an efficiency effect: more pull requests completed at equivalent levels of coding time. Engineers are estimated to complete 40.5% more PRs in their highest GHCP usage weeks relative to their zero-usage weeks, holding measured development effort constant. The gradient is monotonic with diminishing returns at high intensity. Seven robustness and falsification tests target the remaining plausible alternative explanations (non-coding AI engagement, team-level shocks, within-week task reallocation, cross-week contamination, PR slicing into smaller units, shifts toward easier task types, and sensitivity to how the treatment is operationalized). Under an explicitly stated conditional-independence assumption, the within-engineer design estimates a tool-specific efficiency effect that is consistent with all seven robustness tests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper analyzes observational data from 16,223 Microsoft engineers over 43 weeks to estimate GitHub Copilot's effect on productivity. Engineer fixed effects combined with controls for active coding time and browser time in a Poisson PML model with two-way fixed effects yield an estimate that engineers complete 40.5% more PRs in highest-usage weeks relative to zero-usage weeks at equivalent measured effort; the dose-response is monotonic with diminishing returns. Seven robustness and falsification tests are reported to support interpreting the coefficient as a tool-specific efficiency effect under an explicitly stated conditional-independence assumption.

Significance. If the identifying assumption holds, the study supplies one of the largest within-engineer estimates of AI coding-tool productivity effects, with explicit effort controls, a dose-response gradient, and multiple targeted robustness checks. These features address common selection and effort confounds in observational developer-tool research and supply falsifiable predictions.

major comments (1)
  1. [Abstract] Abstract and Methods: the central 40.5% estimate and its causal interpretation rest on the conditional-independence assumption after engineer FE plus measured effort; the manuscript must demonstrate that each of the seven robustness tests directly targets a distinct plausible violation (e.g., unmeasured task difficulty, PR slicing) rather than merely showing coefficient stability, because any single unaddressed threat would undermine the efficiency-effect claim.
minor comments (2)
  1. [Abstract] Abstract: report the exact coefficient, standard error, and sample size (engineer-weeks) underlying the 40.5% figure so readers can assess precision without needing the full tables.
  2. [Methods] Methods: clarify whether the dose-response bins or functional form for GHCP usage were pre-specified or chosen after inspecting the data, as this affects interpretation of the monotonic gradient.

Simulated Author's Rebuttal

1 responses · 0 unresolved

Thank you for the detailed review. We address the major comment below and will revise the manuscript accordingly to strengthen the presentation of our robustness tests.

read point-by-point responses

  1. Referee: [Abstract] Abstract and Methods: the central 40.5% estimate and its causal interpretation rest on the conditional-independence assumption after engineer FE plus measured effort; the manuscript must demonstrate that each of the seven robustness tests directly targets a distinct plausible violation (e.g., unmeasured task difficulty, PR slicing) rather than merely showing coefficient stability, because any single unaddressed threat would undermine the efficiency-effect claim.

    Authors: We agree with the referee that explicitly mapping each robustness test to the specific threat it addresses will improve the clarity of the manuscript and help readers assess the strength of the conditional-independence assumption. In the current manuscript, the abstract lists the seven tests and the threats they target, but we acknowledge that a more direct one-to-one correspondence could be made clearer, particularly in the abstract. In the revised version, we will update the abstract to include a brief mapping (e.g., 'the PR-slicing test addresses concerns about output quality dilution; the task-difficulty test uses within-engineer variation in task types...'). We will also add a dedicated subsection in the Methods that tabulates each test, the violation it targets, and the identifying assumption it probes. This revision does not alter our empirical results or conclusions but addresses the presentation concern directly. revision: yes

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The central estimate of a 40.5% productivity increase is obtained by fitting a Poisson Pseudo-Maximum Likelihood model with engineer fixed effects and controls for measured effort (active coding time and browser time) to observational data from 16,223 engineers. This produces an empirical coefficient under an explicitly stated conditional-independence assumption rather than reducing by construction to a self-defined quantity, a fitted parameter renamed as a prediction, or any self-citation chain. The seven robustness tests address threats to the identifying assumption without altering the model's structure in a self-referential manner. The derivation is therefore self-contained against external benchmarks and receives a score of 0.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on one domain assumption (conditional independence after fixed effects and effort controls) rather than free parameters or new entities; no additional fitted constants or postulated objects are introduced in the abstract.

axioms (1)
  • domain assumption Conditional independence: GHCP usage is independent of other time-varying determinants of PR output after conditioning on engineer fixed effects, active coding time, and browser time.
    Explicitly invoked in the abstract to interpret the coefficient as a tool-specific efficiency effect.

pith-pipeline@v0.9.1-grok · 5805 in / 1473 out tokens · 29786 ms · 2026-06-28T18:46:10.447518+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate

    cs.SE 2026-07 conditional novelty 6.0

    Longitudinal panel study of 802 developers shows an enterprise AI coding mandate doubled per-capita merged pull requests to 2.09x baseline, with gains associated with AI adoption and accumulated use while review proce...

Reference graph

Works this paper leans on

7 extracted references · 2 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Avidit Acharya, Matthew Blackwell, and Maya Sen. 2016. Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects.American Political Science Review110, 3 (2016), 512–529

    2016

  2. [2]

    Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2025. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.Management Science(2025). doi:10.2139/ssrn.4945566 Forthcoming

    work page doi:10.2139/ssrn.4945566 2025

  3. [3]

    Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity.ACM Queue19, 1 (2021), 20–48

    2021

  4. [4]

    Manuel Hoffmann, Sam Boysel, Frank Nagle, Sida Peng, and Kevin Xu. 2025. Generative AI and the Nature of Work. Working Paper 25-021. Harvard Business School

    2025

  5. [5]

    2018.The Book of Why: The New Science of Cause and Effect

    Judea Pearl and Dana Mackenzie. 2018.The Book of Why: The New Science of Cause and Effect. Basic Books

    2018

  6. [6]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.arXiv preprint arXiv:2302.06590(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    J. M. C. Santos Silva and Silvana Tenreyro. 2006. The Log of Gravity.The Review of Economics and Statistics88, 4 (2006), 641–658. 9

    2006