Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Coding agents beat copilots in the first controlled developer study, with 35% more tasks completed and about half the user effort.

desk verdict The first controlled copilot-vs-agent user study and a believable qualitative story, but the headline 35% completion gap rests on grading that needs to be shown before I'd trust the number. read the letter →

arxiv 2507.08149 v2 pith:EIDT3AFK submitted 2025-07-10 cs.SE

Valerie Chen , Ameet Talwalkar , Robert Brennan , Graham Neubig This is my paper
classification cs.SE
keywords codingagentsAIcopilotsdeveloperproductivitycontrolleduserstudyhuman-AIinteractionworkflowanalysisLLM-basedsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports the first controlled user study that directly compares developer interactions with copilot-style assistants and with autonomous coding agents. It claims agents improve measurable productivity: participants completed about 35 percent more tasks correctly and spent about half as much active effort (12.5 minutes versus 25.1 minutes on average). It also claims these gains do not automatically translate into a better experience, since participants understood agent outputs less well, felt less in control, and were not more satisfied overall. The authors conclude that future coding agents should be designed for transparency, balanced proactivity, and effective use of human attention. If correct, these findings mean the binding constraint in AI-assisted programming is shifting from generating code to supervising and steering it.

What carries the argument

The load-bearing machinery is the study's controlled within-participant protocol rather than any single algorithm. Each of 20 experienced copilot users worked on the same type of realistic task in a 40-minute session with GitHub Copilot and in another 40-minute session with OpenHands, with tool order randomized; tasks came from GAIA for data analysis and from SWE-Bench for bug fixes and feature additions, and were chosen so a state-of-the-art agent could not solve them alone. Productivity is measured two ways: binary task correctness, modeled with condition and problem as fixed effects, and user effort, computed for agents by summing the intervals between each user message and the agent's preceding and following actions; that is, it counts the time a person spends inspecting output and composing the next instruction rather than the agent's execution time. User experience is measured with Likert comparisons and Wilcoxon signed-rank tests, and interaction trajectories from event streams and screen recordings are used to characterize where human effort goes.

What would settle it

Replicate the study with the same tasks and design, but have two independent graders who are blind to which tool produced each solution score every final code snapshot against a pre-specified rubric; if the graders disagree at the task level or the agent advantage loses significance, the 35 percent claim is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a measured productivity gap in favor of coding agents over copilots for the same realistic tasks. In a within-participant design, 20 developers who already used GitHub Copilot solved data-analysis, feature-addition, and bug-fixing tasks in 40-minute sessions with each tool, with order randomized. With OpenHands, task correctness rose from 25 percent (standard error 10) to 60 percent (standard error 11), a 35-percentage-point improvement that the paper reports as significant ($p = 0.02$). For tasks completed correctly, average active user effort fell from 25.1 minutes (standard error 6.22) to 12.5 minutes (standard error 2.8, $p = 0.01$), while the agent's own wall-clock time averaged 27.9 minutes, meaning the savings come from shifting work onto the machine rather than reducing total calendar time. Participants also reported significantly lower cognitive load ($p = 0.0006$) and a greater sense of being able to accomplish new tasks ($p = 0.0013$), yet a majority said they understood copilot outputs better, and satisfaction, flow, and perceived speed differences were not significant. The paper interprets these results as evidence that agents move developers from hands-on coding toward oversight, while identifying transparency, calibrated proactivity, and better use of human effort as the design challenges that remain.

Load-bearing premise

The load-bearing premise is that the authors' binary grading of open-ended tasks as correct or incorrect is reliable and unbiased, since the headline 35-percentage-point gap is calculated from those grades.

Editorial extensions

If this is right

  • If agents sustain this advantage, the developer's job in many tasks changes from writing and running code to specifying tasks, reviewing changes, and deciding when to intervene.
  • Because agents spend nearly 28 minutes executing while the user is active for only 12.5 minutes, people who multi-task during agent runs may see larger total-time savings than this single-task study captures.
  • Tasks that users rarely finished with copilots, such as the data-analysis problems, were sometimes completed only with agents, suggesting agents expand what a solo developer can take on.
  • Since participants preferred copilot outputs for comprehension, improving agent transparency about why a file changed and why a fix was made is a prerequisite for user trust and adoption.
  • Bug-fixing tasks showed the smallest agent advantage, indicating that autonomous debugging in unfamiliar repositories remains a weak point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 35-percentage-point completion gap could shrink if participants were experienced agent users; this study deliberately recruited agent novices, so the measured gap is a first-adoption comparison rather than a ceiling on agent performance.
  • The paper's effort metric counts only active user time, so a fair total-cost comparison would also count agent execution time; with the reported numbers (27.9 versus 12.5 minutes), agents look efficient only if the user is productively occupied elsewhere during the difference.
  • A direct next experiment would vary the agent's LLM and the copilot's LLM symmetrically, since this study fixed one model for the agent while letting participants choose any model in the copilot condition.
  • The results suggest a design target for agent interfaces: expose the agent's plan and confidence at natural stopping points, which would address the transparency gap the paper documents and could shift the nonsignificant satisfaction result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a controlled user study comparing two AI coding assistants—GitHub Copilot as the copilot and OpenHands as the coding agent—with N=20 university-student participants who were regular copilot users but novice agent users. Participants solved one of three task types (GAIA-based data analysis, SWE-bench bug fixing, and feature addition) in both conditions in randomized order, each in a 40-minute phase. The central quantitative claims are that the agent condition produced a 35-percentage-point higher task-correctness rate (60% vs. 25%), roughly half the user effort (12.5 min vs. 25.1 min), lower cognitive load, and a greater sense of being able to accomplish new tasks, while users reported worse understanding of agent outputs. The authors characterize the resulting workflow shift from hands-on coding to prompting and oversight and propose three design desiderata for future coding agents: transparency, balanced proactivity, and effective use of human effort.

Significance. Conditional on the measurement foundation, this is a timely and useful contribution: it is one of the first human-in-the-loop controlled comparisons of copilots and coding agents, uses realistic benchmark-derived tasks rather than toy programs, combines telemetry with qualitative trajectory analysis, and releases analysis scripts. The RQ3 workflow diagrams and the three design desiderata are valuable for the HCI and software-engineering communities. The study is not circular—correctness and effort are measured from participant behavior rather than fitted from model outputs. However, the central productivity magnitudes rest on measurement choices that the current manuscript does not fully support: subjective binary correctness grading without a rubric or blinding, an asymmetric effort metric, possible learning effects in the within-participant design, and a small convenience sample. These issues are fixable with additional analysis and reporting, so I regard the contribution as defensible but in need of major revision.

major comments (4)
  1. [Section 3.6 and Section 4.1] The RQ1 headline '35% increase in task correctness' is computed from binary correct/incorrect judgments whose only operationalization is 'we evaluated whether the task description was correctly completed' (Section 3.6). No grading rubric, grader blinding, or inter-rater agreement is reported, and several tasks in Appendix B require open-ended engineering judgment (e.g., 'smooth transitions between plots', 'dark mode toggle', and the extended GAIA analyses). Because the authors are affiliated with All Hands AI, the organization behind OpenHands, the absence of independent, blinded grading leaves the completion gap vulnerable to expectation bias. Please add a scoring rubric, independent blinded graders, inter-rater reliability, and per-task agreement, or temper the as-stated RQ1 numbers.
  2. [Section 3.6 and Section 4.1] The 'about 50% the user effort' claim compares two different quantities. Copilot effort is total wall-clock time from start to end, whereas agent effort is the sum of gaps between the agent's last action preceding a user message and the agent's first action following it; this definition excludes the time users spend reading the agent's final output, any time spent while the agent is executing, and waiting time. The manuscript itself notes that including agent action time raises the agent total to 27.9 minutes, comparable to the copilot's 25.1 minutes. Please report user effort under a common definition, or clearly label the metric as 'active instruction time' in the abstract and results, and provide sensitivity analyses.
  3. [Section 3.4 and Appendix D] The within-participant design assigns the same task type in both phases, so performance in the second phase can reflect learning from the first phase even though the specific task instance changes. Order was randomized, but no statistical test for an order-by-condition interaction is reported; Appendix D.1 asserts a 'similar size effect' for correctness while Appendix D.2 notes the effort ordering difference is 'fairly different.' With N=20, the 35-point gap and effort reduction could be partly carryover artifacts. Please report condition-by-order means, formal interaction tests, and, if possible, estimates adjusted for order.
  4. [Section 3.3, Appendix B.1, and Section 4.1] Two of the six tasks are GAIA data-analysis problems that require web browsing and data scraping (Appendix B.1). OpenHands includes a built-in browser and terminal (Section 3.2), while GitHub Copilot users must perform those steps manually. The paper reports that both data-analysis problems were completed only in the agent condition, so these two tasks may drive much of the 35-percentage-point completion gap. This does not invalidate the comparison of the two tools as deployed, but the abstract should not generalize to 'more autonomous AI tools' without either a capability-matched baseline or an analysis showing the results hold when these two tasks are excluded.
minor comments (5)
  1. [Section 3.6] The statistical model for task correctness is not fully specified: it is described as a linear model with condition and coding problem as fixed effects, but the manuscript does not state whether this is a linear probability model or a logistic model, whether participant random effects are included, or what the coefficient and confidence interval for the condition effect are. Please report the model and the marginal effect with its confidence interval.
  2. [Tables 2 and 3] The distribution columns in Tables 2 and 3 are ambiguous; for example, Table 3's F1 and F2 rows show only two percentages, which cannot map uniquely onto a five-point Likert scale, and the percentages do not sum to 100. A legend or explicit column values is needed.
  3. [Abstract and Section 4.1] The phrase 'with about 50% the user effort' is both awkward and unclear, since the effort metric for the agent condition is operationalized as active instruction time rather than total time. Please revise for precision.
  4. [Appendix D] The interpretations in Appendix D—such as 'fixing bugs seems to be where agents currently struggle' and the ordering-effect discussion—are based on very small per-cell counts (roughly 3–10 participants per cell). Please add the per-cell sample sizes or appropriate test statistics.
  5. [Throughout] There are minor typos and wording issues, including 'users have have different perspectives' (Section 4.2), 'OpenHands can provide automate code running, debugging and testing' (Section 4.3), and inconsistent capitalization of 'GitHub' as 'Github' in several places.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the productivity and experience results are measured outcomes, not quantities reconstructed from the paper's own definitions or from its authors' prior work.

full rationale

This paper reports a controlled within-participant user study (Section 3.4) and contains no derivation chain whose output is equivalent to its inputs. The headline RQ1 quantity, 'a 35% increase in task correctness' (Section 4.1), is computed from binary outcome labels assigned to final code snapshots ('we evaluated whether the task description was correctly completed,' Section 3.6). That label is a measured outcome, not a fitted parameter or a definitional identity: the observed gap could plausibly have gone in either direction, and it is separately tested with a fixed-effects linear model (p = 0.02). The user-effort comparison likewise contrasts two explicitly different operationalizations—total session time under the copilot versus inter-message 'instruction time' under the agent (Section 3.6)—so the reported reduction is an empirical quantity conditioned on that measurement choice, not a tautology. Citations to the authors' own prior work appear (e.g., OpenHands [6], proactive AI assistants [24], RealHumanEval [25]), but none is load-bearing for the central empirical claim, which rests on the collected trajectories, final code snapshots, and Likert responses rather than on cited benchmark numbers or theorems. The most serious concerns are validity and conflict-of-interest risks—unblinded, rubric-free binary grading by authors affiliated with All Hands AI and the choice of OpenHands as the representative agent—but these concern measurement reliability and study design, not a reduction of the conclusion to its inputs. The Limitations section also explicitly acknowledges representativeness limits and the narrow task set. Therefore the analysis finds no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data; the single listed parameter is a hand-chosen operational definition. The study relies on domain assumptions about tool representativeness, grading reliability, self-report validity, and sample representativeness, all acknowledged or implicit in the design.

free parameters (1)
  • User effort operationalization (agent condition) = Interval between the agent's last action before and first action after each user message
    The central effort comparison depends on this hand-chosen definition of what counts as user effort in the agent condition; it excludes agent wall-clock time and assumes the gap reflects user inspection and prompting.
assumptions (4)
  • domain assumption GitHub Copilot and OpenHands are representative of the copilot and agent classes broadly.
    Section 3.2 argues each tool encompasses common features of alternatives, which is a reasonable but untestable assumption given each class is heterogeneous.
  • domain assumption The authors' binary correctness grading is reliable and unbiased despite no rubric or inter-rater reliability reported.
    Section 3.6 says tasks were evaluated for correctness but does not describe the grading process, making this a load-bearing assumption for RQ1.
  • domain assumption Participants' Likert ratings and qualitative comments accurately reflect their experienced cognitive load, satisfaction, and understanding.
    Section 3.5 collects self-reported survey data; the validity of these measures is assumed.
  • domain assumption The 20 recruited students, all regular Copilot users and novice agent users, represent the broader developer population targeted by the study.
    Section 3.1 generalizes from this sample to the 'majority of current software developers' via the 2025 Stack Overflow survey.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows." pith.science (2026). https://pith.science/paper/EIDT3AFK

@misc{pith2026250708149,
  author       = {Pith},
  title        = {Pith review of: Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIDT3AFK}},
  note         = {Machine review of arXiv:2507.08149}
}
read the original abstract

Developers now have access to a growing array of increasingly autonomous AI tools for software development. While many studies examine copilots that provide chat assistance or code completions, evaluations of coding agents -- which can automatically write files and run code -- still rely on static benchmarks. We present the first controlled study of developer interactions with coding agents, characterizing how more autonomous AI tools affect productivity and experience. We evaluate two leading copilot and agentic coding assistants, recruiting participants who regularly use the former. Our results show agents can assist developers in ways that surpass copilots (e.g., completing tasks humans may not have accomplished) and reduce the effort required to finish tasks. Yet challenges remain for broader adoption, including ensuring users adequately understand agent behaviors. Our findings reveal how workflows shift with coding agents and how interactions differ from copilots, motivating recommendations for researchers and highlighting challenges in adopting agentic systems.

Figures

Figures reproduced from arXiv: 2507.08149 by the authors.

Figure 1
Figure 1. The spectrum of developer workflows with AI tools. While prior studies have evaluated how human-only workflows compare to human-copilot workflows, understanding how AI can code with developers, little work exists comparing copilots to the emerging agent workflows, where AI is more often used to code for developers. We study how these differences affect productivity, user experience, and interaction patterns with AI … view at source ↗
Figure 2
Figure 2. Comparing affordances in copilot (left) and agent (right) workflows. While prior work has studied the benefits and challenges of coding with copilots (overviewed in Section 2.3), we ask the same questions about agentic workflows. and assist users in evaluating whether code implementations follow task instructions. However, despite productivity gains, participants noted multiple aspects of the user experience could b… view at source ↗
Figure 3
Figure 3. Overview of our study procedure. We designed a controlled user study to directly contrast participant interactions with copilots and agents. Participants were asked to solve a set of realistic and challenging tasks, including fixing issues in repositories, adding features, and conducting deep research and analysis. impacts user productivity. 3.3 Task Types Prior studies evaluating interactions with AI coding assista… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Measuring the effect of different coding assistants on user productivity. We consider two measurements: the task correctness rate and user effort spent to build a correct solution (i.e., excluding incomplete or incorrect solutions). For each metric, we report the mean …
Figure 5
Figure 5. Figure 5: Comparing developer workflows with copilots and agents. We visualize abstracted user trajectories across three parts of the workflow: implementing functionality, setting up the environment, and debugging. We find that agentic workflows begin to shift more effort to the…
Figure 6
Figure 6. Figure 6: In the chat panel, users can select from ask, edit, and agent mode. In both edit and [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Overview of OpenHands interface. On the left panel, the user message is displayed [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Analysis of task correctness by task version, programming experience, copilot usage, ordering of copilot/agent in study. • I was more in the flow using OpenHands than Github Copilot • I experience less cognitive load using OpenHands than Github Copilot • I have a bette…
Figure 9
Figure 9. Figure 9: Analysis of user effort by task version, programming experience, copilot usage, ordering of copilot/agent in study. D Additional Results D.1 Task Correctness Breakdown In [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mind the Sim2Real Gap in User Simulation for Agentic Tasks

    cs.AI 2026-03 conditional novelty 7.0 of 10

    On τ-bench, LLM user simulators are more cooperative, more verbose, and more lenient than real human users, so agent benchmarks that rely on them overstate real-world performance.

  2. How can we assess human-agent interactions? Case studies in software agent design

    cs.AI 2025-10 conditional novelty 6.0 of 10

    PULSE combines sparse human ratings with prediction-powered inference to cut confidence intervals by ~40% and shows LLM choice matters more than scaffolding for user satisfaction.

Reference graph

Works this paper leans on

74 extracted references · 50 canonical work pages · cited by 2 Pith papers

  1. [1]

    Github copilot - your ai pair programmer, 2022

    Github. Github copilot - your ai pair programmer, 2022. URL https://github.com/ features/copilot

    work page 2022

  2. [2]

    Cursor - the ai code editor, 2023

    Cursor. Cursor - the ai code editor, 2023. URLhttps://www.cursor.com/

    work page 2023

  3. [3]

    Introducing the windsurf editor, 2024

    Windsurf. Introducing the windsurf editor, 2024. URLhttps://windsurf.com/

    work page 2024

  4. [4]

    Continue Dev

    Inc. Continue Dev. Continue: Open-source ai code assistant. https://github.com/ continuedev/continue, 2025. Accessed: 2025-05-08

    work page 2025

  5. [5]

    Introducing devin, the first ai software engineer, 2024

    Cognition. Introducing devin, the first ai software engineer, 2024. URL https://cognition. ai/blog/introducing-devin

    work page 2024

  6. [6]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  7. [7]

    Claude code overview, 2025

    Anthropic. Claude code overview, 2025. URL https://docs.anthropic.com/en/docs/ claude-code/overview

    work page 2025

  8. [8]

    Pair programming: what’s in it for me? In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 120–128, 2008

    Andrew Begel and Nachiappan Nagappan. Pair programming: what’s in it for me? In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 120–128, 2008

    work page 2008

Show all 74 references
  1. [9]

    ’i do have a fair amount of concern.’ the ceo of$61 bil- lion anthropic says ai will take over a crucial part of software engineers’ jobs within a year, 2025

    Sherin Shibu. ’i do have a fair amount of concern.’ the ceo of$61 bil- lion anthropic says ai will take over a crucial part of software engineers’ jobs within a year, 2025. URL https://www.entrepreneur.com/business-news/ anthropic-ceo-predicts-ai-will-take-over-coding-in-12-mo...

    2025

  2. [10]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  3. [11]

    Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859, 2024

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859, 2024

    arXiv

  4. [12]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    2025 arXiv

  5. [13]

    Collaborative gym: A framework for enabling and evaluating human-agent collaboration.arXiv preprint arXiv:2412.15701, 2024

    Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration.arXiv preprint arXiv:2412.15701, 2024. 16

    2024

  6. [14]

    Interactive agents to overcome ambiguity in software engineering.arXiv preprint arXiv:2502.13069, 2025

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neu- big. Interactive agents to overcome ambiguity in software engineering.arXiv preprint arXiv:2502.13069, 2025

    2025

  7. [15]

    Challenges and opportunities of llm-based synthetic personae and data in hci

    Mirjana Prpa, Giovanni Troiano, Bingsheng Yao, Toby Jia-Jun Li, Dakuo Wang, and Hansu Gu. Challenges and opportunities of llm-based synthetic personae and data in hci. In Companion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing, ...

    2024

  8. [16]

    Human subjects research in the age of generative ai: Opportunities and challenges of applying llm-simulated data to hci studies

    Angel Hsing-Chi Hwang, Michael S Bernstein, S Shyam Sundar, Renwen Zhang, Manoel Horta Ribeiro, Yingdan Lu, Serina Chang, Tongshuang Wu, Aimei Yang, Dmitri Williams, et al. Human subjects research in the age of generative ai: Opportunities and challenges of applying llm-simula...

    2025

  9. [17]

    Expectation vs

    Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. InCHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–7, 2022

    2022

  10. [18]

    The impact of ai on developer productivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590, 2023

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590, 2023

    2023 arXiv

  11. [19]

    Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages, 7(OOPSLA1):85–111, 2023

    Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages, 7(OOPSLA1):85–111, 2023

    2023

  12. [20]

    it’s weird that it knows what i want

    James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. “it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers.ACM Trans. Co...

    2023 doi

  13. [21]

    Reading between the lines: Modeling user behavior and costs in ai-assisted programming

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai-assisted programming. InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–16, 2024

    2024

  14. [22]

    Ivie: Lightweight anchored explanations of just-generated code

    Litao Yan, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. Ivie: Lightweight anchored explanations of just-generated code. InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–15, 2024

    2024

  15. [23]

    Vera Liao, and Jennifer Wort- man Vaughan

    Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, and Jennifer Wort- man Vaughan. Generation probabilities are not enough: Uncertainty highlighting in ai code completions.ACM Trans. Comput.-Hum. Interact., 32(1), April 2025. ISSN 1073-0516. doi: 10.1145/3702320. UR...

    2025 doi

  16. [24]

    Need help? designing proactive ai assistants for programming

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. Need help? designing proactive ai assistants for programming. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA, 2025. Associat...

    2025

  17. [25]

    The realhumaneval: Evaluating large language models’ abilities to support programmers

    Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Den- nis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, and David Sontag. The realhumaneval: Evaluating large language models’ abilities to support programmers. Transactions on Machine L...

  18. [26]

    2025 stack overflow developer survey, 2025

    Stack Overflow. 2025 stack overflow developer survey, 2025. URL https://survey. stackoverflow.co/2025/

    2025

  19. [27]

    Gaia: a benchmark for general ai assistants

    Gr´ egoire Mialon, Cl´ ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  20. [28]

    Chatgpt: Optimizing language models for dialogue, 2022

    OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https://openai. com/blog/chatgpt/

    2022

  21. [29]

    Meet claude, 2023

    Anthropic. Meet claude, 2023. URLhttps://www.anthropic.com/claude

    2023

  22. [30]

    Devgpt: Studying developer-chatgpt conversations.arXiv preprint arXiv:2309.03914, 2023

    Tao Xiao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. Devgpt: Studying developer-chatgpt conversations.arXiv preprint arXiv:2309.03914, 2023

    2023 arXiv

  23. [31]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xi- aoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. InFirst Conference on Language Modeling

  24. [32]

    Llm-powered autonomous agents.lilianweng.github.io, Jun 2023

    Lilian Weng. Llm-powered autonomous agents.lilianweng.github.io, Jun 2023. URL https://lilianweng.github.io/posts/2023-06-23-agent/

    2023

  25. [33]

    Cognitive architectures for language agents.Transactions on Machine Learning Research

    Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents.Transactions on Machine Learning Research

  26. [34]

    An interactive agent foundation model.arXiv preprint arXiv:2402.05929, 2024

    Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, et al. An interactive agent foundation model.arXiv preprint arXiv:2402.05929, 2024

    2024 arXiv

  27. [35]

    Evalu- ating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  28. [36]

    The effects of generative ai on high-skilled work: Evidence from three field experiments with software developers.Available at SSRN 4945566, 2025

    Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The effects of generative ai on high-skilled work: Evidence from three field experiments with software developers.Available at SSRN 4945566, 2025

    2025

  29. [37]

    The programmer’s assistant: Conversational interaction with a large language model for software development

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. The programmer’s assistant: Conversational interaction with a large language model for software development. InProceedings of the 28th International Conference on Intelligent User Interfaces...

    2023

  30. [38]

    Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z. Henley. Challenges in using conversational ai for data science. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA ’25, New York, NY, USA, 2025. Associat...

    2025

  31. [39]

    Studying the effect of ai code generators on supporting novice learners in introductory programming

    Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. Studying the effect of ai code generators on supporting novice learners in introductory programming. InProceedings of the 2023 CHI Conference on Human Factors in Computing Sy...

    2023

  32. [40]

    How do analysts understand and verify ai-assisted data analyses? InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–22, 2024

    Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M Drucker. How do analysts understand and verify ai-assisted data analyses? InProceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–22, 2024. 18

    2024

  33. [41]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  34. [42]

    Productivity assessment of neural code completion

    Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. InProceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages...

    2022

  35. [43]

    Weisz, Shraddha Vijay Kumar, Michael Muller, Karen-Ellen Browne, Arielle Goldberg, Katrin Ellice Heintze, and Shagun Bajpai

    Justin D. Weisz, Shraddha Vijay Kumar, Michael Muller, Karen-Ellen Browne, Arielle Goldberg, Katrin Ellice Heintze, and Shagun Bajpai. Examining the use and impact of an ai code assistant on developer productivity and experience in the enterprise. InProceedings of the Extended...

    2025

  36. [44]

    Measuring the impact of early- 2025 ai on experienced open-source developer productivity.arXiv preprint arXiv:2507.09089, 2025

    Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. Measuring the impact of early- 2025 ai on experienced open-source developer productivity.arXiv preprint arXiv:2507.09089, 2025

    2025 arXiv

  37. [45]

    Agents in the wild - dashboard.https://insights

    Mark Niklas M¨ uller Christian M¨ urtz. Agents in the wild - dashboard.https://insights. logicstar.ai, 2025. URL https://doi.org/10.5281/zenodo.15846865. Interactive web dashboard. Code available athttps://github.com/logic-star-ai/insights

    2025 doi

  38. [46]

    The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering

    Hao Li, Haoxiang Zhang, and Ahmed E Hassan. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003, 2025

    2025 arXiv

  39. [47]

    Vibe coding: programming through conversation with artificial intelligence.arXiv preprint arXiv:2506.23253, 2025

    Advait Sarkar and Ian Drosos. Vibe coding: programming through conversation with artificial intelligence.arXiv preprint arXiv:2506.23253, 2025

    2025

  40. [48]

    Zamfirescu-Pereira, Eunice Jun, Michael Terry, Qian Yang, and Bjoern Hartmann

    J.D. Zamfirescu-Pereira, Eunice Jun, Michael Terry, Qian Yang, and Bjoern Hartmann. Beyond code generation: Llm-supported exploration of the program design space. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA, 2025....

    2025

  41. [49]

    Ml-powered coding companion – amazon codewhisperer, 2022

    Amazon. Ml-powered coding companion – amazon codewhisperer, 2022. URL https: //aws.amazon.com/codewhisperer/

    2022

  42. [50]

    The collaborative ai coder, 2024

    Cline. The collaborative ai coder, 2024. URLhttps://cline.bot/

    2024

  43. [51]

    Replit agent, 2024

    Replit. Replit agent, 2024. URLhttps://docs.replit.com/replitai/agent

    2024

  44. [52]

    Does the whole exceed its parts? the effect of ai explanations on complementary team performance

    Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. InProceedings of the 2021 CHI conference on human factors in ...

    2021

  45. [53]

    Explanations can reduce overreliance on ai systems during decision-making.Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1):1–38, 2023

    Helena Vasconcelos, Matthew J¨ orke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael S Bernstein, and Ranjay Krishna. Explanations can reduce overreliance on ai systems during decision-making.Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1):1–38, 2023. 19

    2023

  46. [54]

    ” help me help the ai”: Understanding how explainability can support human-ai interaction

    Sunnie SY Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andr´ es Monroy-Hern´ andez. ” help me help the ai”: Understanding how explainability can support human-ai interaction. Inproceedings of the 2023 CHI conference on human factors in computing systems, pages...

    2023

  47. [55]

    Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies

    Sunnie SY Kim, Jennifer Wortman Vaughan, Q Vera Liao, Tania Lombrozo, and Olga Rus- sakovsky. Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies. InProceedings of the 2025 CHI Conference on Human Factors in Computing...

    2025

  48. [56]

    Interactive debugging and steering of multi-agent ai systems

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang (Eric) Zhu, and Saleema Amershi. Interactive debugging and steering of multi-agent ai systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY...

    2025

  49. [57]

    Teaching models to express their uncertainty in words.arXiv preprint arXiv:2205.14334, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words.arXiv preprint arXiv:2205.14334, 2022

    2022 arXiv

  50. [58]

    Lacie: Listener-aware finetuning for confidence calibration in large language models.arXiv preprint arXiv:2405.21028, 2024

    Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. Lacie: Listener-aware finetuning for confidence calibration in large language models.arXiv preprint arXiv:2405.21028, 2024

    2024 arXiv

  51. [59]

    Morae: Proactively pausing ui agents for user choices.arXiv preprint arXiv:2508.21456, 2025

    Yi-Hao Peng, Dingzeyu Li, Jeffrey P Bigham, and Amy Pavel. Morae: Proactively pausing ui agents for user choices.arXiv preprint arXiv:2508.21456, 2025

    2025 arXiv

  52. [60]

    Directgpt: A direct manipulation interface to interact with large language models

    Damien Masson, Sylvain Malacria, G´ ery Casiez, and Daniel Vogel. Directgpt: A direct manipulation interface to interact with large language models. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2024

    2024

  53. [61]

    The sky is not the limit: multitasking across github projects

    Bogdan Vasilescu, Kelly Blincoe, Qi Xuan, Casey Casalnuovo, Daniela Damian, Premkumar Devanbu, and Vladimir Filkov. The sky is not the limit: multitasking across github projects. InProceedings of the 38th International Conference on Software Engineering, pages 994–1005, 2016

    2016

  54. [62]

    Principles of mixed-initiative user interfaces

    Eric Horvitz. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159–166, 1999. 20 Figure 6: In the chat panel, users can select from ask, edit, and agent mode. In both edit and agent mode, GitHub...

    1999

  55. [63]

    A u t o m a t i c a l l y create a dataset of at least 50 home sales in Pearl City , Hawaii from 2021 -2023 , i nc lu di ng : - Address - Sale date - Sale price - Square footage - Number of bedrooms / ba th ro om s - Year built - Any other relevant features Note this must incl...

    2021

  56. [64]

    Provide the exact sale price of the higher - priced property in a file called result

    D et e rm in e which of these two p r o p e r t i e s in Pearl City , Hawaii sold for more in 2022. Provide the exact sale price of the higher - priced property in a file called result . txt

    2022

  57. [65]

    In a Python script or Jupyter notebook , analyze the dataset and provide analyses for : - Current es ti ma te d value of a typical home in the area - Best time to sell based on seasonal trends ( if any ) - Which home i m p r o v e m e n t s might yield the best return on i n v...

  58. [66]

    A u t o m a t i c a l l y create a c o m p r e h e n s i v e dataset of two shows i nc lu di ng : - Survivor : * Season number * Year aired * Winner name * Runner - up ( s ) * Location * Number of c o n t e s t a n t s * V i e w e r s h i p data - American Idol : * Season numb...

  59. [67]

    Provide the exact numbers for each show and the d i f f e r e n c e between them in a file called result

    D et e rm in e how many more unique winners there have been in the American version of Survivor compared to American Idol as of the end of Survivor ’ s 44 th season . Provide the exact numbers for each show and the d i f f e r e n c e between them in a file called result . txt

  60. [68]

    READ SERR 1 2

    In a Python script or Jupyter notebook , provide v i s u a l i z a t i o n s to compare both shows : - D e m o g r a p h i c s of winners ( age , gender , b a c k g r o u n d ) - V i e w e r s h i p trends over time ( with v i s u a l i z a t i o n s ) - Analysis of how both s...

  61. [69]

    Create a new function called ‘ s m o o t h _ t r a n s i t i o n ( from_data , to_data , duration =1.0 , fps =30 , ** kwargs ) ‘ that : - Takes initial and final data states - Creates a smooth an im at io n t r a n s i t i o n i n g between the states - Supports di ff er en t ...

  62. [70]

    I mp l em en t a c o m p l e m e n t a r y function ‘ t r a n s i t i o n _ p l o t _ s t a t e ( fig_from , fig_to , duration =1.0 , fps =30) ‘ that can t r a n s i t i o n between two c o m p l e t e l y di ff er en t figure states

  63. [71]

    ## R e q u i r e m e n t s

    Create a demo s h o w c a s i n g various t r a n s i t i o n types # Task : Add Dark Mode Toggle to M a t p l o t l i b ## O bj e ct iv e Add a new f u n c t i o n a l i t y that allows users to toggle any existing plot to dark mode with a single function call . ## R e q u i ...

  64. [72]

    Create a new function called ‘ t o g g l e _ d a r k _ m o d e ( ax = None , fig = None ) ‘ that : - Can be applied to either a specific axis , a figure , or the current figure if none is sp ec if ie d - Converts the plot b a c k g r o u n d to a dark color ( e . g . , #121212...

  65. [73]

    The function should be r e v e r s i b l e ( calling it again should toggle back to light mode )

  66. [74]

    How competent is OpenHands compared to a human programmer?

    Create a simple demo script showing the f u n c t i o n a l i t y in action 26 C Study Details C.1 Instructions All instructions are sent to participants in a google document. The order of phases are randomized accordingly, as shown in Figure 3. General instructions. • Timing:...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.