The two main benchmarks for LLM instructed code editing over-represent Python, miss common real-world domains and edit types, and have test coverage issues that limit what they measure.
hub
The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
47 Pith papers cite this work. Polarity classification is still indexing.
47
Pith papers citing it
abstract
The future of software engineering--SE 3.0--is unfolding with the rise of AI teammates: autonomous, goal-driven systems collaborating with human developers. Among these, autonomous coding agents are especially transformative, now actively initiating, reviewing, and evolving code at scale. This paper introduces AIDev, the first large-scale dataset capturing how such agents operate in the wild. Spanning over 456,000 pull requests by five leading agents--OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code--across 61,000 repositories and 47,000 developers, AIDev provides an unprecedented empirical foundation for studying autonomous teammates in software development. Unlike prior work that has largely theorized the rise of AI-native software engineering, AIDev offers structured, open data to support research in benchmarking, agent readiness, optimization, collaboration modeling, and AI governance. The dataset includes rich metadata on PRs, authorship, review timelines, code changes, and integration outcomes--enabling exploration beyond synthetic benchmarks like SWE-bench. For instance, although agents often outperform humans in speed, their PRs are accepted less frequently, revealing a trust and utility gap. Furthermore, while agents accelerate code submission--one developer submitted as many PRs in three days as they had in three years--these are structurally simpler (via code complexity metrics). We envision AIDev as a living resource: extensible, analyzable, and ready for the SE and AI communities. Grounding SE 3.0 in real-world evidence, AIDev enables a new generation of research into AI-native workflows and supports building the next wave of symbiotic human-AI collaboration. The dataset is publicly available at https://github.com/SAILResearch/AI_Teammates_in_SE3. > AI Agent, Agentic AI, Coding Agent, Agentic Coding, Software Engineering Agent
hub tools
citation-role summary
background 2
dataset 1
citation-polarity summary
years
2026 47representative citing papers
AI coding agent adoption causes no change in human contributor count but reduces contributor density and newcomer share by 3.7pp while increasing review depth by 5.3% in a staggered DiD analysis of 11k GitHub projects.
Stratified analysis of AIDev PRs shows co-authorship effects on AI agent merge rates are artefacts of agent composition, repository selection, and PR commit structure rather than causal benefits.
Observational study of 20,574 sessions identifies seven misalignment forms where 90.5% cause effort/trust costs and 91.5% require explicit user correction, varying by interface and over time.
AI agents modify logging less often than humans in 58.4% of repositories but produce higher log density when they change it; explicit logging instructions are rare (4.7%) and ignored 67% of the time, with humans performing 72.5% of post-generation log repairs.
AgenticFlict is a public dataset of 29K+ textual merge conflicts from AI agent PRs, collected via merge simulation on 107K processed PRs and showing a 27.67% conflict rate with variation across agents.
AI-generated TypeScript code uses the 'any' type 9x more often than human code and employs more advanced type constructs that can ignore checks, but agentic PRs have 1.8x higher acceptance rates.
AgenticSZZ reframes bug-inducing commit identification as temporal knowledge graph search navigated by an LLM agent, reporting F1 scores of 0.47-0.79 and up to 34% improvement over prior SZZ methods on three datasets.
AI coding agents produce pull requests with substantially more commits and slightly higher description-to-diff similarity than human developers, based on analysis of 29,095 merged PRs.
Review is the control point for AI-authored code: teams set the sign of AI’s effect through reviewer expertise and process design, not the technology itself.
AI repair agents solve bugs more reliably when reports include executable reproduction scripts, file-level localization cues, and clear structure, while longer prose reports and human-oriented steps to reproduce show no benefit or hurt.
In OSS repos that commit AI chat logs, AI use is heavier in smaller, less collaborative projects; chats almost always precede commits, quality signals do not broadly worsen, and developers trust their own AI code more than others'.
Study of 930k+ agent PRs shows repository explains ~50% of integration friction variance, with agents concentrating it twice as much as humans (ICC 0.30 vs 0.16) after controls.
Within-reviewer analysis of 11,429 reviews shows AI code approval rising from 30.1% to 36.8% with experience, with reduced inline comments and increased latency, consistent with habituation.
An empirical study of 86,156 test patches from five AI agents finds 80.2% lack strong oracle signals, with strong oracles linked to higher merge rates (OR=1.28) after regression controls.
Coding benchmarks conflate the model with the harness, environment, and verifier into a single end-to-end score, which is misaligned with agentic software engineering.
Qualitative analysis of 306 rejected AI-generated PRs reveals 14 rejection reasons in four categories, highlighting needs for better guidance on implementation, validation, and prioritization.
Mixed-methods study creates taxonomy of AI IDE rules from 7310 instances, analyzes evolution drivers, and reports that rule updates raise average artifact compliance from 49.14% to 72.13%.
A new multilingual benchmark reveals that UniLog leads in log statement generation across languages while Python proves harder than JavaScript, showing that scaling alone is insufficient and language-aware tailoring is required.
Analysis of 9,799 human-reviewed agentic PRs shows only 35.7% of rejections reflect clear agent failures, with 31.2% due to workflow constraints and 33.1% lacking clear rationale, plus notable interaction differences across agents.
Empirical analysis of 4707 MoltBook posts shows AI-only technical discourse focuses on security, trust, and abstract topics while lacking concrete runtime and project details found in human GitHub discussions.
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
Hot fixes show urgency patterns with reduced collaboration and testing, differing from regular fixes, and human versus AI agents display over 10 distinct repair behaviors in large-scale GitHub data.
Reviewer bots' higher comment volume on AI agent PRs is associated with slower resolutions and poorer average feedback quality, while feedback quality itself has no association with PR outcomes.
citing papers explorer
-
Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks
The two main benchmarks for LLM instructed code editing over-represent Python, miss common real-world domains and edit types, and have test coverage issues that limit what they measure.
-
Augmentation with Dilution: A Large-Scale Empirical Study of Human Contributor Ecosystems After AI Coding Agent Adoption
AI coding agent adoption causes no change in human contributor count but reduces contributor density and newcomer share by 3.7pp while increasing review depth by 5.3% in a staggered DiD analysis of 11k GitHub projects.
-
Beyond Simpson's Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship
Stratified analysis of AIDev PRs shows co-authorship effects on AI agent merge rates are artefacts of agent composition, repository selection, and PR commit structure rather than causal benefits.
-
How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
Observational study of 20,574 sessions identifies seven misalignment forms where 90.5% cause effort/trust costs and 91.5% require explicit user correction, varying by interface and over time.
-
Do AI Coding Agents Log Like Humans? An Empirical Study
AI agents modify logging less often than humans in 58.4% of repositories but produce higher log density when they change it; explicit logging instructions are rare (4.7%) and ignored 67% of the time, with humans performing 72.5% of post-generation log repairs.
-
AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
AgenticFlict is a public dataset of 29K+ textual merge conflicts from AI agent PRs, collected via merge simulation on 107K processed PRs and showing a 27.67% conflict rate with variation across agents.
-
Mining Type Constructs Using Patterns in AI-Generated Code
AI-generated TypeScript code uses the 'any' type 9x more often than human code and employs more advanced type constructs that can ignore checks, but agentic PRs have 1.8x higher acceptance rates.
-
AgenticSZZ: Temporal Knowledge Graph-Guided Agentic Bug-Inducing Commit Identification
AgenticSZZ reframes bug-inducing commit identification as temporal knowledge graph search navigated by an LLM agent, reporting F1 scores of 0.47-0.79 and up to 34% improvement over prior SZZ methods on three datasets.
-
How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
AI coding agents produce pull requests with substantially more commits and slightly higher description-to-diff similarity than human developers, based on analysis of 29,095 merged PRs.
-
3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse
Review is the control point for AI-authored code: teams set the sign of AI’s effect through reviewer expertise and process design, not the technology itself.
-
What Makes a Good Bug Report for an AI Agent?
AI repair agents solve bugs more reliably when reports include executable reproduction scripts, file-level localization cues, and clear structure, while longer prose reports and human-oriented steps to reproduce show no benefit or hurt.
-
From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software
In OSS repos that commit AI chat logs, AI use is heavier in smaller, less collaborative projects; chats almost always precede commits, quality signals do not broadly worsen, and developers trust their own AI code more than others'.
-
Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software
Study of 930k+ agent PRs shows repository explains ~50% of integration friction variance, with agents concentrating it twice as much as humans (ICC 0.30 vs 0.16) after controls.
-
Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code
Within-reviewer analysis of 11,429 reviews shows AI code approval rising from 30.1% to 36.8% with experience, with reduced inline comments and increased latency, consistent with habituation.
-
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code
An empirical study of 86,156 test patches from five AI agents finds 80.2% lack strong oracle signals, with strong oracles linked to higher merge rates (OR=1.28) after regression controls.
-
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Coding benchmarks conflate the model with the harness, environment, and verifier into a single end-to-end score, which is misaligned with agentic software engineering.
-
Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
Qualitative analysis of 306 rejected AI-generated PRs reveals 14 rejection reasons in four categories, highlighting needs for better guidance on implementation, validation, and prioritization.
-
Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study
Mixed-methods study creates taxonomy of AI IDE rules from 7310 instances, analyzes evolution drivers, and reports that rule updates raise average artifact compliance from 49.14% to 72.13%.
-
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
A new multilingual benchmark reveals that UniLog leads in log statement generation across languages while Python proves harder than JavaScript, showing that scaling alone is insufficient and language-aware tailoring is required.
-
Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study
Analysis of 9,799 human-reviewed agentic PRs shows only 35.7% of rejections reflect clear agent failures, with 31.2% due to workflow constraints and 33.1% lacking clear rationale, plus notable interaction differences across agents.
-
What Software Engineering Looks Like to AI Agents? -- An Empirical Study of AI-Only Technical Discourse on MoltBook
Empirical analysis of 4707 MoltBook posts shows AI-only technical discourse focuses on security, trust, and abstract topics while lacking concrete runtime and project details found in human GitHub discussions.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
-
Hot Fixing in the Wild
Hot fixes show urgency patterns with reduced collaboration and testing, differing from regular fixes, and human versus AI agents display over 10 distinct repair behaviors in large-scale GitHub data.
-
On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories
Reviewer bots' higher comment volume on AI agent PRs is associated with slower resolutions and poorer average feedback quality, while feedback quality itself has no association with PR outcomes.
-
Insights into Security-Related AI-Generated Pull Requests
AI-generated security pull requests frequently contain a small set of recurring weaknesses, with many flawed ones merged and rejections driven by process factors rather than technical issues.
-
ORBIT: Guided Agentic Orchestration for Autonomous C-to-Rust Transpilation
ORBIT achieves 100% compilation success and 91.7% test success on 24 mostly large programs from CRUST-Bench by using dependency-aware orchestration and iterative verification, outperforming prior static and baseline tools.
-
A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories
In real-world repositories, AI-assisted and human-written code differ only modestly on code-level metrics, while commit size, stability, duplication, and language-specific security show clearer patterns.
-
Agentic Business Process Management: A Research Manifesto
Agentic Business Process Management reframes BPM around autonomous agents that must exhibit framed autonomy, explainability, conversational actionability, and self-modification to keep their actions aligned with organizational objectives.
-
Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
An empirical study of real-world issues yields a taxonomy of 34 fault types, symptoms, and root causes in agentic AI systems, validated by 145 practitioners.
-
Governed AI-Assisted Engineering: Graduated Human Oversight for Agentic Code Generation in Regulated Domains
A four-dimension classifier routes agentic coding tasks into HITL, human-over-the-loop, or automated-with-monitoring tiers, analytically estimated to keep ~91% of ungoverned coding velocity under regulatory constraints.
-
CoT-Guard: Small Models for Strong Monitoring
CoT-Guard is a 4B model using SFT and RL that achieves 75% G-mean^2 on hidden objective detection under prompt and code manipulation attacks, outperforming several larger models.
-
These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests
AI-generated PRs on GitHub receive fewer human reviews and more AI-mediated interactions than human-authored PRs.
-
KISS Sorcar: A Stupidly-Simple General-Purpose and Software Engineering AI Assistant
The paper introduces KISS Sorcar, a simple open-source AI agent framework with a five-layer hierarchy and git worktree isolation to address context limits, error propagation, and reviewability in software engineering tasks.
-
Scaling Human-AI Coding Collaboration Requires a Governable Consensus Layer
Agentic Consensus replaces code as the main artifact with a typed property graph world model that maintains commitments and evidence through synchronization operators, shifting evaluation to alignment fidelity and consensus entropy.
-
Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects
AI IDEs with structured guidance can produce functional large-scale code but frequently introduce design flaws such as duplication, complexity, and principle violations that risk long-term maintainability.
-
From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests
Code review agents achieve 45.20% merge rate on PRs versus 68.37% for humans, with 60.2% of agent-only closed PRs showing 0-30% signal quality.
-
Beyond the 'Diff': Addressing Agentic Entropy in Agentic Software Development
Agentic entropy names the systemic drift in AI coding agents away from architectural intent; a new framework using conformity seeding, reasoning monitoring, and causal graph interfaces supplies process-level oversight to complement existing review methods.
-
From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering
Organizational policies constrain agency in AI-mediated software engineering more than individual preferences, with seniors using detailed delegation and pre-AI instincts while juniors oscillate between over-reliance and avoidance.
-
Aleena: Alignment Agent for Research Software Engineering Collaborations
Aleena is an open-source AI agent that ingests multi-modal research software collaboration artifacts and transforms them into structured GitHub records to maintain continuous stakeholder alignment across the project lifecycle.
-
How Do Developers Maintain and Evolve Their Agents' Instructions? An Empirical Study
The authors describe a research plan for mining ACF evolution, classifying changes via a maintenance taxonomy, and linking change types to code quality metrics in agent-driven repositories.
-
Human-AI Collaboration and the Transformation of Software Engineering Work
This interpretive synthesis maps three coexisting software engineering paradigms (Traditional, Generative AI-Enabled, Agentic AI-Enabled), proposes a five-category competency framework, and derives nine testable propositions about the shift toward human-AI collaboration.
-
Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency
RADAR automates review of 331K+ low-risk diffs at Meta with 1/3 the revert rate and 1/50 the production incident rate of non-RADAR diffs, while reducing median close time by over 330%.
-
Quality and Security Signals in AI-Generated Python Refactoring Pull Requests
Empirical analysis of AI refactoring PRs shows quality attribute improvements in 22.5% of cases with new Pylint issues in 24.17% and Bandit findings in 4.7%, yet 73.5% developer acceptance.
-
Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance
Task type dominates AI coding agent PR acceptance rates, with documentation at 82.1% versus 66.1% for new features, and no single agent best across all categories.
-
How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests
The authors present a registered report outlining their planned large-scale empirical study of vulnerability communication in pull requests by different account types.
-
Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Review
A rapid review of fairness in LLM-enabled multi-agent systems for the software development lifecycle concludes that the field lacks standardized evaluations, broad coverage, and effective governance, leaving it unprepared for deployable fair systems.
- Human agency in initial human-AI proof formalization workflows