Pith. sign in

REVIEW 4 major objections 6 minor 68 references

When a simulated user edits code in the middle of a repair, coding agents' resolve rate drops 7.7 points on average, and the drop varies from 1.3 to 16.5 points across nine models—so autonomous skill does not predict shared-workspace awaren

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 06:01 UTC pith:OHH5QM5V

load-bearing objection A solid, well-constructed benchmark with a real effect, though the synthetic edit distribution is adversarial by construction and the headline number should be read as an upper bound on realistic degradation. the 4 major comments →

arxiv 2608.02499 v1 pith:OHH5QM5V submitted 2026-08-03 cs.SE cs.AIcs.CL

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

Yuqiao Tan , Jinxiang Meng , Fangyu Lei , Minzheng Wang , Shizhu He , Jun Zhao , Kang Liu This is my paper
classification cs.SE cs.AIcs.CL
keywords coding agentsshared workspaceuser code editsbenchmarkingconflicting editsstate awarenessagent evaluationrepair robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper wants to establish a new failure dimension for coding agents: handling a shared workspace where a user edits code during an ongoing task. It introduces SWE-Touch, which mines task-critical code regions from repair trajectories, generates small, plausible Counter-Edits that conflict with the task, validates that they neither solve the task alone nor combine with the correct fix, and injects them mid-run with a contextual user message. Across nine models, the controlled edit lowers mean resolve rate by 7.7 percentage points, with per-model losses from 1.3 to 16.5 points and a substantial reshuffle of model rankings; degradation also appears on longer-horizon tasks. Trajectory audits show the failures are not one behavior: agents often keep the conflicting code, sometimes replace it with wrong code, and frequently fail to re-run targeted tests. The paper's own limitation is that this is a controlled, simulated stress test rather than the full distribution of real collaborative user behavior.

Core claim

The central claim is that current coding agents lack reliable state awareness in an evolving workspace: when a user's task-conflicting edit lands in code the agent is working on, most models do not robustly detect the change, reconcile it with the task, and verify the affected behavior before continuing. On a 200-task sample from a standard repository-repair benchmark, the Counter-Edit condition lowered mean resolve rate by 7.7 points across nine models; the strongest autonomous models lost only 1.3-1.8 points, while several competitive open-source models lost 11-16.5 points and dropped multiple ranks. A control that injects a helpful, non-solving edit (Co-Edit) changed resolve rate by only

What carries the argument

The load-bearing object is the Counter-Edit: a small, syntactically valid unified diff that encodes a confident but mistaken developer belief and conflicts with task completion. SWE-Touch builds them in three steps. First, task-critical regions are mined by intersecting the read and edit spans of multiple repair trajectories for the same task. Second, a User Patch Generator—a separate agent that sees the issue, the critical regions, the reference repair, and the fail-to-pass tests—writes a plausible wrong patch, which is validated so that the edit alone does not solve the task, the reference repair alone does, and the two combined still fail. Third, a runtime monitor applies the patch with a

Load-bearing premise

The load-bearing premise is that the User Patch Generator's reference-aware, deliberately conflicting Counter-Edits represent real user edits in shared workspaces; the paper's own limitation section restricts the claim to controlled, simulated task-conflicting edits rather than the full distribution of collaborative user behavior.

What would settle it

Collect a corpus of real user edits made while a human and an agent work on the same repository, replay the SWE-Touch delivery pipeline with those edits, and compare the resolve-rate drop. If the drop approaches zero or the model ranking stops reshuffling, the central claim that agents are not state-aware in shared workspaces would fail to generalize beyond the synthetic stress test.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Autonomous resolve rate is not a reliable predictor of shared-workspace performance: models within about one point of each other on vanilla tasks can differ by more than 10 points once a conflicting user edit is injected.
  • Rankings from static, single-agent benchmarks can mislead deployment choices: several models that look interchangeable in isolation separate sharply under user edits.
  • Message-only interaction benchmarks miss a key channel: removing the code edit and sending only a message had limited, inconsistent effects, while the code edit drove consistent degradation.
  • Robustness training for coding agents needs to target three concrete capabilities: detecting external workspace changes, reconciling conflicting edits with task requirements, and re-validating affected behavior with targeted tests.
  • Additional compute is not a proxy for recovery: most models spent more steps and tokens under Counter-Edit without converting that effort into higher resolve rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the injected edits are generated with access to the reference repair and deliberately target task-critical code, the 7.7-point drop is likely an upper bound on what typical, less targeted user edits would cause—but real edits are also messier and may persist differently, so the real-world gap could be larger in other ways.
  • A direct extension would be to give agents an explicit workspace-diff tool and measure whether the drop closes; the paper's failure analysis predicts most of the loss would disappear if agents reliably inspected user-modified regions before editing.
  • The same generator could be inverted to study integration rather than conflict: injecting partially correct or complementary user edits would test whether agents can build on helpful contributions instead of only defending against harmful ones.
  • The correlation between edit-revision rate and performance suggests a training signal: reward agents for challenging conflicting code only when they also verify the replacement with targeted tests, since 28% of counteracting trajectories still ended unresolved.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SWE-Touch, a benchmark framework that augments repository-level coding-agent evaluation with simulated user edits to the shared codebase during an ongoing repair. Task-critical regions are mined from multiple repair trajectories, a separate User Patch Generator produces small, syntactically valid edits that are validated to block the reference repair, and these Counter-Edits are injected, along with contextual user messages, when the agent touches the relevant code. The authors evaluate nine coding models on 200 SWE-bench Verified tasks, with additional experiments on SWE-Bench Pro and DeepSWE. The main finding is that Counter-Edit reduces the mean resolve rate by 7.7 percentage points, with model-level losses between 1.3 and 16.5 points and substantial rank changes. Ablation and control experiments suggest the drop is specific to task-conflicting edits rather than to external modifications in general, and a two-labeler trajectory audit attributes failures to retained conflicts, incorrect replacement, incomplete reconciliation, and insufficient verification. The paper concludes that strong autonomous performance does not translate into robustness to evolving shared-workspace state.

Significance. If the results hold, SWE-Touch fills a genuine gap in coding-agent evaluation: existing interactive benchmarks restrict user participation to messages, while SWE-Touch directly perturbs the executable repository state. The benchmark construction is careful in several respects: edits are validated by Eq. (3), the Co-Edit control helps isolate conflict-specific effects, and the failure audit uses two independent labelers with 93.9% agreement. The finding that some strong autonomous models degrade substantially under Counter-Edit, and that the degradation is not predicted by Vanilla resolve rate, is a useful and falsifiable result. The authors also ship reproducible artifacts and provide trajectory-level case studies. The main caveats are that the measured drop characterizes a deliberately adversarial, reference-aware edit distribution rather than the full distribution of real user behavior, and that some supporting analyses have statistical and coverage gaps. With appropriate tightening of claims and reporting, this would be a valuable contribution to the community.

major comments (4)
  1. [§5.1, Table 3] The headline claim that Counter-Edit lowers mean resolve rate by 7.7 points is reported as a point estimate without a confidence interval or statistical test. Table 3 gives run-level standard deviations per model, but the aggregate 7.7-point average is not accompanied by uncertainty, and the model-level deltas are not tested (e.g., paired bootstrap over tasks, or a signed-rank test over models). The direction is robust because all nine models have negative deltas, but the exact magnitude and the extent of rank reshuffling need uncertainty bounds. Please add bootstrap CIs for the aggregate and per-model differences, and report variance for the longer-horizon results in Table 4, which currently show only two-run means without errors.
  2. [§5.3, Table 6] The Co-Edit control, which is central to the claim that the drop is conflict-specific rather than due to any external modification, excludes Claude Opus 4.8 and Qwen3-Coder-480B. These are exactly the models with the smallest and largest Counter-Edit losses (-1.8 and -16.5 points in Table 3). The paper does not explain the omission. Across the seven included models the average Co-Edit effect is -0.1 points, but including the two excluded models could materially change this. In addition, Co-Edit is delivered once after the agent changes the target region, whereas Counter-Edit is delivered up to three times under region-triggered scheduling; the two conditions are not matched for frequency. Either extend the Co-Edit control to all nine models, or qualify the interpretation to the seven-model subset and discuss the delivery mismatch.
  3. [§3.3, Eq. (3)] The validation condition VF_i(R^{-*}_i)=0 shows that the reference repair cannot be composed with the Counter-Edit to pass the fail-to-pass tests. It does not show that no correct repair can coexist with the user edit. An alternative correct patch could, in principle, accommodate the edit and still pass the verifier, in which case the edit is not strictly 'task-conflicting' in the sense implied by the term 'validated Counter-Edit.' The paper's own transition data (59 pairs that move from unresolved to solved under Counter-Edit) suggests that at least some edits do not irreconcilably block completion. Please either rename the condition to something like 'reference-conflicting,' weaken the validation claim, or add a stronger check (e.g., verifying that a set of reasonable candidate repairs also fail when composed with the edit).
  4. [§3.3, §5, §7 (Limitations)] The Counter-Edits are constructed with access to the reference repair and the task's fail-to-pass tests (Appendix E.1), and the user simulator is explicitly instructed to be confidently persistent and protective of its incorrect edit (Table 9: 'mistaken, not malicious'; Stage 3: 'insist that the agent stop reverting or rewriting your implementation'). The measured 7.7-point drop and the associated rank changes are therefore properties of a deliberately adversarial, reference-aware edit distribution, not of ordinary user edits. The Limitations section states this clearly, but the abstract and conclusion present the result as what happens 'when users touch the code' and as evidence about 'shared-workspace collaboration.' Please move the scope caveat into the abstract and conclusion, and consider framing the result as an upper bound or stress-test characterization unless additional evidence
minor comments (6)
  1. [§3.4 / §3.3] The terminology 'simulator' is used for the message generator in Table 9, but §3.3 also introduces a 'User Patch Generator.' It would help to explicitly state that the User Patch Generator (which receives the reference patch for edit construction) is distinct from the runtime message simulator (which does not receive the reference patch or verifier information), to avoid the apparent contradiction in §3.4.
  2. [Table 3] The 'Rank Δ' column shows '–' for Claude Opus 4.8, GPT 5.5, and Qwen3-Coder-480B. If this means no rank change, please state so explicitly rather than leaving a dash. Also clarify whether the rank changes are based on mean resolve rate or on some other aggregation.
  3. [Figure 4] Panel (a) lists seven failure categories, but panel (b) aggregates them into four legend items ('Retained conflict', 'Incorrect replacement', 'Incomplete reconciliation', 'Other'). Please explain how the seven labels are grouped for panel (b), or keep the legend consistent.
  4. [§4 / Appendix B.1] The paper says 5,399 scored Counter-Edit runs, but 200 tasks × 9 models × 3 runs is 5,400. Please account for the missing run or correct the number.
  5. [§5.1 / Table 3] The sentence 'Seven of nine agents use both more calls and more tokens under Counter-Edit' is followed by 'GLM 5.1 adds 9.5 calls and 265K tokens' — the table reports tokens in thousands (Tok. (K)), so 265K is consistent, but please verify all token deltas are expressed in the same unit.
  6. [§B.2] The longer-horizon extension uses 25 tasks from SWE-Bench Pro and DeepSWE, 'using the same task sets as SWE-Interact.' Since task selection can materially affect outcomes, please include the exact task IDs or a link to the released task list.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an empirical measurement under externally validated perturbations, not a derivation that reduces to its inputs.

full rationale

The paper's central claim—that Counter-Edit lowers mean resolve rate by 7.7 points on SWE-bench Verified—is an empirical measurement, not a derivation from the benchmark construction. The construction in Eq. (3) guarantees that the injected edit alone does not solve the task and that the edit combined with the reference repair still fails, but it does not determine what the evaluated agents will do. Indeed, Figure 6 shows that 59 majority-unresolved Vanilla pairs became solved under Counter-Edit, so the measured degradation is not analytically forced by the validation equation. The edits are fixed before evaluation, applied identically to all models, and scored by the original SWE-bench verifier, so the outcome is not fitted to a parameter that is then called a prediction. The User Patch Generator's access to the reference patch and fail-to-pass tests is a design choice for controlled stress-testing, and the paper explicitly limits its claim in §7 to 'controlled, simulated task-conflicting edits rather than the full distribution of collaborative user behavior'; this is a generalizability limitation, not circularity. Self-citations in related work ([19], [44]) are not load-bearing: the framework does not rest on an unverified result from the same authors. No equation reduces to its own inputs, and no fitted value is relabeled as a prediction. The benchmark is self-contained against an external, frozen task set with executable verification, which is the kind of independent evidence that keeps the circularity score at 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central result rests on the benchmark's construct validity (Counter-Edits are conflicting and non-solving) and external validity (they resemble real user edits). Internal validity is supported by the Eq. (3) validation and controls; external validity is the main unproven premise, acknowledged in §7. The headline measurement also depends on design choices K=3 and the 200-task cohort.

free parameters (2)
  • Intervention budget K (default 3) = 3
    Hand-chosen number of delivery attempts per task in the main evaluation; ablated across 1, 3, 5 (Table 5, Figure 3b). The magnitude of the resolve-rate drop depends on K, so the headline 7.7-point figure is conditional on this choice.
  • Main task cohort size = 200
    Seeded random sample of SWE-bench Verified tasks with complete trajectories from all three region-mining models. This selection affects the baseline resolve rates and the estimated drop; a different sample would give a different average.
axioms (5)
  • domain assumption SWE-bench Verified verifier outcomes (fail-to-pass and pass-to-pass tests) are a valid and complete measure of task resolution for the agent evaluation.
    The entire resolve-rate metric rests on this; if the verifier is leaky or incomplete, the measured drops could be artifacts. SWE-bench is an external benchmark, so this is a standard domain assumption rather than an ad-hoc-to-paper postulate.
  • domain assumption The three-state validation in Eq. (3) operationally defines a Counter-Edit as conflicting with task completion without solving it.
    The construction pipeline assumes that V_F(R−)=0, V_F(R*)=1, V_F(R−*)=0 is sufficient for a plausible conflicting user edit; in 42/192 cases the reference patch no longer applies, which is a weaker validation but still included.
  • domain assumption A User Patch Generator with access to the reference repair and fail-to-pass tests produces edits representative of real user edits that conflict with a task.
    External validity of the benchmark depends on this. Acknowledged in §7 as a limitation; no human-user validation is performed.
  • domain assumption Region-triggered delivery—applying the edit when the agent accesses an overlapping code region, up to K times—approximates how a real user would intervene in a shared editing session.
    The evaluation protocol assumes this trigger rule does not systematically distort agent behavior relative to real-world intervention timing. Ablations show K dependence, but the trigger itself is not validated against human data.
  • domain assumption The simulated 'Miscalibrated Builder' user messages are a realistic representation of how a mistaken but confident user would communicate.
    Message content affects agent behavior; the ablated message-only condition shows limited effect, but the interaction between message and edit could be sensitive to wording.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Touch: Benchmarking Coding Agents When Users Touch the Code." pith.science (2026). https://pith.science/paper/OHH5QM5V

@misc{pith2026260802499,
  author       = {Pith},
  title        = {Pith review of: SWE-Touch: Benchmarking Coding Agents When Users Touch the Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHH5QM5V}},
  note         = {Machine review of arXiv:2608.02499}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 2 linked inside Pith

  1. [1]

    Introducing Claude Fable 5

    Anthropic. Introducing Claude Fable 5. Model release, 2026

    2026

  2. [2]

    Introducing Claude Opus 4.8

    Anthropic. Introducing Claude Opus 4.8. Model release, 2026

    2026

  3. [3]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models, 2021

    2021

  4. [4]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ2-bench: Evaluating conversational agents in a dual-control environment, 2025

    2025

  5. [5]

    SWE-chat: Coding agent interactions from real users in the wild, 2026

    Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. SWE-chat: Coding agent interactions from real users in the wild, 2026

    2026

  6. [6]

    Evaluating large language models trained on code, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code, 2021

    2021

  7. [7]

    Tran,CalvinSmith,AmeetTalwalkar,andGrahamNeubig

    Valerie Chen, Rohit Malhotra, Xingyao Wang, Juan Michelini, Xuhui Zhou, Aditya Bharat Soni, Hoang H. Tran,CalvinSmith,AmeetTalwalkar,andGrahamNeubig. Howcanweassesshuman-agentinteractions? case studies in software agent design. InProceedings of the 43rd International Conference on Machine Learning, 2026. arXiv:2510.09801

    Pith/arXiv arXiv 2026

  8. [8]

    EDIT-Bench:Evaluating LLM abilities to perform real-world instructed code edits, 2025

    Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos,IonStoica,GrahamNeubig,AmeetTalwalkar,andChrisDonahue. EDIT-Bench:Evaluating LLM abilities to perform real-world instructed code edits, 2025

    2025

  9. [9]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified. OpenAI research publication, 2024. Updated February 24, 2025

    2024

  10. [10]

    FrontierSWE

    Evan Chu, Rajan Agarwal, Abishek Thangamuthu, Brendan Graham, Justus Mattern, Freeman Jiang, Paul Cento, Swarnim Jain, Mersad Abbasi, Mohammad Hossein Rezaei, George Wang, Alex Zhang, Simon Guo, Karina Nguyen, Danna Liu, Arash Bidgoli, Aditya Dalmia, Apoorv Dankar, Ashrut Vaddela, Calvin Chen, Keshav Kumar, Kushagra Vaish, Navid Pour, Rishyanth Kondra, Sa...

    2026

  11. [11]

    DeepSeek V4 Preview release

    DeepSeek-AI. DeepSeek V4 Preview release. API documentation and model release, 2026

    2026

  12. [12]

    SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, 13 SWE-Touch: Benchmarking Coding Agents When Users Touch the...

    2025

  13. [13]

    Swe-marathon: Can agents autonomously complete ultra-long-horizon software work?, 2026

    Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, et al. Swe-marathon: Can agents autonomously complete ultra-long-horizon software work?, 2026

    2026

  14. [14]

    Ask or assume? uncertainty-aware clarification-seeking in coding agents, 2026

    Nicholas Edwards and Sebastian Schuster. Ask or assume? uncertainty-aware clarification-seeking in coding agents, 2026

    2026

  15. [15]

    Coding agents don’t know when to act, 2026

    Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev. Coding agents don’t know when to act, 2026

    2026

  16. [16]

    SyncMind: Measuring agent out-of-sync recovery in collaborative software engineering

    Xuehang Guo, Xingyao Wang, Yangyi Chen, Sha Li, Chi Han, Manling Li, and Heng Ji. SyncMind: Measuring agent out-of-sync recovery in collaborative software engineering. InProceedings of the 42nd International Conference on Machine Learning. PMLR, 2025

    2025

  17. [17]

    Harbor: A framework for evaluating and optimizing agents and models in container environments

    Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. Software framework, version v0.16.1, 2026. Zenodo release

    2026

  18. [18]

    DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks, 2026

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks, 2026

    2026

  19. [19]

    DA-code: Agent data science code generation benchmark for large language models

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. DA-code: Agent data science code generation benchmark for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13487–13521, 2024

    2024

  20. [20]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?, 2023

    2023

  21. [21]

    Dialogue swe-bench: A benchmark for dialogue-driven coding agents, 2026

    Brendan King and Jeffrey Flanigan. Dialogue swe-bench: A benchmark for dialogue-driven coding agents, 2026

    2026

  22. [22]

    Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios, 2025

    Tue Le, Minh VT Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi DQ Bui. Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios, 2025

    2025

  23. [23]

    Liang, Chenyang Yang, and Brad A

    Jenny T. Liang, Chenyang Yang, and Brad A. Myers. A large-scale survey on the usability of AI programming assistants: Successes and challenges. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. ACM, 2024

    2024

  24. [24]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026

    2026

  25. [25]

    MiniMax M2.5: Built for real-world productivity

    MiniMax. MiniMax M2.5: Built for real-world productivity. Model release, 2026

    2026

  26. [26]

    MiniMax M2.7: Early echoes of self-evolution

    MiniMax. MiniMax M2.7: Early echoes of self-evolution. Model release, 2026

    2026

  27. [27]

    Kimi-K2.6

    Moonshot AI. Kimi-K2.6. Model card, 2026. Model record created April 14, 2026

    2026

  28. [28]

    Moonshot AI. Kimi K3. Model card, 2026. 14 SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

    2026

  29. [29]

    GPT-4o system card

    OpenAI. GPT-4o system card. System card, 2024

    2024

  30. [30]

    Introducing GPT-5.5

    OpenAI. Introducing GPT-5.5. Technical report, 2026

    2026

  31. [31]

    Slopcodebench: Benchmarking how coding agents degrade over long-horizon iterative tasks, 2026

    Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, and Aws Albarghouthi. Slopcodebench: Benchmarking how coding agents degrade over long-horizon iterative tasks, 2026

    2026

  32. [32]

    When benchmarks talk: Re-evaluating code LLMs with interactive feedback, 2025

    Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, and Valerie Chen. When benchmarks talk: Re-evaluating code LLMs with interactive feedback, 2025

    2025

  33. [33]

    Icae-bench: Evaluating coding agents as interactive project builders, 2026

    Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, David Lo, Lin Qiu, Xuezhi Cao, Jiyuan He, and Yixin Cao. Icae-bench: Evaluating coding agents as interactive project builders, 2026

    2026

  34. [34]

    UserBench: An interactive gym environment for user-centric agents, 2025

    Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. UserBench: An interactive gym environment for user-centric agents, 2025

    2025

  35. [35]

    Qwen3-Coder: Agentic coding in the world

    Qwen Team. Qwen3-Coder: Agentic coding in the world. Model release, 2025

    2025

  36. [36]

    Qwen3.7: the agent frontier

    Qwen Team. Qwen3.7: the agent frontier. Model release, 2026

    2026

  37. [37]

    SWE-INTERACT: Reimagin- ing SWE benchmarks as user-driven long-horizon coding sessions, 2026

    Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, and Yunzhong He. SWE-INTERACT: Reimagin- ing SWE benchmarks as user-driven long-horizon coding sessions, 2026

    2026

  38. [38]

    Collaborative gym: A framework for enabling and evaluating human-agent collaboration, 2024

    Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration, 2024

    2024

  39. [39]

    Evocode-bench: Evaluating coding agents in multi-turn iterative interactions, 2026

    Haiyang Shen, Xuanzhong Chen, Wendong Xu, Yun Ma, Liang Chen, and Kuan Li. Evocode-bench: Evaluating coding agents in multi-turn iterative interactions, 2026

    2026

  40. [40]

    Non-collaborative user simulators for tool agents

    Jeonghoon Shim, Woojung Song, Cheyon Jin, Seungwon Kook, and Yohan Jo. Non-collaborative user simulators for tool agents. InInternational Conference on Learning Representations, 2026

    2026

  41. [41]

    Hedwig:Dynamic autonomy for coding agents under local oversight, 2026

    TanjalShukla,K.J.KevinFeng,LeijieWang,MohammadRostami,andAmyX.Zhang. Hedwig:Dynamic autonomy for coding agents under local oversight, 2026

    2026

  42. [42]

    HiL- Bench (human-in-loop benchmark): Do agents know when to ask for help?, 2026

    Tu Trinh, Mohamed Elfeki, Guangze Luo, Kelvin Luu, Nathan Hunt, Ernesto Hernandez, Nandan Marwaha, Yannis Yiming He, Charles Wang, Fernando Carabedo, Alessa Castillo, and Bing Liu. HiL- Bench (human-in-loop benchmark): Do agents know when to ask for help?, 2026

    2026

  43. [43]

    Ambig-SWE: Interactive agents to overcome underspecificity in software engineering, 2025

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. Ambig-SWE: Interactive agents to overcome underspecificity in software engineering, 2025

    2025

  44. [44]

    DEMO: Reframing dialogue interaction with fine-grained element modeling

    Minzheng Wang, Xinghua Zhang, Kun Chen, Nan Xu, Haiyang Yu, Fei Huang, Wenji Mao, and Yongbin Li. DEMO: Reframing dialogue interaction with fine-grained element modeling. InFindings of the Association for Computational Linguistics: ACL 2025, pages 11373–11401, Vienna, Austria, 2025. Association for Computational Linguistics

    2025

  45. [45]

    Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik R

    Zora Zhiruo Wang, John Yang, Kilian Lieret, Alexa R. Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik R. Narasimhan, Ludwig Schmidt, Graham Neubig, Daniel Fried, and Diyi Yang. Position: Humans are missing from AI coding agent research. OpenReview preprint, 2026. 15 SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

    2026

  46. [46]

    SWE-Together: Evaluating coding agents in interactive user sessions, 2026

    Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, and Shengzhi Li. SWE-Together: Evaluating coding agents in interactive user sessions, 2026

    2026

  47. [47]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InThe Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  48. [48]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. InProceedings of the 39th Annual Conference on Neural Information Processing Systems (NeurIPS 2025 D&B Spotlight), 2025. arXiv:2504.21798, accepted at ...

    Pith/arXiv arXiv 2025

  49. [49]

    Codeclash: Benchmarking goal-oriented software engineering, 2025

    John Yang, Kilian Lieret, Joyce Yang, Carlos E Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Codeclash: Benchmarking goal-oriented software engineering, 2025

    2025

  50. [50]

    Programbench: Can language models rebuild programs from scratch?, 2026

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. Programbench: Can language models rebuild programs from scratch?, 2026

    2026

  51. [51]

    Talk2code: A multi-turn interaction benchmark with dual-track evaluation for code generation.Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34331–34339, 2026

    Weibin Yang, Liangru Xie, Jieyun Cai, Yuxiang Yan, Hong-Ning Dai, and Hao Wang. Talk2code: A multi-turn interaction benchmark with dual-track evaluation for code generation.Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34331–34339, 2026

    2026

  52. [52]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool- agent-user interaction in real-world domains, 2024

    2024

  53. [53]

    Z.ai. GLM-5.1. Developer documentation, 2026

    2026

  54. [54]

    SWE-Explore: Benchmarking how coding agents explore repositories, 2026

    Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, Wenhao Zeng, Maoquan Wang, Shilin He, Ningyuan Xu, Siyu Ye, Kai Cai, and Xiaodong Gu. SWE-Explore: Benchmarking how coding agents explore repositories, 2026

    2026

  55. [55]

    TOM-SWE: User mental modeling for software engineering agents, 2025

    Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang, Graham Neubig, Maarten Sap, and Xingyao Wang. TOM-SWE: User mental modeling for software engineering agents, 2025

    2025

  56. [56]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, et al. Bigcodebench: Benchmarking code generation with diverse function cal...

    2025

  57. [58]

    Edit regions in implementation files, intersected across trajectories with nonempty edits

  58. [59]

    Edit regions in any non-noise file

  59. [60]

    Any remaining edit region

  60. [61]

    Read regions in implementation files, intersected across trajectories with nonempty reads

  61. [62]

    Read regions in any non-noise file

  62. [63]

    Implementation files are those that appear in both the read intersection and the files modified by at least one trajectory

    Any remaining read region. Implementation files are those that appear in both the read intersection and the files modified by at least one trajectory. Tests,pyproject.toml, and metadata are excluded at tiers 1–2 and 4–5. Adjacent or overlapping intervals are merged, and we keep at most eight regions. If all tiers are empty, we fall back to a region change...

  63. [64]

    Read the software task and identify the required behavior

  64. [65]

    Compare the user-applied edit with the agent's subsequent edits and final repository evidence

  65. [66]

    Use verifier output to identify the concrete remaining failure when available

  66. [67]

    Determine whether the user edit was retained, replaced incorrectly, only partially reconciled, or led the agent away from the required implementation

  67. [68]

    primary_failure_mode

    Choose the narrowest label supported by direct evidence. If evidence is inadequate, use `INSUFFICIENT_EVIDENCE`rather than guessing. ## Evidence requirements - Cite only event IDs supplied in the input, including`V_REPORT`,`V_STDOUT`, and`V_STDERR`. - Use at least two evidence citations when two or more relevant sources are available. - Prefer concrete di...

  68. [2025]

    OpenReview.net, 2025. A. Ethics and Data Governance The evaluation uses only synthetic messages and edits; no human participants are involved. Released artifacts include only aggregate statistics and exclude credentials and private requests. Trajectory release follows upstream licenses. 16 SWE-Touch: Benchmarking Coding Agents When Users Touch the Code Mo...

    2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.