Terminal Agents: A Survey of AI Agents in Command-Line Environments

Chen Qian; Haoxi Zeng; Heng Tao Shen; Jingkuan Song; Pengpeng Zeng; Wei Ye; Wencheng Ye; Wenqi Shao; Xiaoyang Yuan; Yi Bin

REVIEW 2 major objections 6 minor 197 references

Terminal Agents: A Survey of AI Agents in Command-Line Environments

T0 review · 2 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Terminal-agent behavior is jointly shaped by the model, interface, harness, runtime, and environment, so benchmark scores cannot be read as pure model capability.

desk verdict A solid, well-scoped survey whose framework and trace indicators are worth engaging; the same-model LLM judge for P2/P4/P6 is the one real soft spot, but it is auxiliary to the central synthesis. read the letter →

arxiv 2608.20485 v1 pith:PPDZIANW submitted 2026-08-20 cs.AI cs.SE

Yi Bin , Xiaoyang Yuan , Haoxi Zeng , Wencheng Ye , Wenqi Shao , Chen Qian , Wei Ye , Yujuan Ding

show 4 more authors

Zheng Wang Pengpeng Zeng Jingkuan Song Heng Tao Shen

This is my paper · ORCID

classification cs.AIcs.SE

keywords terminalagentscommand-lineinterfaceaction-observationloopagentharnessprocessevaluationbenchmarkdesignsoftwareengineeringcompetenceprofile

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that an AI agent working through a command-line terminal is best understood as a system whose progress comes from a loop of executing commands, reading textual feedback, and changing persistent state. It proposes treating that terminal-mediated loop as the defining feature of "terminal agents" and provides a seven-dimensional competence profile for comparing systems across architecture, training, and evaluation. Its synthesis claims that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment, making outer-loop design a performance-shaping system component rather than a neutral channel. The paper also argues that prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly, and it illustrates this with fixed-condition diagnostics across four benchmark families and three systems.

What carries the argument

The organizing machinery is the terminal-mediated action-observation loop together with a seven-dimensional terminal competence profile. The loop is the repeated cycle in which the agent issues commands, the terminal substrate mutates environment state, and textual feedback—stdout, stderr, exit codes, diffs, and logs—informs the next action. The profile names seven system-level responsibilities: command and action formulation, feedback and artifact interpretation, runtime and environment management, state/task/context tracking, progress verification, recovery and adaptation, and governance and side-effect control. The survey uses this profile to compare architectures, acquisition pipelines, and benchmarks, and it operationalizes process observability through seven trace-derived indicators, P1–P7, four deterministic and three judge-assisted.

What would settle it

Take the same four benchmark-family traces from the benchmark-exposure diagnostic and replace the judge used for P2, P4, and P6 with a different judge model or with human annotators, then compare the resulting process profile. If the profile changes materially—for example, if LongCLI-Bench's low semantic coverage or the feedback-use rates move—the process-observability conclusions depend on the judge rather than on the traces.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that terminal-mediated execution is a coherent organizing lens for a broad class of agents, and that the behavior of such agents is jointly shaped by the model, interface, harness, runtime, and environment. The survey treats the terminal as an execution substrate rather than a mere access surface: progress depends on command execution, textual feedback, and stateful environment interaction. Its conclusion is that outer-loop design—the harness, context management, verification, recovery, and governance—is a performance-shaping system component. Consequently, no terminal-agent benchmark score can be read as a pure measure of model capability, and the diagnostics show that benchmark choice changes which process signals become visible and that system rankings can reverse across benchmarks.

Load-bearing premise

The load-bearing premise is that the rule-constrained judge labels for P2, P4, and P6 are credible evidence about feedback use, state errors, and recovery, even though the judge is the same DeepSeek-V4-Flash model whose behavior is being evaluated in the benchmark-exposure runs; if the judge is biased, the process profile changes.

Editorial extensions

If this is right

Leaderboard scores for terminal agents should be reported with the full system configuration—model, interface, harness, runtime, and environment—since the diagnostics show the same model can look different across benchmark families.
Evaluation should add process evidence alongside final outcomes, because recovery, verification, state tracking, and governance indicators expose failures that pass-rate scoring hides.
Model comparisons require matched harness and runtime conditions; the matched diagnostic shows system ordering can reverse across benchmarks, so single-benchmark comparisons are fragile.
Acquisition pipelines should retain failure trajectories, setup attempts, and recovery decisions, because successful traces alone give weak supervision for recovery and diagnosis.
Fresh and replayable evaluations with a shared trace schema would make process-level evidence comparable across studies and support attribution analyses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the joint-shaping claim is correct, cross-benchmark capability rankings are underdetermined unless harness and runtime conditions are held fixed; a factorial study varying one component at a time on identical tasks would put the claim under direct test.
The same logic transfers to safety evaluation: governance violations are a property of the permission and sandbox policy as much as of the model, so safety scores should be reported together with the policy configuration that produced them.
The seven-dimensional profile could be turned into a standard process-report format for replayable traces, giving every terminal-agent run a compact scorecard across all seven dimensions—an extension the survey motivates but does not implement.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this survey actually earns its framework. The terminal-as-execution-substrate lens and seven-dimensional competence profile give terminal-agent work a usable vocabulary, and the P1–P7 trace indicators are a real attempt to make process observable. The empirical diagnostics are bounded, honestly hedged, and the central claim—that terminal-agent behavior is jointly shaped by model, interface, harness, runtime, and environment, so benchmark scores should not be read as pure model capability—holds up.

What is new: prior surveys organize around general LLM agents, software-engineering agents, or GUI computer use. This one centers terminal-mediated execution as the substrate and adds workload-level boundary tests. The seven dimensions and P1–P7 indicators are not in the cited surveys. The matched fixed-condition runs on Claw-SWE-Bench Lite and SWE-bench Lite are a small but original empirical contribution; the paired McNemar, Cochran's Q, and bootstrap intervals are appropriate for single-run paired data. The corpus protocol is documented with reproducible query templates and screening rules, which is more than most surveys do.

Soft spots, in proportion: the main one is Appendix C.3. P2, P4, and P6 are judged by DeepSeek-V4-Flash, the same model family as the evaluated agent in the benchmark-exposure diagnostic. That is a genuine same-family-judge risk, and it is amplified by the confidence filter (LongCLI-Bench loses 28.2% of labels as uncertain) and by the absence of released traces, judge outputs, and diagnostic code. I would not treat P2/P4/P6 as solid evidence until those are public. But the central synthesis does not rest on them. P1/P3/P5/P7 are deterministic, Table 3's observability coding is independent of the judge, and the cited harness studies support the same joint-shaping conclusion. The concern is real but auxiliary.

One run per task cell is acceptable for an illustrative diagnostic; the paper calls the results bounded and directional and the paired tests respect the single-run structure. Missing traces are the larger reproducibility issue.

Who the paper is for: anyone building or evaluating terminal agents, especially SWE-agent-style harnesses and process-level evaluation. It deserves a serious referee. I would send it to review with a request that the authors release the diagnostic artifacts or downgrade the judge-assisted indicators.

Referee Report

2 major / 6 minor

Summary. This survey organizes the study of LLM-based agents that operate through command-line terminals. It proposes a substrate-centered definition of terminal agents, introduces a seven-dimensional terminal competence profile, and uses that profile to structure a synthesis of the literature on system architecture, competence acquisition, and evaluation. The paper then reports two bounded empirical diagnostics: a benchmark-exposure study that fixes one agent model and measures seven trace-derived process indicators across four benchmark families, and a matched-system comparison of three agents on two SWE benchmarks with two model variants. The central conclusions are that terminal-agent behavior is jointly shaped by the model, interface, harness, runtime, and environment, and that prevailing evaluations emphasize final outcomes while exposing process quality unevenly.

Significance. The survey fills a genuine gap: terminal-mediated agent behavior is currently scattered across software-engineering, tool-use, and computer-use literatures, and the proposed seven-dimension competence profile provides a useful common vocabulary. The corpus construction in Appendix A, with its explicit screening questions, query families, and claim-level calibration, is a strength that makes the synthesis traceable. The qualitative observability matrix in Table 3 and the matched paired statistical tests (exact McNemar, Cochran's Q, bootstrap intervals) are also appropriately careful. The main weakness is that the auxiliary process indicators P2/P4/P6 are produced by the same model family that acts as the agent in the benchmark-exposure diagnostic, with no released traces or independent judgment channel; this limits the quantitative process profile but does not undermine the central survey synthesis, which is independently supported by the literature review, the deterministic indicators P1/P3/P5/P7, and the qualitative coding.

major comments (2)

[Appendix C.3 / Section 6.2] Appendix C.3 states that all formal P2, P4, and P6 labels were generated by DeepSeek-V4-Flash, which is also the agent model in the benchmark-exposure diagnostic (Section 6.2, Table 4). Because P4 (consequential state-error rate) and P6 (successful recovery rate) are used to support the claim that benchmark families foreground different process limitations (e.g., BashArena's higher P4 and P6), the same-family judge creates a concrete risk of systematic bias in the cross-benchmark process profile. The targeted human audit described in C.3 is qualitative and explicitly kept separate from the reported rates, and no traces, episode windows, or judge outputs are released for independent inspection. I request that the authors either (a) provide an independent judgment channel, such as a different model or human labels with inter-annotator agreement statistics on a sample of episodes, (b) release the traces and judge outputs, or (c) explicitly demote P2/P4/P6 from the benchmark-comparison findings and base the 'foregrounding' conclusions on the deterministic indicators P1/P3/P5/P7 and the qualitative coding in Table 3.
[Section 6.1 / Appendix C.2, C.4] The semantic indicators P2/P4/P6 are macro-averaged over eligible episodes after excluding low-confidence and uncertain labels (confidence threshold 0.7, Eq. (2)), with episode-level uncertainty reaching 28.2% for LongCLI-Bench and around 9–11% for the other benchmarks. The paper notes that LongCLI-Bench should be interpreted directionally, but it does not quantify how the exclusions could bias the reported rates for any benchmark. A sensitivity analysis varying the confidence threshold, or a worst-case bounds analysis, would clarify whether the cross-benchmark ordering of P4/P6 is robust. This is a substantive but local issue for the empirical illustration, and it should be addressed before the quantitative process profile is used to support comparative claims.

minor comments (6)

[Figure 7] The y-axis labels such as 'mini-SWE-agent /uni00B7 Flash' appear to contain a Unicode escape artifact ('/uni00B7'); please fix the rendering so the middle dot is displayed correctly.
[Table 3] The column header 'Fresh.' is clipped; consider renaming it to 'Freshness' or 'Eval. freshness' for clarity.
[Section 5.6] The sentence 'SWE-rebench reports evidence consistent with contamination-related inflation on static tasks' should include the reference citation [9] at the end of the sentence.
[Section 6.2] The observation that 'P1 rounds to 0.0% across all four benchmarks' is striking; please report the raw numerators and denominators (e.g., 3/10,000 normalized command events) so readers can judge the sparsity of this signal.
[Appendix C.4] The prose introducing coverage and uncertainty rates would benefit from an explicit definition of 'coverage' and 'uncertainty' as episode-level rates, rather than relying only on the parenthetical in the following sentence.
[Appendix C.1] The sentence 'Because different harnesses expose different action schemas, cross-system process indicators are treated as interface-sensitive trace evidence rather than directly interchangeable action units' is important for interpreting the diagnostics; consider moving it to the main text near Section 6.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the survey's framework and diagnostics are self-contained; the same-model judge is an auxiliary validity limitation, not a circular step.

full rationale

The claimed derivation chain was checked at each load-bearing point. The terminal-agent definition and workload-level boundary tests are used only to select and organize the literature; they do not by themselves imply the empirical conclusions. The seven-dimension profile is presented as an explicit organizing taxonomy, and the Table 3 observability matrix is a qualitative coding ('The coding indicates which constructs each group makes observable rather than the strength of their measurement'), not a quantity derived from the profile. The central joint-shaping and uneven-process-observability claims rest on external controlled studies (Meta-Harness, AutoHarness, Agent Psychometrics, SWE-agent/Agentless comparisons, etc.) and on deterministic trace indicators P1/P3/P5/P7, none of which are fitted to the conclusions. The matched-system diagnostic uses benchmark-native evaluators and standard paired tests, and its results are reported as snapshots rather than as predictions forced by the framework. The only notable soft spot is Appendix C.3, where the auxiliary judge for P2/P4/P6 is DeepSeek-V4-Flash, the same model used as the agent in the benchmark-exposure diagnostic (Section 6). This is a real independence and reproducibility limitation, and the paper itself flags it by calling P2/P4/P6 'auxiliary process indicators' and keeping human adjudications separate from the reported rates. It is not a circular step under the paper's own accounting: no parameter is fitted and renamed as a prediction, and no equation or conclusion reduces to its input by construction. Self-citations to prior surveys ([146], [173]) are background references and are not load-bearing. Accordingly, no significant circularity is found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The paper introduces no fitted physical parameters. Its free parameters are hand-chosen thresholds, window sizes, and pattern lists that shape the diagnostic indicators. The axioms are the domain assumptions underlying the scope and competence taxonomy, plus the assumption that the same model can judge its own behavior reliably. The invented entities are analytical constructs (the seven dimensions and P1-P7 indicators) rather than physical entities.

free parameters (3)

Semantic judge confidence threshold = 0.7
Hand-chosen engineering filter for accepting P2/P4/P6 LLM-judge labels; not calibrated to a probability (Appendix C.2).
Final-window verification definition = final 5 actions or final 20% of trace, whichever larger
Arbitrary operationalization of P5; different windows would change measured verification rates (Section 6.1).
Semantic episode window sizes = P2: 2 preceding/4 following; P4: 8/2; P6: 2/8
Hand-set context windows for the LLM judge; no sensitivity analysis is reported (Appendix C.3).

assumptions (4)

domain assumption Terminal-mediated execution is the correct organizing lens for the surveyed systems, and the three workload-level boundary tests operationalize 'dominant progress-bearing loop'.
Section 2.2 defines the scope; if the boundary tests are too permissive or restrictive, the corpus and synthesis shift.
domain assumption The seven competence dimensions form a meaningful, nearly exhaustive grouping of terminal-agent responsibilities.
Section 2.3 groups responsibilities by object, evidence, and response; this is a modeling choice, not derived from data.
ad hoc to paper DeepSeek-V4-Flash's rule-constrained judgments with confidence at least 0.7 provide unbiased P2/P4/P6 labels.
Appendix C.3 uses the same model family as the evaluated agent; the targeted human audit is qualitative and kept separate from aggregated labels.
standard math Standard statistical tests (exact McNemar, Cochran's Q) apply even with one run per task cell.
Appendix C.6 and C.7; validity relies on task-level paired Bernoulli outcomes, which may be affected by nondeterminism across runs.

invented entities (2)

Seven-dimensional terminal competence profile
purpose: Analytical framework for comparing system responsibilities, learning signals, and observable evidence across terminal agents.
A taxonomy introduced by the paper; it has no external falsifiable prediction and is not independently measurable beyond the paper's coding.
Trace-derived process indicators P1-P7
purpose: Summarize process observability from execution traces, supporting benchmark-exposure and attribution diagnostics.
Hand-defined indicators with no external validation of construct validity beyond the paper's own operational definitions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Terminal Agents: A Survey of AI Agents in Command-Line Environments." pith.science (2026). https://pith.science/paper/PPDZIANW

@misc{pith2026260820485,
  author       = {Pith},
  title        = {Pith review of: Terminal Agents: A Survey of AI Agents in Command-Line Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPDZIANW}},
  note         = {Machine review of arXiv:2608.20485}
}

read the original abstract

Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-mediated execution as an organizing lens, this survey establishes workload-level boundaries and connects system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile. Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment. Executable trajectories ground learning in action consequences, verification, and recovery, whereas prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly. Bounded fixed-condition diagnostics illustrate two implications: benchmark families expose different process signals, and matched system comparisons reveal benchmark-dependent performance and limits of component attribution. These findings motivate explicit reporting of system and runtime conditions, supported by replayable traces and process-level evidence. The framework provides a unified basis for studying terminal-mediated agency across software engineering and emerging application domains.

Figures

Figures reproduced from arXiv: 2608.20485 by the authors.

**Figure 2.** Analytical storyline of the survey. The survey organizes the study of terminal agents around [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Operational scope of terminal agents, linking terminal-mediated interaction to workload [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Shifts in design emphasis toward terminal-mediated agency, from tool-augmented prompt [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Layered architecture of terminal-agent systems. The central interaction loop is shaped by [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Terminal competence acquisition ecosystem. Data sources, interaction traces, runtime and [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Matched-system outcomes for all 12 benchmark–system–model cells. Points show resolved [PITH_FULL_IMAGE:figures/full_fig_p048_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

197 extracted references · 136 canonical work pages

[1]

Riva: Leveraging llm agents for reliable configuration drift detection

Sami Abuzakuk, Lucas Crijns, Anne-Marie Kermarrec, Rafael Pires, and Martijn de Vos. Riva: Leveraging llm agents for reliable configuration drift detection. InProceedings of the Sixth European Workshop on Machine Learning and Systems, pages 499–509, 2026

2026
[2]

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, and Valentin Malykh. Swe-mera: A dynamic benchmark for agenticly evaluating large language models on software engineering tasks.arXiv preprint arXiv:2507.11059, 2025

work page Pith review arXiv 2025
[3]

Ai ides or autonomous agents? measuring the impact of coding agents on software development

Shyam Agarwal, Hao He, and Bogdan Vasilescu. Ai ides or autonomous agents? measuring the impact of coding agents on software development. InProceedings of the 23rd International Conference on Mining Software Repositories, pages 857–862, 2026

2026
[4]

Do agents dream of root shells? partial-credit evaluation of llm agents in capture the flag challenges

Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek, Roland Vízner, Arie van Deursen, and Maliheh Izadi. Do agents dream of root shells? partial-credit evaluation of llm agents in capture the flag challenges. InProceedings of the 3rd ACM International Conference on AI-Powered Software, pages 349–357, 2026. 23

2026
[5]

Agentharm: A benchmark for measuring harmfulness of llm agents

Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of llm agents. InInternational Conference on Learning Representations, volume 2025, pages 79185–79220, 2025

2025
[6]

Claude Code, 2025

Anthropic. Claude Code, 2025. URLhttps://code.claude.com/docs/en/overview. Agen- tic coding tool available through terminal and other development surfaces, with file editing, command execution, and development-tool integration

2025
[7]

SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing soft- ware engineering agents’ ability to bootstrap development environments.arXiv preprint arXiv:2507.09063, 2025

work page Pith review arXiv 2025
[8]

Ciber: A comprehensive benchmark for security evaluation of code interpreter agents.arXiv preprint arXiv:2602.19547, 2026

Lei Ba, Qinbin Li, and Songze Li. Ciber: A comprehensive benchmark for security evaluation of code interpreter agents.arXiv preprint arXiv:2602.19547, 2026

work page arXiv 2026

Show all 197 references

[9]

Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents.Advances in Neural Information Processing Systems, 38, 2026

Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering...

2026
[10]

Swe- rebench v2: Language-agnostic swe task collection at scale.arXiv preprint arXiv:2602.23866, 2026

Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, and Alexander Golubev. Swe- rebench v2: Language-agnostic swe task collection at scale.arXiv preprint arXiv:2602.23866, 2026

2026 arXiv
[11]

Terminal agents suffice for enterprise automation.arXiv preprint arXiv:2604.00073, 2026

Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, and Sai Rajeswar. Terminal agents suffice for enterprise automation.arXiv preprint arXiv:2604.00073, 2026

2026 arXiv
[12]

Agyn: A multi-agent system for team-based autonomous software engineering.arXiv preprint arXiv:2602.01465, 2026

Nikita Benkovich and Vitalii Valkov. Agyn: A multi-agent system for team-based autonomous software engineering.arXiv preprint arXiv:2602.01465, 2026

2026
[13]

Large language models for agentic netops and aiops: Architectures, evaluation, and safety.arXiv preprint arXiv:2605.12729, 2026

Muhammad Bilal, Jon Crowcroft, Ruizhi Wang, Xiaolong Xu, and Schahram Dustdar. Large language models for agentic netops and aiops: Architectures, evaluation, and safety.arXiv preprint arXiv:2605.12729, 2026

2026 arXiv
[14]

Understanding software engineering agents: A study of thought-action-result trajectories

Islem Bouzenia and Michael Pradel. Understanding software engineering agents: A study of thought-action-result trajectories. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 2846–2857. IEEE, 2025

2025
[15]

Building effective ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned.arXiv preprint arXiv:2603.05344, 2026

Nghi DQ Bui. Building effective ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned.arXiv preprint arXiv:2603.05344, 2026

2026
[16]

Understanding software engineering agents through the lens of traceability: An empirical study.arXiv preprint arXiv:2506.08311, 2025

Ira Ceka, Saurabh Pujar, Shyam Ramji, Luca Buratti, Gail Kaiser, and Baishakhi Ray. Understanding software engineering agents through the lens of traceability: An empirical study.arXiv preprint arXiv:2506.08311, 2025

2025 arXiv
[17]

When benchmarks leak: Inference-time decontamina- tion for llms

Jianzhe Chai, Yu Zhe, and Jun Sakuma. When benchmarks leak: Inference-time decontamina- tion for llms. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 44743–44760, 2026. 24

2026
[18]

Mle-bench: Evaluating machine learning agents on machine learning engineering

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. InInternational Conference on Learning Representa...

2025
[19]

Beyond binary correctness: Scaling evaluation of long-horizon agents on subjective enterprise tasks.arXiv preprint arXiv:2603.22744, 2026

Abhishek Chandwani and Ishan Gupta. Beyond binary correctness: Scaling evaluation of long-horizon agents on subjective enterprise tasks.arXiv preprint arXiv:2603.22744, 2026

2026 arXiv
[20]

Securevibebench: Benchmarking secure vibe coding of ai agents via reconstructing vulnerability-introducing scenarios

Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, et al. Securevibebench: Benchmarking secure vibe coding of ai agents via reconstructing vulnerability-introducing scenarios. InProceedings of the 64th Annua...

2026
[21]

Tua-bench: A benchmark for general-purpose terminal-use agents.arXiv preprint arXiv:2606.28480, 2026

Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji, Feiyan Zhou, Xiaohui Zhang, Fanny Yang, and Belinda Zeng. Tua-bench: A benchmark for general-purpose terminal-use agents.arXiv preprint arXiv:2606.28480, 2026

2026 arXiv
[22]

Lps-bench: Benchmarking safety awareness of computer-use agents in long-horizon planning under benign and adversarial scenarios.arXiv preprint arXiv:2602.03255, 2026

Tianyu Chen, Chujia Hu, Ge Gao, Dongrui Liu, Xia Hu, and Wenjie Wang. Lps-bench: Benchmarking safety awareness of computer-use agents in long-horizon planning under benign and adversarial scenarios.arXiv preprint arXiv:2602.03255, 2026

2026
[23]

Aiopslab: A holistic framework to evaluate ai agents for enabling autonomous clouds.Proceedings of Machine Learning and Systems, 7, 2025

Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. Aiopslab: A holistic framework to evaluate ai agents for enabling autonomous clouds.Proceedings of Machine Learning and Systems, 7, 2025

2025
[24]

Stratus: A multi-agent system for autonomous reliability engineering of modern clouds.Advances in Neural Information Processing Systems, 38:50119–50165, 2026

Yinfang Chen, Jiaqi Pan, Jackson Clark, Yiming Su, Noah Zheutlin, Bhavya Bhavya, Rohan R Arora, Yu Deng, Saurabh Jha, and Tianyin Xu. Stratus: A multi-agent system for autonomous reliability engineering of modern clouds.Advances in Neural Information Processing Systems, 38:501...

2026
[25]

Matching matters: A fair quality-efficiency benchmark for command-line agents, 2026

Han Chi, Jiaxin Qi, Yan Cui, Baisheng Lai, and Jianqiang Huang. Matching matters: A fair quality-efficiency benchmark for command-line agents, 2026. URLhttps://arxiv.org/abs/ 2606.21140

2026 arXiv
[26]

Terminalworld: Benchmarking agents on real-world terminal tasks.arXiv preprint arXiv:2605.22535, 2026

Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T Barr, Mark Harman, Federica Sarro, et al. Terminalworld: Benchmarking agents on real-world terminal tasks.arXiv preprint arXiv:2605.22535, 2026

2026 arXiv
[27]

Permission denied: Policy- graded evaluation of coding agents in hardened environments.arXiv preprint arXiv:2608.02670, 2026

Dotan Davidovich, Yair Amar, Hai Rozencwajg, and Or Hiltch. Permission denied: Policy- graded evaluation of coding agents in hardened environments.arXiv preprint arXiv:2608.02670, 2026

2026 arXiv
[28]

Airtbench: Measuringautonomous ai red teaming capabilities in language models.arXiv preprint arXiv:2506.14682, 2025

AdsDawson, RobMulla, NickLanders, andShaneCaldwell. Airtbench: Measuringautonomous ai red teaming capabilities in language models.arXiv preprint arXiv:2506.14682, 2025

2025 arXiv
[29]

Terminal is all you need: Design properties for human-ai agent collabora- tion.arXiv preprint arXiv:2603.10664, 2026

Alexandre De Masi. Terminal is all you need: Design properties for human-ai agent collabora- tion.arXiv preprint arXiv:2603.10664, 2026

2026
[30]

Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025

Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025. 25

2025 arXiv
[31]

Octobench: Benchmarking scaffold-aware instruction following in repository-grounded agentic coding.arXiv preprint arXiv:2601.10343, 2026

Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, et al. Octobench: Benchmarking scaffold-aware instruction following in repository-grounded agentic coding.arXiv preprint arXiv:2601.10343, 2026

2026
[32]

Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents.arXiv preprint arXiv:2512.12730, 2025

Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents.arXiv preprint arXiv:2512.12730, 2025

2025
[33]

Agenther: Hindsight experience replay for llm agent trajectory relabeling.arXiv preprint arXiv:2603.21357, 2026

Liang Ding. Agenther: Hindsight experience replay for llm agent trajectory relabeling.arXiv preprint arXiv:2603.21357, 2026

2026 arXiv
[34]

Wildclawbench: A benchmark for real-world, long-horizon agent evaluation.arXiv preprint arXiv:2605.10912, 2026

Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, et al. Wildclawbench: A benchmark for real-world, long-horizon agent evaluation.arXiv preprint arXiv:2605.10912, 2026

2026 arXiv
[35]

The blind spot of agent safety: How benign user instructions expose critical vulnerabilities in computer-use agents.arXiv preprint arXiv:2604.10577, 2026

Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li, Taiwei Shi, Nicholas Meade, Siva Reddy, Jian Kang, and Jieyu Zhao. The blind spot of agent safety: How benign user instructions expose critical vulnerabilities in computer-use agents.arXiv preprint arXiv:2604.10577, 2026

2026 arXiv
[36]

A survey on code generation with llm-based agents.arXiv preprint arXiv:2508.00083, 2025

Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. A survey on code generation with llm-based agents.arXiv preprint arXiv:2508.00083, 2025

2025 arXiv
[37]

Swe-dev: Evaluating and training autonomous feature-driven software development.arXiv preprint arXiv:2505.16975, 2025

Yaxin Du, Yuzhu Cai, Yifan Zhou, Cheng Wang, Yu Qian, Xianghe Pang, Qian Liu, Yue Hu, and Siheng Chen. Swe-dev: Evaluating and training autonomous feature-driven software development.arXiv preprint arXiv:2505.16975, 2025

2025
[38]

Where do ai coding agents fail? an empirical study of failed agentic pull requests in github

Ramtin Ehsani, Sakshi Pathak, Shriya Rawal, Abdullah Al Mujahid, Mia Mohammad Imran, and Preetha Chatterjee. Where do ai coding agents fail? an empirical study of failed agentic pull requests in github. InProceedings of the 23rd International Conference on Mining Software Repo...

2026
[39]

Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces

Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, et al. Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces. InFindings of the Association for Computa...

2026
[40]

Agenthazard: A benchmark for evaluating harmful behavior in computer-use agents.arXiv preprint arXiv:2604.02947, 2026

Yunhao Feng, Yifan Ding, Yingshui Tan, Xingjun Ma, Yige Li, Yutao Wu, Yifeng Gao, Kun Zhai, and Yanming Guo. Agenthazard: A benchmark for evaluating harmful behavior in computer-use agents.arXiv preprint arXiv:2604.02947, 2026

2026 arXiv
[41]

Camels can use computers too: System-level security for computer use agents.arXiv preprint arXiv:2601.09923, 2026

Hanna Foerster, Tom Blanchard, Kristina Nikolić, Ilia Shumailov, Cheng Zhang, Robert Mullins, Nicolas Papernot, Florian Tramèr, and Yiren Zhao. Camels can use computers too: System-level security for computer use agents.arXiv preprint arXiv:2601.09923, 2026

2026 arXiv
[42]

The scaffolding matters more than the interface: A controlled comparison of mcp and cli tool use across seven agent scaffoldings, five language models, and one software task,

Marc Alier Forment, María José Casañ Guerrero, Francisco José García-Peñalvo, and Juanan Pereira. The scaffolding matters more than the interface: A controlled comparison of mcp and cli tool use across seven agent scaffoldings, five language models, and one software task,
[43]

Endless terminals: Scaling rl environments for terminal agents.arXiv preprint arXiv:2601.16443, 2026

Kanishk Gandhi, Shivam Garg, Noah D Goodman, and Dimitris Papailiopoulos. Endless terminals: Scaling rl environments for terminal agents.arXiv preprint arXiv:2601.16443, 2026. 26

2026
[44]

Agentpulse: A continuous multi-signal framework for evaluating ai agents in deployment.arXiv preprint arXiv:2604.24038, 2026

Yuxuan Gao, Megan Wang, and Yi Ling Yu. Agentpulse: A continuous multi-signal framework for evaluating ai agents in deployment.arXiv preprint arXiv:2604.24038, 2026

2026 arXiv
[45]

Saving swe-bench: A benchmark mutation approach for realistic agent evaluation.arXiv preprint arXiv:2510.08996, 2025

Spandan Garg, Benjamin Steenhoek, and Yufan Huang. Saving swe-bench: A benchmark mutation approach for realistic agent evaluation.arXiv preprint arXiv:2510.08996, 2025

2025
[46]

Beyond state machines: Executing network procedures with agentic tool-calling sequences.arXiv preprint arXiv:2605.02584, 2026

Purna Sai Garigipati, Onur Ayan, Kishor Chandra Joshi, and Xueli An. Beyond state machines: Executing network procedures with agentic tool-calling sequences.arXiv preprint arXiv:2605.02584, 2026

2026 arXiv
[47]

Aider: Ai pair programming in your terminal, 2025

Paul Gauthier. Aider: Ai pair programming in your terminal, 2025. URLhttps://github. com/Aider-AI/aider. Open-source terminal-native AI pair-programming tool for editing and managing codebases with LLMs

2025
[48]

Agent psychometrics: Task-level performance prediction in agentic coding benchmarks.arXiv preprint arXiv:2604.00594, 2026

Chris Ge, Daria Kryvosheieva, Daniel Fried, Uzay Girit, and Kaivalya Hariharan. Agent psychometrics: Task-level performance prediction in agentic coding benchmarks.arXiv preprint arXiv:2604.00594, 2026

2026 arXiv
[49]

Effective strategies for asynchronous software engineering agents.arXiv preprint arXiv:2603.21489, 2026

Jiayi Geng and Graham Neubig. Effective strategies for asynchronous software engineering agents.arXiv preprint arXiv:2603.21489, 2026

2026 arXiv
[50]

Training long-context, multi-turn software engineering agents with reinforcement learning.arXiv preprint arXiv:2508.03501, 2025

Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekra- shevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, et al. Training long-context, multi-turn software engineering agents with reinforcement learnin...

2025
[51]

Secure and efficient access control for computer-use agents via context space.arXiv preprint arXiv:2509.22256, 2025

Haochen Gong, Chenxiao Li, Rui Chang, and Wenbo Shen. Secure and efficient access control for computer-use agents via context space.arXiv preprint arXiv:2509.22256, 2025

2025
[52]

Gemini CLI, 2025

Google. Gemini CLI, 2025. URLhttps://github.com/google-gemini/gemini-cli. Open- source terminal AI agent for Gemini models with file operations, shell commands, web tools, and MCP integration

2025
[53]

Redcode: Risky code execution and generation benchmark for code agents.Advances in Neural Information Processing Systems, 37:106190–106236, 2024

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents.Advances in Neural Information Processing Systems, 37:106190–106236, 2024

2024
[54]

Agenteval: Dag-structured step-level evaluation for agentic workflows with error propagation tracking.arXiv preprint arXiv:2604.23581, 2026

Dongxin Guo, Jikun Wu, and Siu Ming Yiu. Agenteval: Dag-structured step-level evaluation for agentic workflows with error propagation tracking.arXiv preprint arXiv:2604.23581, 2026

2026 arXiv
[55]

Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks.arXiv preprint arXiv:2506.10954, 2025

Lianghong Guo, Yanlin Wang, Caihua Li, Wei Tao, Pengyu Yang, Jiachi Chen, Haoyu Song, Duyu Tang, and Zibin Zheng. Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks.arXiv preprint arXiv:2506.10954, 2025

2025
[56]

Swe-skills-bench: Do agent skills actually help in real-world software engineering?arXiv preprint arXiv:2603.15401, 2026

Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. Swe-skills-bench: Do agent skills actually help in real-world software engineering?arXiv preprint arXiv:2603.15401, 2026

2026
[57]

Procbench: Evaluating process-level defects and control preservation in llm coding agents

Jiawei He, Jie Jia, Chenbo Liu, Chaoyi Xue, Yapeng Song, Xikai Yang, and Dong Sun. Procbench: Evaluating process-level defects and control preservation in llm coding agents. arXiv preprint arXiv:2605.20251, 2026. 27

2026 arXiv
[58]

Evaluating llm-based 0-to-1 software generation in end-to-end cli tool scenarios.arXiv preprint arXiv:2604.06742, 2026

Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, and David Lo. Evaluating llm-based 0-to-1 software generation in end-to-end cli tool scenarios.arXiv preprint arXiv:2604.06742, 2026

2026 arXiv
[59]

Os agents: A survey on mllm-based agents for general computing devices use.arXiv preprint arXiv:2508.04482, 2025

Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use.arXiv preprint arXiv:2508.04482, 2025

2025 arXiv
[60]

Cli-universe: Towards verifiable task synthesis engine for terminal agents.arXiv preprint arXiv:2606.22883, 2026

Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, et al. Cli-universe: Towards verifiable task synthesis engine for terminal agents.arXiv preprint arXiv:2606.22883, 2026

2026 arXiv
[61]

Agentstepper: Interactive debugging of software develop- ment agents.arXiv preprint arXiv:2602.06593, 2026

Robert Hutter and Michael Pradel. Agentstepper: Interactive debugging of software develop- ment agents.arXiv preprint arXiv:2602.06593, 2026

2026
[62]

Tmax: A simple recipe for terminal agents.arXiv preprint arXiv:2606.23321, 2026

Hamish Ivison, Junjie Oscar Yin, Rulin Shao, Teng Xiao, Nathan Lambert, and Hannaneh Hajishirzi. Tmax: A simple recipe for terminal agents.arXiv preprint arXiv:2606.23321, 2026

2026 arXiv
[63]

R2e-gym: Procedural environment generation and hybrid verifiers for scaling open-weights swe agents

Naman Jain, Jaskirat Singh, Manish Shetty, Tianjun Zhang, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environment generation and hybrid verifiers for scaling open-weights swe agents. InSecond Conference on Language Modeling, 2025

2025
[64]

Itbench: Evaluating ai agents across diverse real-world it automation tasks.arXiv preprint arXiv:2502.05352, 2025

Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, et al. Itbench: Evaluating ai agents across diverse real-world it automation tasks.arXiv preprint arXiv:2502.05352, 2025

2025 arXiv
[65]

Reap: Automatic curation of coding agent benchmarks from interactive production usage.arXiv preprint arXiv:2604.01527, 2026

Smriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, and Satish Chandra. Reap: Automatic curation of coding agent benchmarks from interactive production usage.arXiv preprint arXiv:2604.01527, 2026

2026 arXiv
[66]

Coding agents are guessing: Measuring action-boundary violations in underspecified devops instructions.arXiv preprint arXiv:2607.02294, 2026

Zimo Ji, Zekai Zhang, Congying Xu, Zongjie Li, Yudong Gao, Shuai Wang, and Shing-Chi Cheung. Coding agents are guessing: Measuring action-boundary violations in underspecified devops instructions.arXiv preprint arXiv:2607.02294, 2026

2026 arXiv
[67]

Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

2024
[68]

Elt-bench: An end-to-end benchmark for evaluating ai agents on elt pipelines.Proceedings of the VLDB Endowment, 19(2):84–98, 2025

Tengjun Jin, Yuxuan Zhu, and Daniel Kang. Elt-bench: An end-to-end benchmark for evaluating ai agents on elt pipelines.Proceedings of the VLDB Endowment, 19(2):84–98, 2025

2025
[69]

Trace: Capability- targeted agentic training.arXiv preprint arXiv:2604.05336, 2026

Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, and Azalia Mirhoseini. Trace: Capability- targeted agentic training.arXiv preprint arXiv:2604.05336, 2026

2026 arXiv
[70]

Basharena: A control setting for highly privileged ai agents.arXiv preprint arXiv:2512.15688, 2025

Adam Kaufman, James Lucassen, Tyler Tracy, Cody Rushing, and Aryan Bhatt. Basharena: A control setting for highly privileged ai agents.arXiv preprint arXiv:2512.15688, 2025

2025
[71]

From reproduc- tion to replication: Evaluating research agents with progressive code masking.arXiv preprint arXiv:2506.19724, 2025

Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduc- tion to replication: Evaluating research agents with progressive code masking.arXiv preprint arXiv:2506.19724, 2025. 28

2025 arXiv
[72]

Curie: Towardrigorous and automated scientific experimentation with ai agents.arXiv preprint arXiv:2502.16069, 2025

Patrick Tser Jern Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang, Yibo Huang, JayanthSrinivasa, MyungjinLee, MosharafChowdhury, andAngChen. Curie: Towardrigorous and automated scientific experimentation with ai agents.arXiv preprint arXiv:2502.16069, 2025

2025 arXiv
[73]

Exp-bench: Can ai conduct ai research experiments?arXiv preprint arXiv:2505.24785, 2025

Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, et al. Exp-bench: Can ai conduct ai research experiments?arXiv preprint arXiv:2505.24785, 2025

2025 arXiv
[74]

Process-level trajectory evaluation for environment configuration in software engineering agents

Jiayi Kuang, Yinghui Li, Xin Zhang, Yangning Li, Xing Sun, Ying Shen, Philip Yu, et al. Process-level trajectory evaluation for environment configuration in software engineering agents. InInternational Conference on Learning Representations, volume 2026, pages 113832–113855, 2026

2026
[75]

Os-harm: A benchmark for measuring safety of computer use agents.Advances in Neural Information Processing Systems, 38, 2026

Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. Os-harm: A benchmark for measuring safety of computer use agents.Advances in Neural Information Processing Systems, 38, 2026

2026
[76]

Clawforge: Generating executable interactive benchmarks for command-line agents.arXiv preprint arXiv:2605.14133, 2026

Yuxiang Lai, Peng Xia, Haonian Ji, Kaiwen Xiong, Kaide Zeng, Jiaqi Liu, Fang Wu, Jike Zhong, Zeyu Zheng, Cihang Xie, et al. Clawforge: Generating executable interactive benchmarks for command-line agents.arXiv preprint arXiv:2605.14133, 2026

2026 arXiv
[77]

On the impacts of contexts on repository- level code generation

Nam Le Hai, Dung Manh Nguyen, and Nghi DQ Bui. On the impacts of contexts on repository- level code generation. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 1496–1524, 2025

2025
[78]

Ctfusion: A ctf-based benchmark for llm agent evaluation.arXiv preprint arXiv:2605.11504, 2026

Dongjun Lee, Ga-eun Bae, and Insu Yun. Ctfusion: A ctf-based benchmark for llm agent evaluation.arXiv preprint arXiv:2605.11504, 2026

2026 arXiv
[79]

Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

2026 arXiv
[80]

Dacomp: Benchmarking data agents across the full data intelligence lifecycle.arXiv preprint arXiv:2512.04324, 2025

Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, Yitong Zhang, Jianwen Luo, Xin Zou, Ruiyi Yang, Wenbo Shi, Yan Gao, et al. Dacomp: Benchmarking data agents across the full data intelligence lifecycle.arXiv preprint arXiv:2512.04324, 2025

2025
[81]

Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts

Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, et al. Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts. InProceedings of the 64th Annual Meeting of the Association for C...

2026
[82]

Clawsbench: Evaluating capability and safety of llm productivity agents in simulated workspaces.arXiv preprint arXiv:2604.05172, 2026

Xiangyi Li, Kyoung Whan Choe, Yimin Liu, Xiaokun Chen, Chujun Tao, Bingran You, Wenbo Chen, Zonglin Di, Jiankai Sun, Shenghan Zheng, et al. Clawsbench: Evaluating capability and safety of llm productivity agents in simulated workspaces.arXiv preprint arXiv:2604.05172, 2026

2026 arXiv
[83]

Repomod-bench: A benchmark for code repository modernization via implementation-agnostic testing.arXiv preprint arXiv:2602.22518, 2026

Xuefeng Li, Nir Ben-Israel, Yotam Raz, Belal Ahmed, Doron Serebro, and Antoine Raux. Repomod-bench: A benchmark for code repository modernization via implementation-agnostic testing.arXiv preprint arXiv:2602.22518, 2026. 29

2026
[84]

Recursive synthesis for long-horizon terminal tasks

Zhongzhi Li, Yucheng Shi, Zongxia Li, Ruhan Wang, Anhao Li, Zixun Huang, Junyao Yang, Lei Ke, Ninghao Liu, Haitao Mi, et al. Recursive synthesis for long-horizon terminal tasks. arXiv preprint arXiv:2608.05466, 2026

2026 arXiv
[85]

Long-horizon-terminal-bench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading.arXiv preprint arXiv:2607.08964, 2026

Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, et al. Long-horizon-terminal-bench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading.arXiv preprint arXiv:2607.08964, 2026

2026 arXiv
[86]

Swe-next: Scalable real-world software engineering tasks for agents.arXiv preprint arXiv:2603.20691, 2026

Jiarong Liang, Zhiheng Lyu, Zijie Liu, Xiangchao Chen, Ping Nie, Kai Zou, and Wenhu Chen. Swe-next: Scalable real-world software engineering tasks for agents.arXiv preprint arXiv:2603.20691, 2026

2026
[87]

Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Xuanjing Huang, Hang Yan, Zhenhua Han, and Tao Gui. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

2026 arXiv
[88]

Ircopilot: Automated incident response with large language models.arXiv preprint arXiv:2505.20945, 2025

Xihuan Lin, Jie Zhang, Gelei Deng, Tianzhe Liu, Tianwei Zhang, Qing Guo, and Riqing Chen. Ircopilot: Automated incident response with large language models.arXiv preprint arXiv:2505.20945, 2025

2025
[89]

Cli-gym: Scalable cli task generation via agentic environment inversion.arXiv preprint arXiv:2602.10999, 2026

Yusong Lin, Haiyang Wang, Shuzhe Wu, Lue Fan, Feiyang Pan, Sanyuan Zhao, and Dandan Tu. Cli-gym: Scalable cli task generation via agentic environment inversion.arXiv preprint arXiv:2602.10999, 2026

2026
[90]

Agentbench: Evaluating llms as agents

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representations, volume 2024, pages 52989–53046, 2024

2024
[91]

A scalable benchmark for repository- oriented long-horizon conversational context management.arXiv preprint arXiv:2603.06358, 2026

Yang Liu, Li Zhang, Fang Liu, Ping Lin, and Xinyi Li. A scalable benchmark for repository- oriented long-horizon conversational context management.arXiv preprint arXiv:2603.06358, 2026

2026
[92]

Pacebench: A framework for evaluating practical ai cyber-exploitation capabilities.arXiv preprint arXiv:2510.11688, 2025

Zicheng Liu, Lige Huang, Jie Zhang, Dongrui Liu, Yuan Tian, and Jing Shao. Pacebench: A framework for evaluating practical ai cyber-exploitation capabilities.arXiv preprint arXiv:2510.11688, 2025

2025
[93]

Autoharness: improving llm agents by automatically synthesizing a code harness.arXiv preprint arXiv:2603.03329, 2026

Xinghua Lou, Miguel Lázaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P Murphy. Autoharness: improving llm agents by automatically synthesizing a code harness.arXiv preprint arXiv:2603.03329, 2026

2026
[94]

Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. InFindings of the Association for Computat...

2025
[95]

Projdevbench: Benchmarking ai coding agents on end-to-end project development.arXiv preprint arXiv:2602.01655, 2026

Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, et al. Projdevbench: Benchmarking ai coding agents on end-to-end project development.arXiv preprint arXiv:2602.01655, 2026

2026
[96]

Quantifying frontier llm capabilities for container sandbox escape.arXiv preprint arXiv:2603.02277, 2026

Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, and Harry Coppock. Quantifying frontier llm capabilities for container sandbox escape.arXiv preprint arXiv:2603.02277, 2026. 30

2026 arXiv
[97]

Ide-bench: Evaluating large language models as ide agents on real-world software engineering tasks.arXiv preprint arXiv:2601.20886, 2026

Spencer Mateega, Jeff Yang, Tiana Costello, Shaurya Jadhav, Nicole Tian, and Agustin Garcinuño. Ide-bench: Evaluating large language models as ide agents on real-world software engineering tasks.arXiv preprint arXiv:2601.20886, 2026

2026
[98]

Beyond resolution rates: Behavioral drivers of coding agent success and failure.arXiv preprint arXiv:2604.02547, 2026

Tural Mehtiyev and Wesley Assunção. Beyond resolution rates: Behavioral drivers of coding agent success and failure.arXiv preprint arXiv:2604.02547, 2026

2026 arXiv
[99]

Aios: Llm agent operating system.arXiv preprint arXiv:2403.16971, 2024

Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Aios: Llm agent operating system.arXiv preprint arXiv:2403.16971, 2024

2024 arXiv
[100]

Calibforge: Adversarial solver calibration for scaling learnable terminal tasks.arXiv preprint arXiv:2608.06352, 2026

Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, and Kai Jia. Calibforge: Adversarial solver calibration for scaling learnable terminal tasks.arXiv preprint arXiv:2608.06352, 2026

2026 arXiv
[101]

Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

2026 arXiv
[102]

How helpful is llm assistance in network operations? a case study at a large demonstration network.arXiv preprint arXiv:2605.19627, 2026

Ryo Nakamura and Koshi Eguchi. How helpful is llm assistance in network operations? a case study at a large demonstration network.arXiv preprint arXiv:2605.19627, 2026

2026 arXiv
[103]

Mle-star: Machine learning engineering agent via search and targeted refinement.Advances in Neural Information Processing Systems, 38:116692–116712, 2026

Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Arik, and Tomas Pfister. Mle-star: Machine learning engineering agent via search and targeted refinement.Advances in Neural Information Processing Systems, 38:116692–116712, 2026

2026
[104]

Iso-bench: Can coding agents optimize real-world inference workloads?arXiv preprint arXiv:2602.19594, 2026

Ayush Nangia, Shikhar Mishra, Aman Gokrani, and Paras Chopra. Iso-bench: Can coding agents optimize real-world inference workloads?arXiv preprint arXiv:2602.19594, 2026

2026
[105]

Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging

Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Zhenheng Tang, Sen Hu, Bo Li, Chen Hu, Binxing Jiao, et al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. InProceedings of the AAAI Conference on Artificial Intellige...

2026
[106]

Codex CLI, 2025

OpenAI. Codex CLI, 2025. URLhttps://github.com/openai/codex. Open-source terminal coding agent that runs locally and supports command-line coding workflows

2025
[107]

Slopcodebench: Benchmarking how coding agents degrade over long-horizon iterative tasks.arXiv preprint arXiv:2603.24755, 2026

Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, and Aws Albarghouthi. Slopcodebench: Benchmarking how coding agents degrade over long-horizon iterative tasks.arXiv preprint arXiv:2603.24755, 2026

2026 arXiv
[108]

Semagent: A semantics aware program repair agent.arXiv preprint arXiv:2506.16650, 2025

Anvith Pabba, Alex Mathai, Anindya Chakraborty, and Baishakhi Ray. Semagent: A semantics aware program repair agent.arXiv preprint arXiv:2506.16650, 2025

2025 arXiv
[109]

Ml-dev-bench: Comparative analysis of ai agents on ml development workflows.arXiv preprint arXiv:2502.00964, 2025

Harshith Padigela, Chintan Shah, and Dinkar Juyal. Ml-dev-bench: Comparative analysis of ai agents on ml development workflows.arXiv preprint arXiv:2502.00964, 2025

2025 arXiv
[110]

Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024. 31

2024 arXiv
[111]

Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

2022 arXiv
[112]

Engineering llm powered multi-agent framework for autonomous cloudops

Kannan Parthasarathy, Karthik Vaidhyanathan, Rudra Dhar, Venkat Krishnamachari, Adyansh Kakran, Sreemaee Akshathala, Shrikara Arun, Amey Karan, Basil Muhammed, Sumant Dubey, et al. Engineering llm powered multi-agent framework for autonomous cloudops. In2025 IEEE/ACM 4th Inter...

2025
[113]

Hyperagent: Generalist software engineering agents to solve coding tasks at scale.arXiv preprint arXiv:2409.16299, 2024

Huy Nhat Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist software engineering agents to solve coding tasks at scale.arXiv preprint arXiv:2409.16299, 2024

2024 arXiv
[114]

On data engineering for scaling llm terminal capabilities.arXiv preprint arXiv:2602.21193, 2026

Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, and Wei Ping. On data engineering for scaling llm terminal capabilities.arXiv preprint arXiv:2602.21193, 2026

2026
[115]

Comparing ai coding agents: A task-stratified analysis of pull request acceptance

Giovanni Pinna, Jingzhi Gong, David Williams, and Federica Sarro. Comparing ai coding agents: A task-stratified analysis of pull request acceptance. InProceedings of the 23rd International Conference on Mining Software Repositories, pages 792–796, 2026

2026
[116]

Mle-dojo: Interactive environments for empowering llm agents in machine learning engineering.Advances in Neural Information Processing Systems, 38, 2026

Rushi Qiang, Yuchen Zhuang, Yinghao Li, Dingu Sagar VK, Rongzhi Zhang, Changhao Li, Ian Wong, Sherry Yang, Percy Liang, Chao Zhang, et al. Mle-dojo: Interactive environments for empowering llm agents in machine learning engineering.Advances in Neural Information Processing Sys...

2026
[117]

On the robustness of agentic function calling

Ella Rabinovich and Ateret Anaby Tavor. On the robustness of agentic function calling. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 298–304, 2025

2025
[118]

Dsagentbench: Can agents automate end-to-end data-science workflows in real computer environments?, 2026

Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, and Enamul Hoque Prince. Dsagentbench: Can agents automate end-to-end data-science workflows in real computer environments?, 2026. URLhttps://arxiv.org/ abs/2608.10366

2026 arXiv
[119]

Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint arXiv:2504.08703, 2025

Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents.arXiv pr...

2025 arXiv
[120]

Androidworld: A dynamic benchmarking environment for autonomous agents

Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. InInternational Conference on Learning Repre...

2025
[121]

A self-evolving framework for efficient terminal agents via observational context compression.arXiv preprint arXiv:2604.19572, 2026

Jincheng Ren, Siwei Wu, Yizhi Li, Kang Zhu, Shu Xu, Boyu Feng, Ruibin Yuan, Wei Zhang, Riza Batista-Navarro, Jian Yang, et al. A self-evolving framework for efficient terminal agents via observational context compression.arXiv preprint arXiv:2604.19572, 2026

2026 arXiv
[122]

Agen- tic much? adoption of coding agents on github.ACM Transactions on Software Engineering and Methodology, 2026

Romain Robbes, Théo Matricon, Thomas Degueule, Andre Hora, and Stefano Zacchiroli. Agen- tic much? adoption of coding agents on github.ACM Transactions on Software Engineering and Methodology, 2026. 32

2026
[123]

Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36: 68539–68551, 2023

2023
[124]

Kubeintellect: A modular llm- orchestrated agent framework for end-to-end kubernetes management: Ms ardebili, a

Mohsen Seyedkazemi Ardebili and Andrea Bartolini. Kubeintellect: A modular llm- orchestrated agent framework for end-to-end kubernetes management: Ms ardebili, a. bartolini. Journal of Grid Computing, 24(3):17, 2026

2026
[125]

Agentrm: An os-inspired resource manager for llm agent systems.arXiv preprint arXiv:2603.13110, 2026

Jianshu She. Agentrm: An os-inspired resource manager for llm agent systems.arXiv preprint arXiv:2603.13110, 2026

2026
[126]

Seta: Scaling environments for terminal agents.arXiv preprint arXiv:2607.10891, 2026

Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Aznaur Aliev, Jay Rainton, Ahmed Awelkair, Zhichen Zeng, Jiajun Li, Shi Dong, Yueming Yuan, et al. Seta: Scaling environments for terminal agents.arXiv preprint arXiv:2607.10891, 2026

2026 arXiv
[127]

Towards trustworthy gui agents: A survey.arXiv preprint arXiv:2503.23434, 2025

Yucheng Shi, Wenhao Yu, Jingyuan Huang, Wenlin Yao, Wenhu Chen, and Ninghao Liu. Towards trustworthy gui agents: A survey.arXiv preprint arXiv:2503.23434, 2025

2025
[128]

Echo: Terminal agents learn world models for free.arXiv preprint arXiv:2605.24517, 2026

Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, and Dimitris Papailiopoulos. Echo: Terminal agents learn world models for free.arXiv preprint arXiv:2605.24517, 2026

2026 arXiv
[129]

Security in the age of ai teammates: An empirical study of agentic pull requests on github.arXiv preprint arXiv:2601.00477, 2026

Mohammed Latif Siddiq, Xinye Zhao, Vinicius Carvalho Lopes, Beatrice Casey, and Joanna Santos. Security in the age of ai teammates: An empirical study of agentic pull requests on github.arXiv preprint arXiv:2601.00477, 2026

2026
[130]

kraig: A natural language-driven agent for automated dataops pipeline generation.arXiv preprint arXiv:2603.20311, 2026

Rohan Siva, Kai Cheung, Lichi Li, and Ganesh Sundaram. kraig: A natural language-driven agent for automated dataops pipeline generation.arXiv preprint arXiv:2603.20311, 2026

2026
[131]

Swe-master: Unleashing the potential of software engineering agents via post-training.arXiv preprint arXiv:2602.03411, 2026

Huatong Song, Lisheng Huang, Shuang Sun, Jinhao Jiang, Ran Le, Daixuan Cheng, Guoxin Chen, Yiwen Hu, Zongchao Chen, Yiming Jia, et al. Swe-master: Unleashing the potential of software engineering agents via post-training.arXiv preprint arXiv:2602.03411, 2026

2026
[132]

Anchor: Automated alignment auditing for cli agents on real-world harm.arXiv preprint arXiv:2607.10455, 2026

Kefan Song and Yanjun Qi. Anchor: Automated alignment auditing for cli agents on real-world harm.arXiv preprint arXiv:2607.10455, 2026

2026 arXiv
[133]

Cross-context verification: Hierarchical detection of benchmark contamination through session-isolated analysis.arXiv preprint arXiv:2603.21454, 2026

Tae-Eun Song. Cross-context verification: Hierarchical detection of benchmark contamination through session-isolated analysis.arXiv preprint arXiv:2603.21454, 2026

2026
[134]

Atime- consistent benchmark for repository-level software engineering evaluation.arXiv preprint arXiv:2603.26137, 2026

Haonan Sun, Tian Yu, Sheng Ma, Qincheng Zhang, Lifei Rao, Chen Tian, et al. Atime- consistent benchmark for repository-level software engineering evaluation.arXiv preprint arXiv:2603.26137, 2026

2026
[135]

Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows

Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. InInternational Conference on Learning Representati...

2026
[136]

Scaling long-horizon llm agent via context-folding.arXiv preprint arXiv:2510.11967, 2025

Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. Scaling long-horizon llm agent via context-folding.arXiv preprint arXiv:2510.11967, 2025. 33

2025
[137]

Hongyuan Tao, Ying Zhang, Zhenhao Tang, Hongen Peng, Xukun Zhu, Bingchang Liu, Yingguang Yang, Ziyin Zhang, Zhaogui Xu, Haipeng Zhang, et al. Code graph model (cgm): A graph-integrated large language model for repository-level software engineering tasks.Advances in Neural Info...

2026
[138]

Livesqlbench: A dynamic and contamination-free benchmark for evaluating llms on real-world text-to-sql tasks, 2024

BIRD Team et al. Livesqlbench: A dynamic and contamination-free benchmark for evaluating llms on real-world text-to-sql tasks, 2024

2024
[139]

Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios.arXiv preprint arXiv:2512.18470, 2025

Minh VT Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi DQ Bui. Swe-evo: Benchmarking coding agents in long-horizon software evolution scenarios.arXiv preprint arXiv:2512.18470, 2025

2025 arXiv
[140]

Dcas: Decoupling cli agent scaffolding to internalize planning across scaffolds.arXiv preprint arXiv:2608.06113, 2026

Kishanthan Thangarajah, Boyuan Chen, and Ahmed E Hassan. Dcas: Decoupling cli agent scaffolding to internalize planning across scaffolds.arXiv preprint arXiv:2608.06113, 2026

2026 arXiv
[141]

ShellGPT, 2026

TheR1D. ShellGPT, 2026. URLhttps://github.com/TheR1D/shell_gpt. Command-line productivity tool powered by large language models for generating shell commands, code snippets, and documentation

2026
[142]

The dual-state architecture for reliable llm agents, 2026

Matthew Thompson. The dual-state architecture for reliable llm agents, 2026. URLhttps: //arxiv.org/abs/2512.20660

2026
[143]

Appworld: A controllable world of apps and people for benchmarking interactive coding agents

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annual Mee...

2024
[144]

Netagentbench: A state-centric benchmark for evaluating agentic network configuration.arXiv preprint arXiv:2604.09678, 2026

Ahmed Twabi, Yepeng Ding, and Tohru Kondo. Netagentbench: A state-centric benchmark for evaluating agentic network configuration.arXiv preprint arXiv:2604.09678, 2026

2026 arXiv
[145]

The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation.arXiv preprint arXiv:2607.22585, 2026

Naman Vats and Oleg Golev. The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation.arXiv preprint arXiv:2607.22585, 2026

2026 arXiv
[146]

Ai agentic program- ming: A survey of techniques, challenges, and opportunities.arXiv preprint arXiv:2508.11126, 2025

Huanting Wang, Jingzhi Gong, Huawei Zhang, Jie Xu, and Zheng Wang. Ai agentic program- ming: A survey of techniques, challenges, and opportunities.arXiv preprint arXiv:2508.11126, 2025

2025
[147]

A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

2024
[148]

Agentfly: Extensible and scalable reinforcement learning for lm agents.arXiv preprint arXiv:2507.14897, 2025

Renxi Wang, Rifo Ahmad Genadi, Bilal El Bouardi, Yongxin Wang, Fajri Koto, Zhengzhong Liu, Timothy Baldwin, and Haonan Li. Agentfly: Extensible and scalable reinforcement learning for lm agents.arXiv preprint arXiv:2507.14897, 2025

2025 arXiv
[149]

Executable code actions elicit better llm agents.arXiv preprint arXiv:2402.01030, 2024

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents.arXiv preprint arXiv:2402.01030, 2024

2024 arXiv
[150]

The openhands software agent sdk: A composable and extensible foundation for production agents.arXiv preprint arXiv:2511.03690, 2025

Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, et al. The openhands software agent sdk: A composable and extensible foundation for production agents.arXiv preprint arXiv:2511.03690,...

2025 arXiv
[151]

Agents in software engineering: Survey, landscape, and vision.Automated Software Engineering, 32(2):70, 2025

Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. Agents in software engineering: Survey, landscape, and vision.Automated Software Engineering, 32(2):70, 2025

2025
[152]

Clawsafety:" safe" llms, unsafe agents.arXiv preprint arXiv:2604.01438, 2026

Bowen Wei, Yunbei Zhang, Jinhao Pan, Kai Mei, Xiao Wang, Jihun Hamm, Ziwei Zhu, and Yingqiang Ge. Clawsafety:" safe" llms, unsafe agents.arXiv preprint arXiv:2604.01438, 2026

2026 arXiv
[153]

Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.Advances in Neural Information Processing Systems, 38:78500–78525, 2026

Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.Advances in Neural Information Processing Systems...

2026
[154]

Git context controller: Manage the context of llm-based agents like git.arXiv preprint arXiv:2508.00031, 2025

Junde Wu, Minhao Hu, Jiayuan Zhu, Jiazhen Pan, Yuyuan Liu, Min Xu, and Yueming Jin. Git context controller: Manage the context of llm-based agents like git.arXiv preprint arXiv:2508.00031, 2025

2025 arXiv
[155]

Large-scale terminal agentic trajectory generation from dockerized environments.arXiv preprint arXiv:2602.01244, 2026

Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, et al. Large-scale terminal agentic trajectory generation from dockerized environments.arXiv preprint arXiv:2602.01244, 2026

2026 arXiv
[156]

The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

2025
[157]

Demystifying llm-based software engineering agents.Proceedings of the ACM on Software Engineering, 2(FSE):801–824, 2025

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software engineering agents.Proceedings of the ACM on Software Engineering, 2(FSE):801–824, 2025

2025
[158]

Live-swe-agent: Can software engineering agents self-evolve on the fly?arXiv preprint arXiv:2511.13646, 2025

Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live-swe-agent: Can software engineering agents self-evolve on the fly?arXiv preprint arXiv:2511.13646, 2025

2025
[159]

Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Sy...

2024
[160]

Astra-bench: Evaluating tool-use agent reasoning and action planning with personal user context.arXiv preprint arXiv:2603.01357, 2026

Zidi Xiu, David Q Sun, Kevin Cheng, Maitrik Patel, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, et al. Astra-bench: Evaluating tool-use agent reasoning and action planning with personal user context.arXiv preprint arXiv:2603.01357, 2026

2026
[161]

Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

2026
[162]

Cleaner: Self-purified trajectories boost agentic reinforcement learning.arXiv preprint arXiv:2601.15141, 2026

Tianshi Xu, Yuteng Chen, and Meng Li. Cleaner: Self-purified trajectories boost agentic reinforcement learning.arXiv preprint arXiv:2601.15141, 2026

2026 arXiv
[163]

When the specification emerges: Benchmarking faithfulness loss in long-horizon coding agents.arXiv preprint arXiv:2603.17104, 2026

Lu Yan, Xuan Chen, and Xiangyu Zhang. When the specification emerges: Benchmarking faithfulness loss in long-horizon coding agents.arXiv preprint arXiv:2603.17104, 2026. 35

2026
[164]

Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback.Advances in Neural Information Processing Systems, 36:23826–23854, 2023

2023
[165]

Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

2024
[166]

Swe-smith: Scaling data for software engineering agents.Advances in Neural Information Processing Systems, 38, 2026

John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents.Advances in Neural Information Processing Systems, 38, 2026

2026
[167]

What makes interaction trajectories effective for training terminal agents?arXiv preprint arXiv:2606.03461, 2026

Sidi Yang, Chaofan Tao, Jierun Chen, Tiezheng Yu, Ruoyu Wang, Yuxin Jiang, Yiming Du, Wendong Xu, Jing Xiong, Taiqiang Wu, et al. What makes interaction trajectories effective for training terminal agents?arXiv preprint arXiv:2606.03461, 2026

2026 arXiv
[168]

Ace-bench: Agent configurable evaluation with scalable horizons and controllable difficulty under lightweight environments

Wang Yang, Chaoda Song, Xinpeng Li, Debargha Ganguly, Chuang Ma, Shouren Wang, Zhihao Dou, Yuli Zhou, Vipin Chaudhary, and Xiaotian Han. Ace-bench: Agent configurable evaluation with scalable horizons and controllable difficulty under lightweight environments. arXiv e-prints, ...

2026
[169]

Kimi-dev: Agentless training as skill prior for swe-agents.arXiv preprint arXiv:2509.23045, 2025

Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yingwei Ma, et al. Kimi-dev: Agentless training as skill prior for swe-agents.arXiv preprint arXiv:2509.23045, 2025

2025
[170]

React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

2022 arXiv
[171]

Claw-eval: Towards trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132, 2026

Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Towards trustworthy evaluation of autonomous agents.arXiv preprint arXiv:2604.06132, 2026

2026 arXiv
[172]

Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025

2025 arXiv
[173]

A survey on efficient vision-language-action models.arXiv preprint arXiv:2510.24795, 2025

Zhaoshu Yu, Bo Wang, Pengpeng Zeng, Haonan Zhang, Ji Zhang, Zheng Wang, Lianli Gao, Jingkuan Song, Nicu Sebe, and Heng Tao Shen. A survey on efficient vision-language-action models.arXiv preprint arXiv:2510.24795, 2025

2025
[174]

debug-gym: A text-based environment for interactive debugging.arXiv preprint arXiv:2503.21557, 2025

Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, et al. debug-gym: A text-based environment for interactive debugging.arXiv preprint arXiv:2503.21557, 2025

2025 arXiv
[175]

Multi-swe-bench: A multilingual benchmark for issue resolving.Advances in Neural Information Processing Systems, 38, 2026

Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.Advances in Neural Information Processing Systems, 38, 2026

2026
[176]

davinci-dev: Agent-native mid-training for software engineering.arXiv preprint arXiv:2601.18418, 2026

Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, et al. davinci-dev: Agent-native mid-training for software engineering.arXiv preprint arXiv:2601.18418, 2026. 36

2026
[177]

Swe-hub: A unified production system for scalable, executable software engineering tasks.arXiv preprint arXiv:2603.00575, 2026

Yucheng Zeng, Shupeng Li, Daxiang Dong, Ruijie Xu, Zimo Chen, Liwei Zheng, Yuxuan Li, Zhe Zhou, Haotian Zhao, Lun Tian, et al. Swe-hub: A unified production system for scalable, executable software engineering tasks.arXiv preprint arXiv:2603.00575, 2026

2026
[178]

Agentfore- sight: Online auditing for early failure prediction in multi-agent systems.arXiv preprint arXiv:2605.08715, 2026

Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, and Ruixiang Tang. Agentfore- sight: Online auditing for early failure prediction in multi-agent systems.arXiv preprint arXiv:2605.08715, 2026

2026 arXiv
[179]

Docksmith: Scaling reliable coding environments via an agentic docker builder.arXiv preprint arXiv:2602.00592, 2026

Jiaran Zhang, Luck Ma, Yanhao Li, Fanqi Wan, Di Qi, Xu Zhao, Jieyi Hou, Zhe Xie, Mengqiang Ren, Xin Wu, et al. Docksmith: Scaling reliable coding environments via an agentic docker builder.arXiv preprint arXiv:2602.00592, 2026

2026 arXiv
[180]

Debugging the debuggers: Failure- anchored structured recovery for software engineering agents.arXiv preprint arXiv:2605.08717, 2026

Chenyu Zhao, Shenglin Zhang, Yihang Lin, Wenwei Gu, Zhimin Chen, Yongqian Sun, Dan Pei, Chetan Bansal, Saravan Rajmohan, and Minghua Ma. Debugging the debuggers: Failure- anchored structured recovery for software engineering agents.arXiv preprint arXiv:2605.08717, 2026

2026 arXiv
[181]

Fail- ure as a process: An anatomy of cli coding agent trajectories.arXiv preprint arXiv:2607.09510, 2026

Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T Barr, Federica Sarro, and He Ye. Fail- ure as a process: An anatomy of cli coding agent trajectories.arXiv preprint arXiv:2607.09510, 2026

2026 arXiv
[182]

Lifelongagentbench: Evaluating llm agents as lifelong learners.arXiv preprint arXiv:2505.11942, 2025

Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, ZhongZhi Li, Yingying Zhang, Le Song, and Qianli Ma. Lifelongagentbench: Evaluating llm agents as lifelong learners.arXiv preprint arXiv:2505.11942, 2025

2025 arXiv
[183]

Claw-swe-bench: A benchmark for evaluating openclaw-style agent harnesses on coding tasks.arXiv preprint arXiv:2606.12344, 2026

Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, et al. Claw-swe-bench: A benchmark for evaluating openclaw-style agent harnesses on coding tasks.arXiv preprint arXiv:2606.12344, 2026

2026 arXiv
[184]

Towards agentic os: An llm agent framework for linux schedulers.arXiv preprint arXiv:2509.01245, 2025

Yusheng Zheng, Yanpeng Hu, Wei Zhang, and Andi Quinn. Towards agentic os: An llm agent framework for linux schedulers.arXiv preprint arXiv:2509.01245, 2025

2025
[185]

Hardening agent benchmarks with adversarial hacker-fixer loops.arXiv preprint arXiv:2606.08960, 2026

Ziqian Zhong, Ivgeni Segal, Ivan Bercovich, Shashwat Saxena, Kexun Zhang, and Aditi Raghunathan. Hardening agent benchmarks with adversarial hacker-fixer loops.arXiv preprint arXiv:2606.08960, 2026

2026 arXiv
[186]

Externalization in llm agents: A unified review of memory, skills, protocols and harness engineering.arXiv preprint arXiv:2604.08224, 2026

Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, et al. Externalization in llm agents: A unified review of memory, skills, protocols and harness engineering.arXiv preprint arXiv:2604.08224, 2026

2026 arXiv
[187]

Memento: Fine-tuning llm agents without fine-tuning llms.arXiv preprint arXiv:2508.16153, 2025

Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, et al. Memento: Fine-tuning llm agents without fine-tuning llms.arXiv preprint arXiv:2508.16153, 2025

2025 arXiv
[188]

Webarena: A realistic web environment for building autonomous agents

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, volume 2024, pag...

2024
[189]

Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents.arXiv preprint arXiv:2602.07274, 2026

Kaijie Zhu, Yuzhou Nie, Yijiang Li, Yiming Huang, Jialian Wu, Jiang Liu, Ximeng Sun, Zhenfei Yin, Lun Wang, Zicheng Liu, et al. Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents.arXiv preprint arXiv:2602.07274, 2026. 37

2026
[190]

Where llm agents fail and how they can learn from failures.arXiv preprint arXiv:2509.25370, 2025

Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, et al. Where llm agents fail and how they can learn from failures.arXiv preprint arXiv:2509.25370, 2025

2025
[191]

Agentclick: A skill-based human-in- the-loop review layer for terminal ai agents

Haomin Zhuang, Hanwen Xing, and Xiangliang Zhang. Agentclick: A skill-based human-in- the-loop review layer for terminal ai agents. InProceedings of the ACM Conference on AI and Agentic Systems, pages 1372–1378, 2026. APPENDIX A Review Protocol and Corpus Construction This sup...

2026
[193]

Does the agent execute terminal commands, operate CLI tools, or interact with a terminal- mediated environment?
[194]

Does stdout, stderr, logs, diffs, return codes, or execution feedback materially shape subsequent actions?
[195]

Does the system produce real or simulated environment state changes through execution?
[196]

Does the work provide executable verification, trajectory data, a benchmark, an acquisition pipeline, or a runtime architecture relevant to terminal-mediated execution or terminal-agent systems?
[197]

terminal agent

What claim scope does the work support: core terminal-agent evidence, terminal-hybrid evidence, executable SWE-adjacent evidence, boundary comparison, or background framing? Based on these questions, each retained entry received a corpus status and one or more analytical roles...

2022
[2026]

URLhttps://arxiv.org/abs/2608.08654

arXiv

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.

[1] [1]

Riva: Leveraging llm agents for reliable configuration drift detection

Sami Abuzakuk, Lucas Crijns, Anne-Marie Kermarrec, Rafael Pires, and Martijn de Vos. Riva: Leveraging llm agents for reliable configuration drift detection. InProceedings of the Sixth European Workshop on Machine Learning and Systems, pages 499–509, 2026

2026

[2] [2]

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, and Valentin Malykh. Swe-mera: A dynamic benchmark for agenticly evaluating large language models on software engineering tasks.arXiv preprint arXiv:2507.11059, 2025

work page Pith review arXiv 2025

[3] [3]

Ai ides or autonomous agents? measuring the impact of coding agents on software development

Shyam Agarwal, Hao He, and Bogdan Vasilescu. Ai ides or autonomous agents? measuring the impact of coding agents on software development. InProceedings of the 23rd International Conference on Mining Software Repositories, pages 857–862, 2026

2026

[4] [4]

Do agents dream of root shells? partial-credit evaluation of llm agents in capture the flag challenges

Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek, Roland Vízner, Arie van Deursen, and Maliheh Izadi. Do agents dream of root shells? partial-credit evaluation of llm agents in capture the flag challenges. InProceedings of the 3rd ACM International Conference on AI-Powered Software, pages 349–357, 2026. 23

2026

[5] [5]

Agentharm: A benchmark for measuring harmfulness of llm agents

Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of llm agents. InInternational Conference on Learning Representations, volume 2025, pages 79185–79220, 2025

2025

[6] [6]

Claude Code, 2025

Anthropic. Claude Code, 2025. URLhttps://code.claude.com/docs/en/overview. Agen- tic coding tool available through terminal and other development surfaces, with file editing, command execution, and development-tool integration

2025

[7] [7]

SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing soft- ware engineering agents’ ability to bootstrap development environments.arXiv preprint arXiv:2507.09063, 2025

work page Pith review arXiv 2025

[8] [8]

Ciber: A comprehensive benchmark for security evaluation of code interpreter agents.arXiv preprint arXiv:2602.19547, 2026

Lei Ba, Qinbin Li, and Songze Li. Ciber: A comprehensive benchmark for security evaluation of code interpreter agents.arXiv preprint arXiv:2602.19547, 2026

work page arXiv 2026