Pith. sign in

REVIEW 3 major objections 5 minor 72 references

LLM-agent failures often arise at the boundary between the model and the harness: the paper identifies agent-reactive (AR) bugs as failures that appear only when a specific LLM behavior meets harness code that mishandles it, and supports th

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:33 UTC pith:FICEURDQ

load-bearing objection A genuinely useful taxonomy of agent-reactive bugs with a clear definition and persuasive examples, though the headline prevalence numbers are conditional on project-specific engagement filters and unmeasured annotation agreement. the 3 major comments →

arxiv 2607.15684 v1 pith:FICEURDQ submitted 2026-07-17 cs.SE

Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports

Jingyi Chen , Songqiang Chen , Hengcheng Zhu , Jialun Cao , Jiasi Shen , Shing-Chi Cheung This is my paper
classification cs.SE
keywords agent-reactive bugsLLM agentsmodel-harness interactionbug taxonomysilent errorsissue report miningtest oraclestool-use failures
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a meaningful share of LLM-agent failures cannot be blamed on the model or the harness alone. It defines agent-reactive (AR) bugs as failures whose manifestation depends on a specific LLM behavior, such as ignoring instructions or calling a tool with unexpected arguments, combined with a harness that mishandles that behavior. Analyzing 255 bug reports from two command-line agents and two agent frameworks, it finds five observable symptoms and eight triggering behaviors, with silent errors the most common (108 of 255). If correct, this means agent reliability work needs to study the model-harness boundary, not either component in isolation, and needs test oracles and reproduction techniques tailored to these interaction-dependent failures.

Core claim

On its own terms, the paper claims that agent-reactive bugs are a real, nameable class: a bug is AR when a particular LLM response (e.g., non-compliant instruction, unexpected tool argument, template mismatch, fabricated statement, empty response, context overflow, tool hallucination) is handled by harness code in a way that produces a user-visible symptom (silent error, crash, error in output, retry loop, hang). Across 255 engagement-filtered issues, the paper builds a two-axis taxonomy and reports that silent errors dominate; that the same behavior maps to different symptoms depending on harness validation and recovery; and that users and developers often disagree about whether the fix bel

What carries the argument

The carrying mechanism is the two-axis taxonomy that pairs each AR bug's observable symptom with the LLM behavior that triggered it, built by manual annotation of 3,037 engagement-filtered issue reports from four projects. The taxonomy does the analytical work: it converts a vague report like 'the agent failed' into (triggering behavior × symptom), and it exposes the central mechanism of AR bugs — that the harness's parsing, validation, retry, and recovery policy decides which symptom a given LLM behavior produces. The work also uses a repair-target rubric (harness PR, model attribution, ambiguous, no engagement) to track user proposals against developer actions.

Load-bearing premise

The results depend on the human labels: the counts and symptom/behavior distributions assume that the two annotators' classification of 3,037 engaged issues accurately reflects which bugs truly depend on a specific LLM response, even though the paper does not report inter-rater agreement and filters out unengaged issues that might contain AR bugs.

What would settle it

A reader could test the central claim by re-annotating a random sample of the 2,782 issues the study classified as non-AR: if independent annotators find a substantial number of agent-reactive bugs in that pool, or show low agreement on the original 255, the 8.4% prevalence and the symptom/behavior distributions are measurement artifacts rather than stable properties of LLM agents. A second check is to run the same pipeline on another set of open-source agent projects and see whether silent errors still dominate and whether the same behavior-to-symptom mapping appears.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Agent reliability studies should treat the model-harness boundary as a first-class fault location; a bug can be invisible when either side is inspected alone.
  • Testing agents requires trace-based oracles: because silent errors and fabricated statements look fluent, checks must compare the agent's narration against tool logs and workspace state.
  • Reproduction support should preserve or mock the specific triggering LLM behavior (argument values, template-breaking messages, long context), not just replay user inputs.
  • Fault-localization tools for agents should distinguish model limitations from harness robustness gaps; the taxonomy gives a vocabulary for that attribution.
  • Maintainers can use the symptom/trigger/repair-target checklist to structure issue reports and triage discussions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 8.4% prevalence holds beyond these four projects, agent evaluation benchmarks that score only final answers will systematically miss the most common failure mode; benchmark harnesses should log and audit trajectories.
  • The user/developer mismatch suggests a coordination problem: when vendors control both model and harness, they may prefer model-side fixes; independent framework maintainers may need explicit fault-attribution guidelines to keep users engaged.
  • A direct testable extension: apply the same annotation protocol to additional open-source agents (or later issue windows) and check whether silent errors still dominate and whether instruction non-compliance still produces the most repair-target disagreement.
  • The triggering-behavior taxonomy could seed automated classification models that label new issue reports as AR, giving maintainers an early-warning signal before deep manual triage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper defines agent-reactive (AR) bugs as failures that manifest only when a backend LLM emits a particular behavior and the agent harness handles that behavior improperly. The authors mine GitHub issues from Codex, Gemini-CLI, LangChain, and CrewAI, filter them to 3,037 'engaged' issues, manually label 255 as AR, and construct a two-axis taxonomy of five symptom categories and eight triggering LLM behaviors. They report that silent errors are the most frequent symptom (108/255), that instruction non-compliance and unexpected tool arguments are the most common triggers, and that users' harness-side repair proposals are frequently not matched by developer actions, with developer responses often ambiguous or attributed to the model. The paper claims this is the first empirical study dedicated to this bug class, and it motivates future work on test oracles, reproduction support, and fault localization for LLM agents.

Significance. If the empirical findings hold, this is a useful first characterization of a boundary-layer bug class that prior agent-bug taxonomies largely treated as either model-capability failures or harness defects. The two-axis symptom/trigger taxonomy and the user-developer repair-target mismatch analysis are concrete, well-illustrated contributions with direct implications for designing trace-based oracles and behavior-preserving reproduction techniques. The paper is clearly written and the representative issue examples are persuasive. However, the empirical core is weakened by the absence of an inter-rater reliability statistic, the lack of a released dataset or artifact, and the unexamined engagement-filter selection step on which all headline prevalence and distribution numbers are conditioned.

major comments (3)
  1. [Section III-B, Table I] The engagement filter is project-specific and is applied before any AR/non-AR labeling. Codex/Gemini-CLI require a linked PR or at least three non-author commenters; LangChain requires a linked PR and at least three commenters; CrewAI requires a linked PR and at least one commenter. This leaves 3,037 of 11,288 LLM-reachable issues. Every headline number — 8.4% prevalence, the silent-error majority, the 5x8 counts, the CLI-vs-framework comparisons — is conditional on this filter, and the filter plausibly biases the internal distribution. Silent errors may attract fewer comments precisely because users do not realize a failure occurred, so the reported frequency could be an artifact of which issues got discussed. Section VI acknowledges manual subjectivity but does not address this internal-validity threat. Please either (a) annotate a random sample of the 8,251 filtered-out LLM-reachable
  2. [Section III-B, Phase 2] No inter-rater reliability metric is reported. In Phase 2, the two annotators partitioned the remaining 95% of the engagement-filtered pool and each labeled their own portion; they did not independently label a shared sample to measure agreement. Because the taxonomy categories and the binary AR/non-AR labels are subjective judgments, unreported agreement rates leave the stability of all counts and distributions open to question. The paper should report Cohen's kappa (or a comparable statistic) on at least a sizable shared sample, along with the annotation guidelines and conflict-resolution details.
  3. [Section III-B / Contributions] The paper's third contribution is a manually annotated dataset of 255 AR bugs, but no dataset or replication artifact is released or even referenced. Without the issue identifiers, the annotated labels, and the final taxonomy mapping, readers cannot verify the counts, reproduce the distributions, or build on the dataset. For an empirical study whose main product is a labeled corpus, this is a load-bearing reproducibility gap. Please provide a public artifact (e.g., a repository with the 255 bugs, the labels, and the coding rubric).
minor comments (5)
  1. [Section IV-B (Statement Fabrication)] Typos: 'For exmaple' and 'Another exmaple' should be 'example'.
  2. [Section IV-A] The phrase 'Summing the bottomTot row' should read 'Total row'.
  3. [Table II] The table is rendered as one long line per row and is very hard to parse. Please present each project as a separate subtable with clear column headers, or use a multi-index table with a proper matrix layout.
  4. [Section III-B] The 'LLM-reachable' filter is defined as reproduction steps containing an LLM invocation. Many issue reports lack explicit reproduction steps; please state how such reports are handled and whether this could exclude AR bugs that are only observable in a live session.
  5. [Abstract / Introduction] The 8.4% figure is stated as if it is a natural prevalence, but it is explicitly 'of all actively discussed issues.' Please carry that qualifier into the abstract and introduction to avoid over-generalization.

Circularity Check

0 steps flagged

No circularity: the study is an empirical taxonomy built from manually annotated issue reports, with no fitted parameters, derived equations, or load-bearing self-citations.

full rationale

This paper is a qualitative empirical study, not a derivation with fitted constants or predicted outputs. The central claim—that AR bugs are a distinct class whose manifestation depends on both a specific LLM behavior and harness handling—is defined conceptually and then evidenced by manually labeling 255 issue reports. The taxonomy of five symptoms and eight triggering behaviors is induced from the annotated corpus, not assumed in the selection criteria. No equation reduces a reported result to an input definition: the AR/non-AR label is applied by human annotators to issues in the engagement-filtered pool, and the prevalence (8.4%) is a measured count, not a fitted parameter. The project-specific engagement thresholds described in Section III-B may affect the generalizability or representativeness of the prevalence and distribution, but this is a measurement validity threat, not circularity: the filters are not defined in terms of the AR label, and the annotation question ('does the bug’s manifestation depend on a specific LLM response?') is independent of the filter. The paper's own Section VI acknowledges manual subjectivity and limited generalization, which is appropriate, and does not hide a circular step. Self-citations in the references (e.g., [24], [25]) concern prior code-generation studies and are not load-bearing for the AR-bug taxonomy. There is no invocation of a uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result presented as a derivation. The main risks—annotation subjectivity, absence of reported inter-rater agreement, and filter bias—are concerns about evidence quality, not about circular reasoning. Accordingly, no significant circularity is present.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

No fitted model parameters beyond the hand-set engagement thresholds; the paper makes no quantitative prediction. The central empirical claims rest on sampling and annotation assumptions: representativeness of four projects, unbiasedness of the engagement filter, and accuracy of consensus labeling. The authors acknowledge the first in Section VI; the engagement-filter conditionality is visible in Table I but not treated as a bias.

free parameters (1)
  • Project-specific engagement thresholds = Codex/Gemini: PR or ≥3 commenters; LangChain: PR and ≥3; CrewAI: PR and ≥1
    Section III-B: these hand-set thresholds determine the 3,037-issue pool from which the 8.4% AR prevalence and all RQ1–RQ3 distributions are computed.
axioms (3)
  • domain assumption The four studied projects (Codex, Gemini-CLI, LangChain, CrewAI) represent the class of LLM agents.
    Section III.A subject selection; Section VI concedes closed-source agents like Claude Code are excluded and generalizability is uncertain.
  • domain assumption The engagement-filtered subset (3,037 issues) is a reliable sample for estimating AR-bug prevalence and repair dynamics.
    Section III.B filters 11,288 LLM-reachable issues down by project-specific thresholds; prevalence and mismatch figures are conditional on this subset.
  • domain assumption Two annotators' consensus labels (without reported inter-rater reliability) are ground truth for AR/non-AR and taxonomy categories.
    Section III.B Phases 1–2 and Section VI acknowledge subjectivity; no kappa/agreement statistic is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 19372 in / 11812 out tokens · 94881 ms · 2026-08-01T22:33:48.283570+00:00 · methodology

0 comments
read the original abstract

LLM agents span command-line interfaces (e.g., Codex) and agent frameworks (e.g., LangChain), integrating backend LLMs with harness code that parses model outputs, controls agent loops, and manages context. Both the harness and LLM-generated responses jointly shape an agent's execution. This architecture gives rise to bugs that cannot be readily understood by inspecting either component alone, because some bugs occur only when a particular LLM response elicits an abnormal reaction from the agent. Prior empirical studies of agent bugs have largely attributed failures either to limited model capabilities or to harness-side defects, such as outdated APIs and configuration misalignment, without characterizing these AR bugs. We conduct the first empirical study focused on agent-reactive (AR) bugs. Through manual analysis of 255 bug reports from Codex, Gemini-CLI, LangChain, and CrewAI, we construct a two-axis taxonomy covering observable symptoms and the LLM behaviors that trigger them. Our findings show that many AR bugs manifest as silent errors without well-defined test oracles, which makes detection difficult. The stochasticity of LLM responses further complicates bug reproduction. We additionally examine fixes proposed by users and implemented by developers. This analysis exposes a mismatch: users frequently advocate harness-side guardrails, whereas developers may attribute the issue to the LLM or respond slowly to user-proposed fixes. These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs. Overall, the study highlights challenges specific to LLM agents and motivates the design of test oracles, reproduction support, and fault-localization techniques for AR bugs.

Figures

Figures reproduced from arXiv: 2607.15684 by Hengcheng Zhu, Jialun Cao, Jiasi Shen, Jingyi Chen, Shing-Chi Cheung, Songqiang Chen.

Figure 1
Figure 1. Figure 1: Codex issue 13491: a forked worker inherits parent intent and attempts [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Codex issue 5957: after auto-compaction, the agent should continue [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: CrewAI issue 668: an omitted tool argument becomes an error in the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Codex issue 4337: after timeout, the wrapper kills the main process, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Repair target codes by user and developer. Ribbon width indicates [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 9 linked inside Pith

  1. [1]

    Swe-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real- world github issues?” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  2. [2]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024

    2024

  3. [3]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, U. Alon, and G. Neubig, “Webarena: A realistic web environment for building autonomous agents,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  4. [4]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “ τ-bench: A benchmark for tool-agent-user interaction in real-world domains,”CoRR, vol. abs/2406.12045, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, pp. 42:1–42:28, 2025

    2025

  6. [6]

    When agents fail: A comprehensive study of bugs in LLM agents with automated labeling,

    N. Islam, R. S. Ayon, D. Thomas, S. Ahmed, and M. Wardat, “When agents fail: A comprehensive study of bugs in LLM agents with automated labeling,”CoRR, vol. abs/2601.15232, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    An empirical study of bugs in modern LLM agent frameworks,

    X. Zhu, J. Wu, X. Zhang, T. Li, Y . Mu, J. Zhai, C. Shen, C. Fang, and Y . Liu, “An empirical study of bugs in modern LLM agent frameworks,” CoRR, vol. abs/2602.21806, 2026

    arXiv 2026

  8. [8]

    Charac- terizing faults in agentic AI: A taxonomy of types, symptoms, and root causes,

    M. B. Shah, M. M. Morovati, M. M. Rahman, and F. Khomh, “Charac- terizing faults in agentic AI: A taxonomy of types, symptoms, and root causes,”CoRR, vol. abs/2603.06847, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Dissecting bug triggers and failure modes in modern agentic frameworks: An empirical study,

    X. Zhang, H. Zhang, and S. H. Tan, “Dissecting bug triggers and failure modes in modern agentic frameworks: An empirical study,”CoRR, vol. abs/2604.08906, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Agentbench: Evaluating llms as agents,

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y . Su, H. Sun, M. Huang, Y . Dong, and J. Tang, “Agentbench: Evaluating llms as agents,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRevi...

    2024

  11. [11]

    GAIA: a benchmark for general AI assistants,

    G. Mialon, C. Fourrier, T. Wolf, Y . LeCun, and T. Scialom, “GAIA: a benchmark for general AI assistants,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  12. [12]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,”ACM Trans. Inf. Syst., vol. 43, no. 2, pp. 42:1–42:55, 2025

    2025

  13. [13]

    Spectool: A benchmark for characterizing errors in tool-use llms,

    S. Kokane, M. Zhu, T. Awalgaonkar, J. Zhang, T. Hoang, A. Prabhakar, Z. Liu, T. Lan, L. Yang, J. Tan, R. Murthy, W. Yao, Z. Liu, J. C. Niebles, H. Wang, S. Heinecke, C. Xiong, and S. Savarese, “Spectool: A benchmark for characterizing errors in tool-use llms,”CoRR, vol. abs/2411.13547, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  15. [15]

    Reducing tool hallucination via reliability alignment,

    H. Xu, Z. Zhu, L. Pan, Z. Wang, S. Zhu, D. Ma, R. Cao, L. Chen, and K. Yu, “Reducing tool hallucination via reliability alignment,” in Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, ser. Proceedings of Machine Learning Research, vol. 267. PMLR / OpenReview.net, 2025

    2025

  16. [16]

    Instruction-following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” CoRR, vol. abs/2311.07911, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Instruction-following evaluation in function calling for large language models,

    N. Skripko, “Instruction-following evaluation in function calling for large language models,”CoRR, vol. abs/2509.18420, 2025

    arXiv 2025

  18. [18]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Trans. Assoc. Comput. Linguistics, vol. 12, pp. 157–173, 2024

    2024

  19. [19]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE Trans. Software Eng., vol. 41, no. 5, pp. 507–525, 2015

    2015

  20. [20]

    Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,

    F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, “Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,” Empir. Softw. Eng., vol. 29, no. 1, p. 10, 2024

    2024

  21. [21]

    An empirical analysis of flaky tests,

    Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” inProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014. ACM, 2014, pp. 643–653

    2014

  22. [22]

    Large language models for software engineer- ing: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineer- ing: A systematic literature review,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 220:1–220:79, 2024

    2024

  23. [23]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, and et al., “Evaluating large language models trained on code,”CoRR, vol. abs/2107.03374, 2021

    Pith/arXiv arXiv 2021

  24. [24]

    When llms meet API documentation: Can retrieval augmentation aid code generation just as it helps developers?

    J. Chen, S. Chen, J. Cao, J. Shen, and S. Cheung, “When llms meet API documentation: Can retrieval augmentation aid code generation just as it helps developers?”CoRR, vol. abs/2503.15231, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Can emulating semantic translation help llms with code translation? A study based on pseudocode,

    S. Chen, C. Xu, J. Chen, J. Cao, J. Wu, and S. Cheung, “Can emulating semantic translation help llms with code translation? A study based on pseudocode,”CoRR, vol. abs/2510.00920, 2025

    arXiv 2025

  26. [26]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Trans. Software Eng., vol. 50, no. 1, pp. 85–105, 2024

    2024

  27. [27]

    The rise and potential of large language model based agents: a survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, and B. H. et al., “The rise and potential of large language model based agents: a survey,”Sci. China Inf. Sci., vol. 68, no. 2, 2025

    2025

  28. [28]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  29. [29]

    Agent harness for large language model agents: A survey,

    Q. Meng, Y . Wang, L. Chen, W. Wu, Y . Li, W. Jiang, Q. Wang, C. Lu, Y . Gao, Y . Wu, and Y . Hu, “Agent harness for large language model agents: A survey,”Preprints, 2026, preprint, DOI 10.20944/preprints202604.0428

    arXiv 2026

  30. [30]

    OpenAI Codex, issue #5957,

    “OpenAI Codex, issue #5957,” https://github.com/openai/codex/issues/ 5957, gitHub issue

  31. [31]

    OpenAI Codex, issue #6562,

    “OpenAI Codex, issue #6562,” https://github.com/openai/codex/issues/ 6562, gitHub issue

  32. [32]

    LangChain, issue #1358,

    “LangChain, issue #1358,” https://github.com/langchain-ai/langchain/ issues/1358, gitHub issue

  33. [33]

    Gemini CLI, issue #13292,

    “Gemini CLI, issue #13292,” https://github.com/google-gemini/ gemini-cli/issues/13292, gitHub issue

  34. [34]

    CrewAI, issue #668,

    “CrewAI, issue #668,” https://github.com/crewAIInc/crewAI/issues/668, gitHub issue

  35. [35]

    Gemini CLI, issue #5629,

    “Gemini CLI, issue #5629,” https://github.com/google-gemini/gemini-cli/ issues/5629, gitHub issue

  36. [36]

    OpenAI Codex, issue #4337,

    “OpenAI Codex, issue #4337,” https://github.com/openai/codex/issues/ 4337, gitHub issue

  37. [37]

    OpenAI Codex, issue #13491,

    “OpenAI Codex, issue #13491,” https://github.com/openai/codex/issues/ 13491, gitHub issue

  38. [38]

    Gemini CLI, issue #3037,

    “Gemini CLI, issue #3037,” https://github.com/google-gemini/gemini-cli/ issues/3037, gitHub issue

  39. [39]

    CrewAI, issue #3154,

    “CrewAI, issue #3154,” https://github.com/crewAIInc/crewAI/issues/ 3154, gitHub issue

  40. [40]

    Gemini CLI, issue #7223,

    “Gemini CLI, issue #7223,” https://github.com/google-gemini/gemini-cli/ issues/7223, gitHub issue

  41. [41]

    CrewAI, issue #3843,

    “CrewAI, issue #3843,” https://github.com/crewAIInc/crewAI/issues/ 3843, gitHub issue

  42. [42]

    OpenAI Codex, issue #5807,

    “OpenAI Codex, issue #5807,” https://github.com/openai/codex/issues/ 5807, gitHub issue

  43. [43]

    OpenAI Codex, issue #10828,

    “OpenAI Codex, issue #10828,” https://github.com/openai/codex/issues/ 10828, gitHub issue

  44. [44]

    Gemini CLI, issue #533,

    “Gemini CLI, issue #533,” https://github.com/google-gemini/gemini-cli/ issues/533, gitHub issue

  45. [45]

    LangChain, issue #12077,

    “LangChain, issue #12077,” https://github.com/langchain-ai/langchain/ issues/12077, gitHub issue

  46. [46]

    LangChain, issue #5163,

    “LangChain, issue #5163,” https://github.com/langchain-ai/langchain/ issues/5163, gitHub issue

  47. [47]

    LangChain, issue #11408,

    “LangChain, issue #11408,” https://github.com/langchain-ai/langchain/ issues/11408, gitHub issue

  48. [48]

    Gemini CLI, issue #1484,

    “Gemini CLI, issue #1484,” https://github.com/google-gemini/gemini-cli/ issues/1484, gitHub issue

  49. [49]

    LangChain, issue #34910,

    “LangChain, issue #34910,” https://github.com/langchain-ai/langchain/ issues/34910, gitHub issue

  50. [50]

    Automated structural testing of llm-based agents: Methods, framework, and case studies,

    J. Kohl, O. Kruse, Y . Mostafa, A. Luckow, K. Schroer, T. Riedl, R. French, D. Katz, M. P. Luitz, T. Takher, K. E. Friedl, and C. Laurent-Winter, “Automated structural testing of llm-based agents: Methods, framework, and case studies,” inIEEE International Conference on Big Data, BigData 2025, Macau, China, December 8-11, 2025. IEEE, 2025, pp. 1847–1856

    2025

  51. [51]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, and P. D. et al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    2020

  52. [52]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” inAdvances in Neural Information Processing Systems 35...

    2022

  53. [53]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022

    2022

  54. [54]

    Autogen: Enabling next-gen LLM applications via multi-agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen LLM applications via multi-agent conversation framework,”CoRR, vol. abs/2308.08155, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Metagpt: Meta programming for A multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “Metagpt: Meta programming for A multi-agent collaborative framework,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    2024

  56. [56]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, “Chatdev: Communicative agents for software development,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. Association for...

    2024

  57. [57]

    Hidden technical debt in machine learning systems,

    D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V . Chaudhary, M. Young, J. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” inAdvances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015, pp. 2503–2511

    2015

  58. [58]

    Software engineering for machine learning: a case study,

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. C. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: a case study,” inProceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25-31, 2019. IEEE / ACM, 2019, pp. 291–300

    2019

  59. [59]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” inProceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019. ACM, 2019, pp. 510–520

    2019

  60. [60]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” inICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, 2020, pp. 1110–1121

    2020

  61. [61]

    Deepxplore: Automated whitebox testing of deep learning systems,

    K. Pei, Y . Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” inProceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 28-31, 2017. ACM, 2017, pp. 1–18

    2017

  62. [62]

    CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries,

    H. V . Pham, T. Lutellier, W. Qi, and L. Tan, “CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries,” inPro- ceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. IEEE / ACM, 2019, pp. 1027–1038

    2019

  63. [63]

    Beyond accuracy: Behavioral testing of NLP models with checklist,

    M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with checklist,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, 2020, pp. 4902–4912

    2020

  64. [64]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 2022, pp. 3214–3252

    2022

  65. [65]

    Factscore: Fine-grained atomic eval- uation of factual precision in long form text generation,

    S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic eval- uation of factual precision in long form text generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computat...

    2023

  66. [66]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. J. F. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 9004–9017

    2023

  67. [67]

    Halueval: A large- scale hallucination evaluation benchmark for large language models,

    J. Li, X. Cheng, X. Zhao, J. Nie, and J. Wen, “Halueval: A large- scale hallucination evaluation benchmark for large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. Association for Computational Linguistics, 2023, pp. 6449–6464

    2023

  68. [68]

    On faithfulness and factuality in abstractive summarization,

    J. Maynez, S. Narayan, B. Bohnet, and R. T. McDonald, “On faithfulness and factuality in abstractive summarization,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, 2020, pp. 1906–1919

    2020

  69. [69]

    What makes a good bug report?

    N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” inProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2008, Atlanta, Georgia, USA, November 9-14, 2008. ACM, 2008, pp. 308–318

    2008

  70. [70]

    Who should fix this bug?

    J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in 28th International Conference on Software Engineering (ICSE 2006), Shanghai, China, May 20-28, 2006. ACM, 2006, pp. 361–370

    2006

  71. [71]

    It’s not a bug, it’s a feature: how misclassification impacts bug prediction,

    K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: how misclassification impacts bug prediction,” in35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013. IEEE Computer Society, 2013, pp. 392–401

    2013

  72. [2024]

    OpenReview.net, 2024

    2024