Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Building SE agents reorganizes the old software process into a seven-stage loop steered by evaluation.

desk verdict First real interview study of how practitioners build SE agents; the process model is useful but the adaptive interview protocol makes the 'discovered' workflow less secure than claimed. read the letter →

arxiv 2607.10856 v2 pith:4AMDZFIH submitted 2026-07-12 cs.SE cs.AIcs.HC

Yunbo Lyu , David Williams , Jieke Shi , Zhensu Sun , Chao Peng , Zhou Yang , Federica Sarro , David Lo This is my paper
classification cs.SEcs.AIcs.HC
keywords softwareengineeringagentsLLMdevelopmentworkflowevaluation-drivencomprehensiondebtagentharnessmixed-methodsstudyprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish, from interviews with 20 practitioners who build LLM-based software-engineering agents and a survey of 80 more, that agent development does not abolish the familiar software process but reorganizes it. Its central claim is a recurring seven-stage loop — requirements, evaluation, data, construction, testing/deployment, human feedback, adaptive maintenance — in which evaluation is defined early and repeatedly steers iteration. If true, it gives teams a concrete picture of where work actually goes: coding becomes cheaper, but specification, review, coordination, and deployment remain bottlenecks, and new kinds of work (reviewing generated code, evaluating agent behavior) take center stage. The paper also names six practical failure points, including unreliable evaluation signals and 'comprehension debt,' along with twelve practices teams use. A sympathetic reader would take this as a systematic map of an emerging engineering discipline.

What carries the argument

The central object is the seven-stage workflow itself, with evaluation as its steering mechanism: evaluation criteria are set early, reused during construction, and revisited after deployment, so that every other stage is organized around producing and interpreting an evaluation signal. The second key piece is the combination of a cheapest-first model strategy (API/prompt, then LoRA or fine-tuning, then pretraining) with a persistent harness — the wrapper that supplies context, memory, tools, skills, permissions, and orchestration — which survives model swaps and becomes the main locus of engineering effort. This machinery explains both why evaluation recurs throughout the loop and why provi

What would settle it

Have two independent teams, blinded to this paper's framework, segment and cluster the same 20 interview transcripts; if they do not converge on a similar stage structure, or if measured agreement between independent coders on assigning passages to the seven stages is low, then the claimed recurring workflow would be an artifact of the authors' coding rather than a property of practitioner experience.

Watch

Extended reading notes

Core claim

The discovery is that building SE agents in practice follows an agile-like seven-stage loop rather than a linear pipeline, with evaluation as the recurring mechanism that decides whether an iteration is good enough to continue. Construction follows a cheapest-first model strategy — start with prompts on an existing model, escalate to fine-tuning, only rarely pretrain — while a shared harness (the layer managing context, memory, tools, skills, permissions, and orchestration) persists across model choices. Evaluation is defined early and reused at every stage, and specifications such as prompts, skills, and context definitions become first-class artifacts tested and versioned alongside code. T

Load-bearing premise

That the seven-stage workflow and the five process shifts genuinely reflect how SE-agent builders work, rather than how the three researchers who coded the 20 interview transcripts chose to group the material, since the categories were settled by consensus without independent coding or inter-rater reliability checks.

Editorial extensions

If this is right

  • If the seven-stage loop is accurate, teams should expect to spend more early effort on requirements and evaluation design, because those stages now steer all subsequent iteration.
  • If evaluation-driven development is real, agent-building teams need continuous evaluation infrastructure — regression-style benchmarks and production-derived signals — rather than a single final validation pass.
  • If specifications are first-class artifacts, then prompts, skills, context definitions, and scaffold configuration should be versioned, reviewed, and tested in the same pipeline as code.
  • If 'change nothing, change everything' holds, provider model updates are a routine source of behavioral drift that teams must monitor and re-evaluate against even when their own code is unchanged.
  • If comprehension debt accumulates, code volume alone is a misleading productivity metric; review burden and the ability to later modify generated code are the more relevant costs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'regenerative software' idea could be tested by comparing teams that preserve specifications, tests, and skills as the durable asset against teams that preserve the generated artifact itself, measuring maintainability and time-to-repair over several months.
  • The 'change nothing, change everything' effect implies that model choice becomes a coupling point: SE-agent teams may need model-agnostic harness abstractions and explicit upgrade/pinning policies, analogous to dependency management for the model itself.
  • If evaluation-steered development generalizes, the natural next step is to check whether the same seven-stage loop appears in organizations that are not big tech or in agent-building teams with less research exposure; the paper's own sample skew leaves that open.
  • Comprehension debt may be measurable with proxy signals such as growth in the unreviewed pull-request backlog or the time from merge to a developer's first successful explanation of a change — signals future work could turn into dashboard metrics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a mixed-methods study of how practitioners build LLM-based software engineering (SE) agents. The authors conducted semi-structured interviews with 20 practitioners from 12 organizations (mostly large technology companies), analyzed the transcripts using thematic analysis and hybrid card sorting, and then ran an online survey of 80 practitioners to validate the findings. The central claim is that building SE agents does not eliminate traditional software engineering but reorganizes it into a recurring seven-stage loop (requirements, evaluation, data, system construction, testing/deployment, human feedback, adaptive maintenance) in which evaluation increasingly acts as a central steering mechanism. The paper also identifies five process shifts (cheaper implementation, unmasked/created effort, evaluation-driven development, shrinking role boundaries, specifications as first-class artifacts), six challenges, and 12 associated practices. The authors report that member checking confirmed the findings and that the survey showed broad agreement (91% for the workflow, 71–95% for the shifts).

Significance. If the empirical characterization is trustworthy, this is a valuable contribution: it is among the first interview-based studies of SE-agent construction, and it provides concrete evidence for process changes that have previously been anecdotal. The proposed constructs—evaluation-driven development as an observed practice, 'comprehension debt', the 'change nothing, change everything' effect, and 'regenerative software'—are plausible and useful for future research. The paper is also transparent about its sample and limitations, and it commits to a replication package. However, the independence of the evidence for the central workflow is weaker than the text suggests: the interview guide was revised on the basis of provisional analyses, the coding used consensus without inter-rater reliability, and the survey items were derived from the same researchers' interpretations. These issues are addressable, but they currently leave the central claim more vulnerable than the paper's confident framing warrants.

major comments (4)
  1. [Section III-B2] The paper states that after the 5th, 10th, and 15th interviews the researchers 'discussed newly emerging findings and refined a provisional development workflow' and that these discussions 'also informed adjustments to the questions and probes used in subsequent interviews.' Because later participants were asked about a workflow that had already been constructed from earlier data, the claim that the workflow 'recurs' across participants (Section IV-A) and that saturation was approached after 18 interviews is weaker than it appears: the recurring pattern may be partly an artifact of the probes. Please (i) report exactly what was changed in the protocol after each revision point, (ii) show whether the seven stages are present in the first five interviews before any provisional workflow was discussed, and (iii) provide a saturation analysis that separates themes mentioned spontaneously from
  2. [Section III-B3] The analysis section reports that the three authors 'jointly clustered the units and resolved disagreements through discussion and consensus rather than independent coding or inter-rater reliability.' This is a legitimate mode of analysis, but it makes the central workflow's validity depend on the researchers' joint judgment. The member-checking step (Section III-B4) only shows that participants recognize the researchers' framing, not that the workflow would emerge independently. Please provide the codebook and stage definitions, an audit trail of the card-sorting process, and at least a subset of transcripts independently coded by two researchers with agreement reported. If inter-rater reliability is not used, justify this choice in relation to standard qualitative practices and provide alternative trustworthiness evidence such as negative-case or deviant-case analysis.
  3. [Section III-C1] The survey is described as validating the interview findings, but the survey statements are reformulations of the same analysis that produced the workflow and shifts. The 91% agreement for W1 (Table II) therefore does not break the circularity: respondents are endorsing the authors' interpretation, not independently discovering the same structure. The survey is better characterized as an extension of the interview findings to a wider population than as an independent corroboration. Please reframe the language accordingly and, if possible, include at least some survey items that are direct quotes or minimally paraphrased raw statements from participants, and report open-ended survey responses separately.
  4. [Section IV-A / Fig. 2] Figure 2 defines data as a 'conditional component: not appear in every team,' yet the paper repeatedly describes the workflow as a 'seven-stage' loop (e.g., the Section IV-A takeaway and Section IV-C). If one of the seven components is absent for some teams, the universality of the seven-stage structure is unclear. The paper should either restrict the claim to teams that perform data work or state more precisely which components are invariant and which are conditional.
minor comments (5)
  1. [Section V] The text contains rendering artifacts such as '♂lightbulbImplications' and '/char◎-barx%'. These should be replaced with standard labels and percentage formatting.
  2. [Tables II and III] Please clarify whether the 'Agreement' column is the percentage of respondents choosing 'Agree' or 'Strongly agree' (or the sum), and report the number of respondents per row, since some items had an 'I don't know' option.
  3. [Section III-B3] The criteria for selecting 'five process findings and six challenges' from the 52 findings are under-specified. State the selection rule explicitly or provide the full list of findings in the replication package.
  4. [Section VI] The confirmation-bias threat is acknowledged in the threats section, but its concrete implications for the iterative guide (Section III-B2) and consensus coding (Section III-B3) should also be discussed in the methods section, not only in the limitations.
  5. [Abstract] The abstract claims this is the first study of process changes in SE agent development; the text more cautiously says 'To our knowledge.' Keep the cautious phrasing in the abstract as well.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity in the survey validation and interview-guide evolution; the qualitative corpus itself is not derived from a formula or self-citation chain.

  1. other [Section III-B2 (Semi-structured Interviews)]
    "After the 5th, 10th, and 15th interviews, three researchers discussed newly emerging findings and refined a provisional development workflow. These discussions also informed adjustments to the questions and probes used in subsequent interviews."

    The load-bearing empirical claim—that a recurring seven-stage workflow exists—rests on interview data collected with a guide that was itself adjusted to probe for a provisional workflow after interviews 5, 10, and 15. Later confirmations and the saturation statement ('After 18 interviews, thematic saturation was approaching') are therefore not fully independent evidence: the instrument and the phenomenon it claims to discover co-evolved. The authors acknowledge this only as 'confirmation bias' in Section VI, but for the central workflow claim it is a partial construction-by-design rather than a purely discovered regularity.

  2. self definitional [Section III-C1 and Table II, item W1]
    "To validate the interview findings, we conducted a follow-up survey... presented statements on the identified process shifts, challenges, and practices (Tables II and III). ... W1 Overall workflow. The seven-stage loop reflects how SE agents are built in interviewees' experience."

    The survey is presented as corroborating the interview findings ('Our survey supports the overall workflow (91% agreement)'), but the survey item is a restatement of the very construct it is supposed to validate. Asking respondents to rate 'The seven-stage loop reflects how SE agents are built' cannot confirm that the seven-stage loop is a natural, independently reproducible pattern; it only measures whether practitioners recognize the researchers' framing. The agreement rate is thus partly an artifact of item wording rather than an independent test of the qualitative derivation.

full rationale

This is a qualitative mixed-methods paper, so there are no equations or fitted parameters whose prediction reduces by construction; nor is there a load-bearing self-citation chain, imported uniqueness theorem, or ansatz smuggled via citation. The central evidence—the 20 transcripts, verbatim participant quotes, and the card-sorting analysis—is genuine raw data and gives the workflow and process-shift claims independent content. However, two steps in the validation chain are partially self-referential. First, the interview guide was deliberately refined around a provisional workflow after interviews 5, 10, and 15, and those refinements changed later probes; this makes the reported 'recurrence' and saturation partly dependent on the researchers' evolving construct. Second, the survey was constructed from the interview findings and then used to 'corroborate' those same findings: the W1 item simply asks respondents to agree that the seven-stage loop is how agents are built, so 91% agreement is a recognition check, not an independent test. The paper itself concedes in Section VI that 'confirmation bias remains possible' because the same researchers designed the guide and analyzed the data. These issues do not make the entire study circular—the interview corpus and member checks provide real grounding—but they do mean the validation step is partly forced by the researchers' own definitions. A score of 4 reflects this partial, validation-layer circularity rather than a fully self-referential derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 3 invented entities

The paper introduces three conceptual entities (comprehension debt, regenerative software, change-nothing-change-everything) and relies on several domain assumptions about participant honesty, saturation, and the stability of the coding-derived workflow. No free parameters in the mathematical sense are present; the survey agreement percentages are descriptive, not fitted.

assumptions (4)
  • domain assumption Interviewees and survey respondents accurately and honestly report their development practices
    All data are self-reported; the paper acknowledges this in Section VI. There is no independent verification of participant claims.
  • domain assumption Thematic saturation after 20 interviews indicates comprehensive coverage of the phenomenon
    Saturation is a judgment call; the authors state that two additional interviews yielded no new findings (Section III-B2), but saturation is not a formal guarantee and may be influenced by the sample's homogeneity.
  • ad hoc to paper The seven-stage workflow is a stable recurring pattern across teams, not an artifact of the coding process
    This is the paper's central synthesis from hybrid card-sorting, confirmed by consensus without inter-rater reliability (Section III-B3). It is not independently tested with a pre-registered coding scheme.
  • domain assumption Member checking by 8 of 15 invited interviewees provides valid evidence of interpretive accuracy
    Member checking is a recognized technique, but the respondents who chose to respond may be not representative of all participants; the paper does not describe the selection or non-response analysis (Section III-B4).
invented entities (3)
  • Comprehension debt
    purpose: Characterizes the accumulation of generated code that is accepted before developers can understand and review it; introduced in Section V-E.
    The term is coined based on interview themes and illustrated with quotes, but it lacks external validation or a quantitative measure.
  • Regenerative software
    purpose: An emerging alternative where teams preserve specifications, tests, and infrastructure to regenerate software rather than preserving the artifact itself; described in Section V-E.
    Proposed based on a single participant's vision (P8); not observed as a widespread practice and not independently validated.
  • Change nothing, change everything
    purpose: Describes how provider-side model updates can alter agent behavior even when the team changes no code, prompts, or tools; defined in Section V-B.
    A conceptual label for a documented phenomenon, but it is not measured or independently reproduced outside the paper's interview and survey data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study." pith.science (2026). https://pith.science/paper/4AMDZFIH

@misc{pith2026260710856,
  author       = {Pith},
  title        = {Pith review of: How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AMDZFIH}},
  note         = {Machine review of arXiv:2607.10856}
}
read the original abstract

The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing work in requirements, coordination, and deployment becomes more visible, while reviewing generated code and evaluating agent behavior become new and increasingly central forms of work. We characterize a seven-stage workflow and five process shifts, including a move toward evaluation-driven development, in which evaluation is increasingly defined early and steers iteration, and the emergence of specifications as first-class artifacts that teams test and version alongside code. We further identify six challenges that teams face, together with 12 corresponding practices they use or propose to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.

Figures

Figures reproduced from arXiv: 2607.10856 by the authors.

Figure 1
Figure 1. Overview of our research method. We adopt an exploratory sequential mixed-methods design. The interview study proceeds from scoping and interview [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Workflow for building SE agents. The process forms an agile-like loop from requirements and early evaluation to construction, deployment, feedback, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The formation and responses to comprehension debt. Coding-agent [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. We Must Have Missed This Comment: Detecting and Repairing Stale Function References in Linux Kernel Comments

    cs.SE 2026-08 accept novelty 6.0 of 10

    ReCite detects 869 stale function references in Linux kernel v6.18-rc1 comments, with 89% of sampled repairs judged useful and 42.5% directly applicable.

  2. Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A prefix-only 0.6B monitor can forecast SWE-agent failure and, paired with an edit-overlay restart, reclaim tokens and improve resolution.

Reference graph

Works this paper leans on

97 extracted references · 15 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  2. [2]

    Claude Code,

    Anthropic, “Claude Code,” https://claude.com/product/claude-code

  3. [3]

    OpenAI, “Codex,” https://openai.com/codex/

  4. [4]

    When AI builds itself,

    M. Favaro and J. Clark, “When AI builds itself,” https://www.anthropic. com/institute/recursive-self-improvement, Anthropic Institute, 2026

    2026

  5. [5]

    Sources: Cursor in talks to raise $2b+ at $50b valuation as enterprise growth surges,

    M. Temkin, “Sources: Cursor in talks to raise $2b+ at $50b valuation as enterprise growth surges,” TechCrunch. https://techcrunch.com/2026/04/17/sources-cursor-in-talks-to-raise- 2b-at-50b-valuation-as-enterprise-growth-surges/, 2026

    2026

  6. [6]

    An empirical study of agent developer practices in ai agent frameworks,

    Y . Wang, X. Xu, J. Chen, T. Bi, W. Gu, and Z. Zheng, “An empirical study of agent developer practices in ai agent frameworks,”arXiv preprint arXiv:2512.01939, 2025

    arXiv 2025

  7. [7]

    What challenges do developers face in ai agent systems? an empirical study on stack overflow & github issues,

    A. Asgari, A. Panichella, P. Derakhshanfar, and M. Olsthoorn, “What challenges do developers face in ai agent systems? an empirical study on stack overflow & github issues,”arXiv preprint arXiv:2510.25423, 2025

    arXiv 2025

  8. [8]

    An empirical study of testing practices in open source ai agent frameworks and agentic applications,

    M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, and A. E. Hassan, “An empirical study of testing practices in open source ai agent frameworks and agentic applications,”Empirical Software Engineering, vol. 31, no. 5, p. 124, 2026

    2026

Show all 97 references
  1. [9]

    Measuring agents in production,

    M. Z. Pan, N. Arabzadeh, R. Cogo, Y . Zhu, A. Xiong, L. A. Agrawal, H. Mao, E. Shen, S. Pallerla, L. Patelet al., “Measuring agents in production,”arXiv preprint arXiv:2512.04123, 2025

    2025 arXiv

  2. [10]

    Collaboration challenges in building ml-enabled systems: communication, documentation, engi- neering, and process,

    N. Nahar, S. Zhou, G. Lewis, and C. K ¨astner, “Collaboration challenges in building ml-enabled systems: communication, documentation, engi- neering, and process,” ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 413–425

    2022

  3. [11]

    How does machine learning change software development practices?

    Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1857–1871, 2021

    2021

  4. [12]

    Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products,

    N. Nahar, C. K ¨astner, J. Butler, C. Parnin, T. Zimmermann, and C. Bird, “Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products,” in2025 IEEE/ACM 47th In- ternational Conference on Software Engineering: Software Engineer...

    2025

  5. [13]

    Building your own product copilot: Challenges, opportunities, and needs,

    C. Parnin, G. Soares, R. Pandita, S. Gulwani, J. Rich, and A. Z. Henley, “Building your own product copilot: Challenges, opportunities, and needs,” in2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2025, pp. 338–348

    2025

  6. [14]

    Member check- ing: a tool to enhance trustworthiness or merely a nod to validation?

    L. Birt, S. Scott, D. Cavers, C. Campbell, and F. Walter, “Member check- ing: a tool to enhance trustworthiness or merely a nod to validation?” Qualitative health research, vol. 26, no. 13, pp. 1802–1811, 2016

    2016

  7. [15]

    Large language model-based agents for software engineering: A sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A sur- vey,”ACM Trans. Softw. Eng. Methodol., Mar. 2026, just Accepted

    2026

  8. [16]

    Agentless: De- mystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  9. [17]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,”arXiv preprint arXiv:2407.16741, 2024

    2024 arXiv

  10. [18]

    Pomona: Continuous code quality improvement via small, automated changes at bloomberg,

    D. Williams, A. Evripiotis, S. Kirbas, H. Morgan, S. Magidovich, P. Wainwright, and F. Sarro, “Pomona: Continuous code quality improvement via small, automated changes at bloomberg,” 2026. [Online]. Available: https://arxiv.org/abs/2606.06752

    2026 arXiv

  11. [19]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    2024

  12. [20]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. ...

    2024

  13. [21]

    Ga4gc: Greener agent for greener code via multi-objective configura- tion optimization,

    J. Gong, Y . Bian, L. De la Cal, G. Pinna, A. Uteem, D. Williams, M. Zamorano, K. Even-Mendoza, W. Langdon, H. Menendezet al., “Ga4gc: Greener agent for greener code via multi-objective configura- tion optimization,”arXiv preprint arXiv:2510.04135, 2025

    2025

  14. [22]

    Agents in software engineering: Survey, landscape, and vision,

    Y . Wang, W. Zhong, Y . Huang, E. Shi, M. Yang, J. Chen, H. Li, Y . Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: Survey, landscape, and vision,”Automated Software Engineering, vol. 32, no. 2, p. 70, 2025

    2025

  15. [23]

    From llms to llm- based agents for software engineering: A survey of current, challenges and future,

    H. Jin, L. Huang, H. Cai, J. Yan, B. Li, and H. Chen, “From llms to llm- based agents for software engineering: A survey of current, challenges and future,”arXiv preprint arXiv:2408.02479, 2024

    2024 arXiv

  16. [24]

    Agentszz: Teaching the llm agent to play detective with bug-inducing commits,

    Y . Lyu, J. Shi, H. J. Kang, R. Widyasari, J. He, Y . Niu, C. Yang, J. Chen, Z. Yang, J. Lawallet al., “Agentszz: Teaching the llm agent to play detective with bug-inducing commits,”arXiv preprint arXiv:2604.02665, 2026

    2026 arXiv

  17. [25]

    Understanding software engineering agents: A study of thought-action-result trajectories,

    I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,”arXiv preprint arXiv:2506.18824, 2025

    2025

  18. [26]

    Washizaki, Ed.,Guide to the Software Engineering Body of Knowl- edge (SWEBOK): Version 4.0a

    H. Washizaki, Ed.,Guide to the Software Engineering Body of Knowl- edge (SWEBOK): Version 4.0a. IEEE Computer Society, 2025

    2025

  19. [27]

    ISO/IEC/IEEE,ISO/IEC/IEEE 24748-1:2018 Systems and Software Engineering—Life Cycle Management—Part 1: Guidelines for Life Cy- cle Management, 2018

    2018

  20. [28]

    Software development lifecycle models,

    N. B. Ruparelia, “Software development lifecycle models,”ACM SIG- SOFT Software Engineering Notes, vol. 35, no. 3, pp. 8–13, 2010

    2010

  21. [29]

    Managing the development of large software systems: concepts and techniques,

    W. W. Royce, “Managing the development of large software systems: concepts and techniques,” inProceedings of the 9th international con- ference on Software Engineering, 1987, pp. 328–338

    1987

  22. [30]

    R. S. Pressman,Software engineering: a practitioner’s approach. Pal- grave macmillan, 2005

    2005

  23. [31]

    A spiral model of software development and enhance- ment,

    B. W. Boehm, “A spiral model of software development and enhance- ment,”Computer, vol. 21, no. 5, pp. 61–72, 1988

    1988

  24. [32]

    The relationship of system engineering to the project cycle,

    K. Forsberg, H. Moozet al., “The relationship of system engineering to the project cycle,”Center for Systems Management, vol. 5333, pp. 4–6, 1991

    1991

  25. [33]

    Manifesto for agile software development,

    Agile Alliance, “Manifesto for agile software development,” 2001

    2001

  26. [34]

    Schwaber and M

    K. Schwaber and M. Beedle,Agile software development with Scrum. Prentice Hall PTR, 2001

    2001

  27. [35]

    Beck,Extreme programming explained: embrace change

    K. Beck,Extreme programming explained: embrace change. addison- wesley professional, 2000

    2000

  28. [36]

    USA: Addison-Wesley Longman Publishing Co., Inc., 2002

    Beck,Test Driven Development: By Example. USA: Addison-Wesley Longman Publishing Co., Inc., 2002. 11

    2002

  29. [37]

    ISO/IEC/IEEE,ISO/IEC/IEEE 32675:2022 Information Technology— DevOps—Building Reliable and Secure Systems Including Application Build, Package and Deployment, 2022

    2022

  30. [38]

    Machine learning operations (mlops): Overview, definition, and architecture,

    D. Kreuzberger, N. K ¨uhl, and S. Hirschl, “Machine learning operations (mlops): Overview, definition, and architecture,”IEEE access, vol. 11, pp. 31 866–31 879, 2023

    2023

  31. [39]

    Large language model operations (llmops): Definition, challenges, and lifecycle management,

    J. Diaz-De-Arcaya, J. L ´opez-De-Armentia, R. Mi ˜n´on, I. L. Ojanguren, and A. I. Torre-Bastida, “Large language model operations (llmops): Definition, challenges, and lifecycle management,” in2024 9th Inter- national Conference on Smart and Sustainable Technologies (SpliTech...

    2024

  32. [40]

    Agentops: Enabling observability of llm agents,

    L. Dong, Q. Lu, and L. Zhu, “Agentops: Enabling observability of llm agents,”arXiv preprint arXiv:2411.05285, 2024

    2024 arXiv

  33. [41]

    Evaluation- driven development and operations of llm agents: A process model and reference architecture,

    B. Xia, Q. Lu, L. Zhu, Z. Xing, D. Zhao, and H. Zhang, “Evaluation- driven development and operations of llm agents: A process model and reference architecture,”arXiv preprint arXiv:2411.13768, 2024

    2024

  34. [42]

    Software engineering for machine learning: A case study,

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Na- gappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” in2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEI...

    2019

  35. [43]

    Professional software developers don’t vibe, they control: Ai agent use for coding in 2025,

    R. Huang, A. Reyna, S. Lerner, H. Xia, and B. Hempel, “Professional software developers don’t vibe, they control: Ai agent use for coding in 2025,”arXiv preprint arXiv:2512.14012, 2025

    2025

  36. [44]

    A Meta- Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners ,

    N. Nahar, H. Zhang, G. Lewis, S. Zhou, and C. Kastner, “A Meta- Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners ,” in2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN...

    2023

  37. [45]

    The pipeline for the continuous development of artificial intelligence models—current state of research and practice,

    M. Steidl, M. Felderer, and R. Ramler, “The pipeline for the continuous development of artificial intelligence models—current state of research and practice,”Journal of Systems and Software, vol. 199, p. 111615, 2023

    2023

  38. [46]

    The design space of llm-based ai coding assistants: An analysis of 90 systems in academia and industry,

    S. Lau and P. J. Guo, “The design space of llm-based ai coding assistants: An analysis of 90 systems in academia and industry,”2025 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 300–313, 2025

    2025

  39. [47]

    Advanced mixed methods research designs,

    J. W. Creswell, V . L. Plano Clark, M. L. Gutmann, and W. E. Hanson, “Advanced mixed methods research designs,”Handbook of mixed meth- ods in social and behavioral research, vol. 209, no. 240, pp. 209–240, 2003

    2003

  40. [48]

    App store effects on software engineering practices,

    A. A. Al-Subaihin, F. Sarro, S. Black, L. Capra, and M. Harman, “App store effects on software engineering practices,”IEEE Transactions on Software Engineering, vol. 47, no. 2, pp. 300–319, 2021

    2021

  41. [49]

    A mixed- method study of hot fixing in industry: Practices, bottlenecks, and opportunities,

    C. Hanna, D. Elliman, W. Emmerich, F. Sarro, and J. Petke, “A mixed- method study of hot fixing in industry: Practices, bottlenecks, and opportunities,”ACM Trans. Softw. Eng. Methodol., May 2026

    2026

  42. [50]

    The Product Beyond the Model – An Empirical Study of Repositories of Open-Source ML Products,

    N. Nahar, H. Zhang, G. Lewis, S. Zhou, and C. K ¨astner, “The Product Beyond the Model – An Empirical Study of Repositories of Open-Source ML Products,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, ...

    2025

  43. [51]

    Whatscode: Large-scale genai deployment for developer efficiency at whatsapp,

    K. Mao, T. Kapus, C. T. ˚Ahs, M. Marescotti, D. Ip, ´A. Hajdu, S. Cela, and A. Banerjee, “Whatscode: Large-scale genai deployment for developer efficiency at whatsapp,”arXiv preprint arXiv:2512.05314, 2025

    2025

  44. [52]

    A survey on code generation with llm-based agents,

    Y . Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, “A survey on code generation with llm-based agents,”arXiv preprint arXiv:2508.00083, 2025

    2025 arXiv

  45. [53]

    Demystifying evals for AI agents,

    Anthropic, “Demystifying evals for AI agents,” https://www.anthropic. com/engineering/demystifying-evals-for-ai-agents, Jan. 2026

    2026

  46. [54]

    How AI is transforming work at Anthropic,

    S. Huang, B. Seethor, E. Durmus, K. Handa, M. McCain, M. Stern, and D. Ganguli, “How AI is transforming work at Anthropic,” Dec

  47. [55]

    Open-sourcing adaptive workflows for AI-driven development life cycle (AI-DLC),

    W. Matos, R. Jain, S. Jog, and R. SP, “Open-sourcing adaptive workflows for AI-driven development life cycle (AI-DLC),” Nov. 2025. [On- line]. Available: https://aws.amazon.com/blogs/devops/open-sourcing- adaptive-workflows-for-ai-driven-development-life-cycle-ai-dlc/

    2025

  48. [56]

    An empirical study on challenges for llm application developers,

    X. Chen, C. Gao, C. Chen, G. Zhang, and Y . Liu, “An empirical study on challenges for llm application developers,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–37, 2025

    2025

  49. [57]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  50. [58]

    Replication package,

    Anonymous, “Replication package,” https://figshare.com/s/ ead7dd318ec49b37a20f, 2026, accessed: June 30, 2026

    2026

  51. [59]

    Saturation in qualitative research: exploring its conceptualization and operationalization,

    B. Saunders, J. Sim, T. Kingstone, S. Baker, J. Waterfield, B. Bartlam, H. Burroughs, and C. Jinks, “Saturation in qualitative research: exploring its conceptualization and operationalization,”Quality & quantity, vol. 52, no. 4, pp. 1893–1907, 2018

    1907

  52. [60]

    Smart contract development: Challenges and opportunities,

    W. Zou, D. Lo, P. S. Kochhar, X.-B. D. Le, X. Xia, Y . Feng, Z. Chen, and B. Xu, “Smart contract development: Challenges and opportunities,” IEEE Transactions on Software Engineering, vol. 47, no. 10, pp. 2084– 2106, 2021

  53. [61]

    Turnover-induced knowledge loss in practice,

    M. P. Robillard, “Turnover-induced knowledge loss in practice,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 1292–1302

    2021

  54. [62]

    Using thematic analysis in psychology,

    V . Braun and V . Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology, vol. 3, no. 2, pp. 77–101, 2006

    2006

  55. [63]

    Recommended steps for thematic synthesis in software engineering,

    D. S. Cruzes and T. Dyba, “Recommended steps for thematic synthesis in software engineering,” in2011 international symposium on empirical software engineering and measurement. IEEE, 2011, pp. 275–284

    2011

  56. [64]

    Lazar, J

    J. Lazar, J. H. Feng, and H. Hochheiser,Research methods in human- computer interaction. Morgan Kaufmann, 2017

    2017

  57. [65]

    Spencer,Card sorting: Designing usable categories

    D. Spencer,Card sorting: Designing usable categories. Rosenfeld Media, 2009

    2009

  58. [66]

    Reliability and inter-rater reliability in qualitative research: Norms and guidelines for cscw and hci practice,

    N. McDonald, S. Schoenebeck, and A. Forte, “Reliability and inter-rater reliability in qualitative research: Norms and guidelines for cscw and hci practice,”Proceedings of the ACM on human-computer interaction, vol. 3, no. CSCW, pp. 1–23, 2019

    2019

  59. [67]

    In- vestigating the role of cultural values in adopting large language models for software engineering,

    S. Lambiase, G. Catolino, F. Palomba, F. Ferrucci, and D. Russo, “In- vestigating the role of cultural values in adopting large language models for software engineering,”ACM Transactions on Software Engineering and Methodology, vol. 35, no. 1, pp. 1–43, 2025

    2025

  60. [68]

    Assessing programming language impact on development and maintenance: A study on c and c++,

    P. Bhattacharya and I. Neamtiu, “Assessing programming language impact on development and maintenance: A study on c and c++,” in Proceedings of the 33rd international conference on software engineer- ing, 2011, pp. 171–180

    2011

  61. [69]

    Why psychologists should by default use welch’s t-test instead of student’s t-test,

    M. Delacre, D. Lakens, and C. Leys, “Why psychologists should by default use welch’s t-test instead of student’s t-test,”International Review of Social Psychology, vol. 30, no. 1, pp. 92–101, 2017

    2017

  62. [70]

    Evaluating szz implementations: An empirical study on the linux kernel,

    Y . Lyu, H. J. Kang, R. Widyasari, J. Lawall, and D. Lo, “Evaluating szz implementations: An empirical study on the linux kernel,”IEEE Trans. Softw. Eng., vol. 50, no. 9, p. 2219–2239, Sep. 2024

    2024

  63. [71]

    ” my productivity is boosted, but

    Y . Lyu, Z. Yang, J. Shi, J. Chang, Y . Liu, and D. Lo, “” my productivity is boosted, but. . . ” demystifying users’ perception on ai coding assis- tants,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 191–203

    2025

  64. [72]

    Measuring the impact of early-2025 ai on experienced open-source developer productivity,

    J. Becker, N. Rush, E. Barnes, and D. Rein, “Measuring the impact of early-2025 ai on experienced open-source developer productivity,”arXiv preprint arXiv:2507.09089, 2025

    2025 arXiv

  65. [73]

    How coding agents fail their users: A large-scale analysis of developer-agent misalignment in 20,574 real-world sessions,

    N. Tang, C. Chen, G. Xu, Y . Shi, Y . Huang, C. McMillan, T. Dong, and T. J.-J. Li, “How coding agents fail their users: A large-scale analysis of developer-agent misalignment in 20,574 real-world sessions,”arXiv preprint arXiv:2605.29442, 2026

    2026 arXiv

  66. [74]

    F. P. Brooks,No Silver Bullet Essence and Accidents of Software Engineering. Washington, DC, USA: IEEE Computer Society Press, Apr. 1987, vol. 20, no. 4

    1987

  67. [75]

    F. P. Brooks Jr,The mythical man-month: essays on software engineer- ing. Pearson Education, 1995

    1995

  68. [76]

    Vibe coding in product teams: Reconfiguring ai-assisted workflows, prototyping, and collaboration,

    J. Li, Y . Hou, L. Lin, R. Zhu, H. Cao, and A. El Ali, “Vibe coding in product teams: Reconfiguring ai-assisted workflows, prototyping, and collaboration,” inProceedings of the 5th Annual Symposium on Human- Computer Interaction for Work, 2026, pp. 1–16

    2026

  69. [77]

    Product manager practices for delegating work to generative ai:

    M. Ulloa, J. L. Butler, S. Haniyur, C. Miller, B. Amos, A. Sarkar, and M.-A. Storey, “Product manager practices for delegating work to generative ai:” accountability must not be delegated to non-human actors”,”arXiv preprint arXiv:2510.02504, 2025

    2025

  70. [78]

    Llm-assisted repository-level generation with structured spec-driven engineering,

    S. Feng, B. Chen, B. H. Meyer, and G. Mussbacher, “Llm-assisted repository-level generation with structured spec-driven engineering,” arXiv preprint arXiv:2605.02455, 2026

    2026 arXiv

  71. [79]

    On testing non-testable programs,

    E. J. Weyuker, “On testing non-testable programs,”The Computer Journal, vol. 25, no. 4, pp. 465–470, 1982. 12

    1982

  72. [80]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    2015

  73. [81]

    Empirical and sustainability aspects of software engineering research in the era of large language models: A reflection,

    D. Williams, M. Hort, M. Kechagia, A. Aleti, J. Petke, and F. Sarro, “Empirical and sustainability aspects of software engineering research in the era of large language models: A reflection,”arXiv preprint arXiv:2510.26538, 2026

    2026

  74. [82]

    Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Raneet al., “Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?”arXiv preprint arXiv:2509.16941, 2025

    2025 arXiv

  75. [83]

    Hidden tech- nical debt in machine learning systems,

    D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V . Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden tech- nical debt in machine learning systems,”Advances in neural information processing systems, vol. 28, 2015

    2015

  76. [84]

    The bitter lesson, 2019,

    R. Sutton, “The bitter lesson, 2019,”URL http://www. incompleteideas. net/IncIdeas/BitterLesson. html, 2019

    2019

  77. [85]

    Injecagent: Benchmark- ing indirect prompt injections in tool-integrated large language model agents,

    Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “Injecagent: Benchmark- ing indirect prompt injections in tool-integrated large language model agents,” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 10 471–10 506

    2024

  78. [86]

    Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,” inProceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90

    2023

  79. [87]

    ” your ai, my shell

    Y . Liu, Y . Zhao, Y . Lyu, T. Zhang, H. Wang, and D. Lo, “” your ai, my shell”: Demystifying prompt injection attacks on agentic ai coding editors,”arXiv preprint arXiv:2509.22040, 2025

    2025 arXiv

  80. [88]

    Securevibebench: Benchmarking secure vibe coding of ai agents via reconstructing vulnerability-introducing scenarios,

    J. Chen, H. Huang, Y . Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y . Li, Z. Liet al., “Securevibebench: Benchmarking secure vibe coding of ai agents via reconstructing vulnerability-introducing scenarios,” inProceedings of the 64th Annual Meeting of the Association for C...

    2026

  81. [89]

    Effective context engineering for AI agents,

    Anthropic, “Effective context engineering for AI agents,” https://www.anthropic.com/engineering/effective-context-engineering- for-ai-agents, 2025, accessed: 2026-06-01

    2025

  82. [90]

    Agentic context engineering: Evolving contexts for self-improving language models,

    Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V . Kamanuru, J. Rainton, C. Wu, M. Ji, H. Liet al., “Agentic context engineering: Evolving contexts for self-improving language models,”arXiv preprint arXiv:2510.04618, 2025

    2025 arXiv

  83. [91]

    Acquiring and sharing tacit knowledge in software development teams: An empirical study,

    S. Ryan and R. V . O’Connor, “Acquiring and sharing tacit knowledge in software development teams: An empirical study,”Information and software technology, vol. 55, no. 9, pp. 1614–1624, 2013

    2013

  84. [92]

    Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used,

    F. O. Bjørnson and T. Dingsøyr, “Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used,”Information and software technology, vol. 50, no. 11, pp. 1055–1068, 2008

    2008

  85. [93]

    Context length alone hurts llm performance despite perfect retrieval,

    Y . Du, M. Tian, S. Ronanki, S. Rongali, S. Bodapati, A. Galstyan, A. Wells, R. Schwartz, E. A. Huerta, and H. Peng, “Context length alone hurts llm performance despite perfect retrieval,”arXiv preprint arXiv:2510.05381, 2025

    2025

  86. [94]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the association for computational linguistics, vol. 12, pp. 157–173, 2024

    2024

  87. [95]

    From technical debt to cognitive and intent debt: Rethinking software health in the age of ai,

    M.-A. Storey, “From technical debt to cognitive and intent debt: Rethinking software health in the age of ai,” 2026. [Online]. Available: https://arxiv.org/abs/2603.22106

    2026 arXiv

  88. [96]

    The fast and spurious: Developer productivity with genai

    S. Afroz, Z. Feng, T. Menezes, K. Kimura, B. Trinkenreich, I. Stein- macher, and A. Sarma, “The fast and spurious: Developer productivity with genai.” ACM International Conference on the Foundations of Software Engineering, 2026. 13

    2026

  89. [2025]

    Available: https://www.anthropic.com/research/how-ai- is-transforming-work-at-anthropic

    [Online]. Available: https://www.anthropic.com/research/how-ai- is-transforming-work-at-anthropic

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.