Three-Phase Evaluation of AI-Assisted Software Development Life Cycle

Carson Crockett; Jacob Viehe; Jason Ferraro; Joshua Strubel; Nathan Londhe; Professor Carrie Russell; Uzayr Syed

Three-Phase Evaluation of AI-Assisted Software Development Life Cycle

T0 review · 3 major / 0 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Higher AI autonomy cut development hours by roughly 84% while raising requirement adherence and lowering mental load, and the agentic platform still mattered at matched autonomy.

desk verdict Useful pilot numbers on agentic workflows, but the Copilot-vs-Kiro architecture claim is confounded by sequential learning on one student team. read the letter →

arxiv 2607.05125 v1 pith:IUVQLZWY submitted 2026-07-06 cs.SE cs.AI

Joshua Strubel , Professor Carrie Russell , Carson Crockett , Jason Ferraro , Nathan Londhe , Uzayr Syed , Jacob Viehe This is my paper · ORCID

classification cs.SEcs.AI

keywords AI-assistedsoftwareengineeringagenticAIproductivitydevelopercognitionhuman-AIcollaborationGitHubCopilotAWSKiroNASA-TLX

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a three-phase pilot in which the same four developers rebuilt one full-stack web application under rising AI autonomy: partial assistance with an in-editor coding agent, then fully AI-exclusive orchestration on that same agent, then AI-exclusive work on a specification-driven agentic platform. Development hours fell from 36.0 to 9.4 to 5.9, requirement-traceability scores rose from 18 to 19 to 20 of 22, and self-reported effort and mental demand declined, while frustration rose modestly. At the same AI-exclusive autonomy level, the specification-driven platform used fewer hours, achieved higher prompt success, and scored higher on requirements than the in-editor agent. A sympathetic reader cares because teams are already deciding how much implementation work to hand to agents and which tools to buy; the study supplies directional evidence that both autonomy level and tooling architecture move productivity, requirement fidelity, and cognitive load—and that those dimensions need not trade off in a well-scoped greenfield setting.

What carries the argument

Three-phase sequential reimplementation of one fixed full-stack app by a fixed four-developer team, with AI assistance defined by share of model-generated code, scored by development hours, RITM requirement-traceability (0–2 per feature, max 22), prompt success rate and prompts-per-feature, and NASA-TLX workload dimensions.

What would settle it

A counterbalanced or parallel experiment with independent teams of similar skill, each assigned only one phase condition on the same fixed requirements, that fails to reproduce the hour reduction, the RITM lift, or the platform gap at matched AI-exclusive autonomy would falsify attributing those gains primarily to autonomy and architecture.

Watch

Extended reading notes

Core claim

Across three sequential reimplementations of the same application, higher AI autonomy was associated with large reductions in development hours, incremental gains in requirement adherence (RITM), and lower self-reported mental demand and effort, while frustration increased modestly. Holding autonomy at AI-exclusive, the specification-driven platform outperformed the agentic in-editor assistant on hours, prompt success rate, and RITM score, which the authors take as directional evidence that tooling architecture can influence outcomes independently of autonomy level.

Load-bearing premise

The gains across phases are treated as driven mainly by AI autonomy and platform design, even though the same small team rebuilt the same app three times and therefore accumulated familiarity with the requirements, architecture, and prompting strategies.

Editorial extensions

If this is right

Fully agentic greenfield workflows may cut development hours sharply without degrading requirement adherence relative to partial AI assistance.
Platform selection should be treated as an empirical variable once autonomy is held constant, not as a commodity choice.
Organizations adopting agentic workflows should plan for rising frustration and a role shift from code authorship to orchestration even when effort and mental demand fall.
Prompt engineering and requirement translation become core developer competencies as autonomy rises.
Comparing platforms requires a joint view of hours, requirement fidelity, prompt success, and cognitive load rather than any single efficiency proxy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If sequential learning effects dominate, multi-team randomized replications may shrink the apparent autonomy and platform advantages reported here.
The modest frustration rise may foreshadow agency-loss or skill-atrophy costs that only appear in longer-running or legacy codebases.
Specification-driven agents may scale better to multi-service systems because they externalize task decomposition that in-editor agents leave to the developer.
Requirement-traceability gains under exclusive AI may reverse when requirements are ambiguous or change mid-cycle rather than fixed and well-scoped.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a small, honest pilot: same four students rebuild the same greenfield full-stack app three times (partial Copilot → AI-exclusive Copilot → AI-exclusive Kiro) and report hours, RITM, prompt metrics, and NASA-TLX. The new pieces are the fully AI-exclusive conditions and the head-to-head of two commercial agentic platforms under matched requirements. Hours drop 36 → 9.4 → 5.9, RITM rises 18 → 19 → 20, mental demand/effort fall, frustration edges up, and Kiro looks best on most of those. They are transparent about n=4, descriptive-only stats, self-reported time, and subjective prompt-success labels, and they ship a clear RITM rubric. That is more careful than a lot of tool-vendor anecdotes.

What they do well is the framing and the limitations section. RQ4 is the interesting one (architecture at fixed autonomy), the metrics mix is sensible for practice, and §VII correctly flags sequential learning as the biggest problem. Citations to Peng et al. and the field Copilot work are on-point; the paper does not overclaim statistical significance.

The soft spot is load-bearing for the abstract’s punchline. Phase 2 vs 3 is the only place they can claim “tooling architecture may influence outcomes independently of AI autonomy.” Same team, same app, no randomization or counterbalancing, so familiarity with requirements, architecture, and prompting is completely entangled with the platform switch. Hours 9.4 → 5.9 and prompt success 85% → 96% are consistent with learning as easily as with Kiro’s spec-driven design. Methods also say the external RITM evaluator was phase-informed; the Appendix says the opposite—small but real inconsistency. No code, logs, or model versions are shared, so the custom analysis tool is a black box. None of this makes the directional trends worthless; it just means they are hypotheses, not evidence for org-level tool selection.

Who it is for: SE researchers designing the next multi-team agentic study, and practitioners who want a concrete measurement template. It is not yet evidence for “switch to Kiro” or “84% savings in production.” I would still send it to peer review as an exploratory methods paper; referees can force the causal language down and demand the blinding fix. Worth a skim if you care about agentic evaluation design; not something I would cite as a result until the confounds are cleaned up.

Referee Report

3 major / 0 minor

Summary. This exploratory pilot study has a four-person student team reimplement the same full-stack web application across three sequential phases of AI assistance: partial Copilot use, AI-exclusive Copilot agentic mode, and AI-exclusive AWS Kiro. The authors report monotonic reductions in development hours (36.0 → 9.4 → 5.9), incremental RITM requirement-adherence gains (18 → 19 → 20 / 22), rising prompt success rates, and lower NASA-TLX effort and mental demand, with a modest rise in frustration. They interpret the Phase 2→3 gap at nominally matched autonomy as directional evidence that tooling architecture can matter independently of AI autonomy level, and offer the protocol as a replicable comparative framework.

Significance. If the directional patterns held under designs that separate learning from treatment, the work would speak to a timely SE question: how agentic autonomy and platform architecture jointly affect productivity, requirement fidelity, and developer workload. Strengths include multi-metric coverage (effort, RITM, interaction efficiency, NASA-TLX), explicit research questions, an external RITM rubric in the Appendix, and unusually candid Limitations (§VII). As written, however, the study is a single-team, n=4, non-randomized sequential case study; its main contribution is methodological and hypothesis-generating rather than causal evidence on autonomy or platform architecture.

major comments (3)

The central claim that tooling architecture may influence outcomes independently of AI autonomy (Abstract; RQ4; §V AI-Interaction Efficiency / Development Efficiency; §VI) rests on Phase 2 (Copilot AI-exclusive) vs Phase 3 (Kiro AI-exclusive). Methodology (Study Design) and §VII state that the same four developers reimplemented the identical application sequentially, with neither randomization nor counterbalancing, and name learning effects the most significant limitation. Hours 9.4→5.9, RITM 19→20, and prompt success 85%→96% therefore cannot be attributed to architecture vs accumulated familiarity with requirements, architecture, and prompting. Either reframe Abstract/Discussion/Conclusion to pure within-team sequential description without independence language, or add a design that separates learning from platform (e.g., counterbalanced teams). As currently framed, the independence cla
Evaluator blinding is internally inconsistent and load-bearing for RITM (RQ2). §IV Metrics states the external industry evaluator was “informed of which application was tied to which phase of development,” while the Appendix rubric states the evaluator was “not informed of which phase produced each implementation.” Resolve which protocol was used; if the evaluator was phase-informed, treat RITM as non-blind and discuss expectancy bias explicitly when interpreting the 18→19→20 progression.
Inferential overreach relative to the data. Table I and §V present purely descriptive means (n=4, no significance testing), time is self-reported, and prompt success is a real-time binary developer judgment without a standardized rubric (§IV Metrics; §VII). The Abstract and §IX still state that higher autonomy was “associated with” reduced effort, improved adherence, and lower mental workload, and that Kiro “achieved the strongest overall performance.” Tighten causal/associational language throughout to match an exploratory sequential pilot, and avoid percentage claims that imply general treatment effects (e.g., §IX “declined by 87%” vs §V “84% total relative to Phase 1”).

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation circularity: empirical observational study with measured outcomes, not fitted-or-defined-by-construction predictions.

full rationale

This paper is a three-phase empirical pilot comparing AI-assisted workflows on matched reimplementations of one full-stack app. Its load-bearing claims are associations among measured quantities (self-reported hours, external RITM scores, NASA-TLX means, prompt counts/success rates), not algebraic or definitional rearrangements of model inputs. There is no fitted parameter later relabeled as a prediction, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known closed-form result. References are external productivity and agentic-AI literature (Peng et al., Cui et al., Hart & Staveland NASA-TLX, etc.); none of the central premises reduce to an unverified self-citation chain. Subjective prompt-success judgments and sequential reuse of the same requirements knowledge are measurement/confound issues (acknowledged in §VII), not circular derivation: hours, RITM, and TLX are not forced by construction from those operational choices. The Phase 2→3 platform-architecture claim is underdetermined by design (learning effects), which is a causal-validity problem, not circularity under this analyzer’s criteria. Score 0 is therefore the correct honest finding.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

Empirical SE pilot with almost no fitted constants. Load-bearing content is domain assumptions about what counts as AI autonomy, how lines and prompts are labeled, and that sequential same-team phases can still speak to autonomy/platform effects. No new physical entities; RITM and NASA-TLX are imported instruments. The main “cost” is unseparated learning and subjective coding of success, not free parameters in a model.

assumptions (5)

domain assumption AI-coding assistance level is operationalized as the percentage of lines directly generated by an LLM; any line written or later modified by a human counts as human-written.
Methodology Study Design; drives the partial vs AI-exclusive contrast and line-category logs.
ad hoc to paper A prompt is “successful” if the developer judges in real time that output was directly usable or meaningfully advanced the feature (binary, no standardized rubric).
Metrics section; acknowledged as subjective bias source; feeds prompt success rate trends used in RQ4 discussion.
domain assumption Identical functional/non-functional requirements and the same four developers across sequential phases isolate AI assistance level/tool as the intended experimental variable.
Study Design; authors later admit this fails due to learning—still the design premise behind causal-sounding RQs.
domain assumption RITM 0–2 feature scores from one external evaluator adequately represent “code quality” / requirement adherence for comparing phases.
Metrics and RQ2; paper notes RITM ignores security, maintainability, and technical debt.
domain assumption NASA-TLX means from n=4 at phase end validly track cognitive-load differences across autonomy levels.
Standard instrument [Hart & Staveland] applied descriptively; small-n and end-of-phase timing assumed sufficient for directional claims.

invented entities (1)

Custom Copilot Analysis Tool (for normalizing Copilot Chat JSON and Kiro logs)
purpose: Compute derived prompt and efficiency metrics consistently across phases.
Mentioned in Data Collection; no public artifact or validation of the tool is provided, so metric pipelines are not independently checkable.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Three-Phase Evaluation of AI-Assisted Software Development Life Cycle." pith.science (2026). https://pith.science/paper/IUVQLZWY

@misc{pith2026260705125,
  author       = {Pith},
  title        = {Pith review of: Three-Phase Evaluation of AI-Assisted Software Development Life Cycle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUVQLZWY}},
  note         = {Machine review of arXiv:2607.05125}
}

read the original abstract

This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 4 linked inside Pith

[1]

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,

S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” arXiv:2302.06590, Feb. 2023

arXiv 2023
[2]

The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot,

B. Cui, E. Yeh, P. Cihon, and M. Demirer, “The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot,” MIT Generative AI Lab, 2024

2024
[3]

Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice,

P. Khojah, M. Mohamad, P. Leitner, and R. G. de Oliveira Neto, “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice,” in Proc. FSE, 2024

2024
[4]

Generative AI and the Transformation of Software Development Practices,

V. Mastropaolo, N. Gu, and L. Gazzola, “Generative AI and the Transformation of Software Development Practices,” arXiv:2510.10819, Oct. 2025. [Verify ID before submission.]

arXiv 2025
[5]

AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden,

A. Tamburri and M. Palomba, “AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden,” arXiv:2510.10165, Oct. 2024. [Verify ID before submission.]

arXiv 2024
[6]

Development of NASA- TLX (Task Load Index): Results of Empirical and Theoretical Research,

S. G. Hart and L. E. Staveland, “Development of NASA- TLX (Task Load Index): Results of Empirical and Theoretical Research,” Advances in Psychology, vol. 52, pp. 139–183, 1988

1988
[7]

Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025,

R. Huang, A. Reyna, S. Lerner, H. Xia, and B. Hempel, “Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025,” arXiv:2512.14012, Dec. 2024. [Verify ID before submission.]

arXiv 2025
[8]

Autonomous Generative AI Agents: Under Development,

D. Stewart, P. Raman, and F. Muratovic, “Autonomous Generative AI Agents: Under Development,” Deloitte Insights, Dec. 2024

2024

Show all 11 references

[9]

Requirement Tracing using Term Extraction,

N. Al-Saati and R. Abdul-Jaleel, “Requirement Tracing using Term Extraction,” arXiv:1506.08789, 2015

2015 arXiv
[10]

Towards Decoding Developer Cognition in the Age of AI Assistants,

I. Ozkaya, “Towards Decoding Developer Cognition in the Age of AI Assistants,” arXiv:2501.02684, Jan. 2025

2025 arXiv
[11]

Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects,

M. Rasheed, Z. Rasheed, and Q. Umer, “Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects,” arXiv:2406.17910, Jun. 2024. 7 APPENDIX RITM Scoring Rubric Each of the eleven evaluated features was scored on a 0 –2 s...

2024 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.

[1] [1]

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,

S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” arXiv:2302.06590, Feb. 2023

arXiv 2023

[2] [2]

The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot,

B. Cui, E. Yeh, P. Cihon, and M. Demirer, “The Productivity Effects of Generative AI: Evidence from a Field Experiment with GitHub Copilot,” MIT Generative AI Lab, 2024

2024

[3] [3]

Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice,

P. Khojah, M. Mohamad, P. Leitner, and R. G. de Oliveira Neto, “Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice,” in Proc. FSE, 2024

2024

[4] [4]

Generative AI and the Transformation of Software Development Practices,

V. Mastropaolo, N. Gu, and L. Gazzola, “Generative AI and the Transformation of Software Development Practices,” arXiv:2510.10819, Oct. 2025. [Verify ID before submission.]

arXiv 2025

[5] [5]

AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden,

A. Tamburri and M. Palomba, “AI-Assisted Programming Decreases the Productivity of Experienced Developers by Increasing the Technical Debt and Maintenance Burden,” arXiv:2510.10165, Oct. 2024. [Verify ID before submission.]

arXiv 2024

[6] [6]

Development of NASA- TLX (Task Load Index): Results of Empirical and Theoretical Research,

S. G. Hart and L. E. Staveland, “Development of NASA- TLX (Task Load Index): Results of Empirical and Theoretical Research,” Advances in Psychology, vol. 52, pp. 139–183, 1988

1988

[7] [7]

Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025,

R. Huang, A. Reyna, S. Lerner, H. Xia, and B. Hempel, “Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025,” arXiv:2512.14012, Dec. 2024. [Verify ID before submission.]

arXiv 2025

[8] [8]

Autonomous Generative AI Agents: Under Development,

D. Stewart, P. Raman, and F. Muratovic, “Autonomous Generative AI Agents: Under Development,” Deloitte Insights, Dec. 2024

2024