Pith. sign in

REVIEW 4 major objections 5 minor 101 references

Agentic AI workflows produce the most functionally correct code fastest, but human-in-the-loop workflows produce better-organised tests — so the right interaction model depends on the development objective.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:49 UTC pith:ED6UTLQE

load-bearing objection Useful exploratory four-model TDD comparison, but agentic advantages are confounded by newer LLMs; treat as hypothesis-generating. the 4 major comments →

arxiv 2607.22406 v1 pith:ED6UTLQE submitted 2026-07-24 cs.SE

Vibe Coding: An Experiment with Test-Driven Development

Moritz Mock , Barbara Russo This is my paper
classification cs.SE
keywords vibe codingtest-driven developmentconversational LLMshuman-AI collaborationagentic codingcode qualitytest qualityprocess efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks how a human should collaborate with a conversational AI through the lens of Test-Driven Development. It builds four interaction models — human-only, human-AI pair, fully automated single-model, and autonomous multi-agent — and runs the same two Python tasks through all of them. The main claim is that no interaction model dominates: the agentic workflow is the fastest and scores highest on objective correctness, while the collaborative workflow yields test suites closest to what an expert human would write. A sympathetic reader should care because this reframes the choice as one of development objective — speed and functional correctness versus test quality, structure, and control — rather than a single 'best' way to use AI in coding. The evidence is exploratory: 16 professionals in a controlled pre-experiment for the human workflows, plus repeated automated runs for the autonomous ones.

Core claim

On the paper's own terms, the discovery is that the interaction model systematically changes both the code and the process. The agentic runs produced solutions with the highest baseline-test pass rate (mean 94.58%) and the shortest completion time, but with higher McCabe complexity and lower branch coverage relative to the baseline suite, which the authors attribute to implementation decisions the agents made beyond the functional specification. The collaborative runs produced test suites whose structure (number of methods, assertions per method) resembled the baseline suite more closely, and human test suites achieved higher statement and branch coverage than agentic-generated ones. Process

What carries the argument

The central object is a family of four TDD workflows — solo, collaborative, fully automated, and agentic — implemented with structured prompts built from documented prompt patterns (game play, persona, template, context manager, question refinement) and Python scripts. The argument is carried by the evaluation framework: a baseline test suite generated from the feature specification by category-partition testing, and a metric set covering production code (test pass rate, McCabe complexity, statement/branch coverage), test code (assertions per method, method count, coverage), and process (iterations, time). Non-parametric Kruskal-Wallis screening and Mann-Whitney pairwise tests with Holm-Bonf

Load-bearing premise

The paper assumes the observed differences between workflows are caused by the interaction model rather than by the different AI models behind them: the agentic runs used newer, stronger models than the API-based workflows, so if model capability drives most of the gap, the workflow comparison collapses.

What would settle it

A controlled replication that keeps the same underlying model (same version and settings) across all four workflows on the same tasks. If the agentic workflow no longer leads in test-pass rate and time, the ranking is a model-capability artefact. A complementary check: inspect the uncovered branches in agentic output for defect density; if those extra branches contain more bugs than human-written branches, the 'untested decision points' are a real reliability cost, not just added complexity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the ranking is correct, development teams that prioritise shipping speed and functional correctness should look first at agentic workflows.
  • Teams that prioritise maintainability and thorough tests should keep a human in the loop, even though it is slower.
  • Prompt-driven automation with a single model is faster than human collaboration but does not match agentic correctness or human test quality.
  • Agentic output should be treated as carrying spec-extraneous decision points that need explicit review or extra tests, since uncovered branches were a measurable feature of those solutions.
  • Collaborative tooling needs better failure handling: with models repeatedly failing an assertion, returning control to the user produced frequent, costly rework.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The agentic advantage may be partly a model-generation effect, because the agentic runs used newer, stronger models while the non-agentic automated runs used an older API model; a same-model replication is needed to attribute the gap to the workflow itself.
  • Inference: The 'untested decision points' result suggests a concrete engineering response: run branch/statement coverage analysis on agentic output and flag any branch not reachable from the spec before merging.
  • Inference: The assertion-rework pattern implies that the limiting step in collaborative vibe coding is not code generation but test-spec negotiation; tools that suggest prompt revisions after repeated failures could cut the observed overhead.
  • Inference: With small, well-specified tasks, agentic platforms look strong; the open question is whether the speed advantage survives on larger codebases where integration, architecture, and long-range consistency dominate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes four interaction models for CLLM-assisted Test-Driven Development (solo, collaborative, fully-automated, agentic), implements concrete workflows for each, and compares them on two small Python tasks. A static-group pre-experimental study with 16 professionals compares solo and collaborative workflows; the fully-automated and agentic workflows are executed repeatedly without human participation. The headline findings are that agentic workflows produce functionally correct production code fastest, while human-in-the-loop collaborative workflows yield better-organized test suites. The paper also contributes a prompt-pattern library, an evaluation framework, and a replication package.

Significance. If the findings held, this would be a useful empirical contribution to the rapidly growing literature on LLM-based software development. The paper's strengths include a reproducibility-conscious design (open replication package, Google Colab tooling), a baseline test suite derived systematically via category-partition testing, and a combination of quantitative and qualitative analysis. However, the central quantitative comparison is undermined by a cross-condition confound between the interaction model and the underlying LLM generation capability: agentic runs used GPT-5/Claude-4, while the collaborative and fully-automated workflows used GPT-3.5-Turbo. The abstract's unconditional phrasing ('agentic workflows are best suited...') is not supported by the design. The result should be reframed as a conditional, exploratory finding. The paper is a candidate for major revision, not rejection, because the human-only comparison and the qualitative analysis remain of interest.

major comments (4)
  1. [§4.2.1, §4.2.2, Tables 9, 12, 13] The agentic condition confounds interaction model with model generation capability. Collaborative and fully-automated workflows used OpenAI's GPT-3.5 Turbo through its API (§4.2.1), whereas the agentic workflow used ChatGPT-5 and Claude Sonnet 4 on MetaGPT X (§4.2.2). The headline results — higher TPR (Table 9: agentic>solo p=7.81e-5), lower Time (Table 13: agentic vs all), and lower SC/BC for agentic tests (Table 12) — are therefore attributable to either the interaction model or the newer, stronger LLM. A newer model can plausibly produce correct code in fewer attempts, showing up exactly as high TPR and low time. Section 7's construct-validity discussion acknowledges different evaluation settings but does not identify this cross-condition model inequality as a confound. The conclusion should be conditional: 'agentic platforms with current-generation models outperformed,' not 'agentic
  2. [§4.2.2, §5.3] The agentic workflow does not implement TDD, so the process-efficiency comparisons do not compare like with like. §4.2.2 states that MetaGPT X's SOPs 'do not support a TDD-like interaction with the user; therefore, the solution was provided in a single response,' and iterations were not collected. RQ3 (Table 13) compares Time and # Iterations across workflows, but the agentic condition is a different development process, not a different interaction model within TDD. The abstract's claim that 'agentic workflows are best suited for rapid development' is thus a statement about a different process. This limitation should be stated explicitly, or the efficiency claims should be restricted to the models that actually follow TDD.
  3. [Tables 9–13] The reported U statistics and effect sizes are internally inconsistent with the row labels. For example, Table 9, row 'collaborative>agentic' (TPR) reports U=69.5 with group sizes 7 and 12; treating U as the count of A>B pairs gives Cliff's delta ≈ +0.65, yet the table reports -0.65. Similarly, 'fully-automated>agentic' reports U=36.0 with n1=3 and n2=12, which is the maximum possible U and should yield δ=+1.0, yet δ=-1.00 is reported. The footnote 'negative effect size flips the direction of the test' does not resolve the confusion because p-values are then reported for the flipped direction while the row label remains 'A>B'. This makes it impossible for the reader to determine the actual hypothesis tested in each pairwise comparison. The statistical reporting must be corrected or reorganized.
  4. [Abstract, §5.2, Tables 11–12] The abstract's claim that 'collaborative workflows produce higher-quality, better-organized test suites' is not strongly supported by the statistical evidence. After Holm–Bonferroni correction, no comparison involving the collaborative workflow remains significant in Table 12 (collaborative vs agentic BC p=0.0271, above the 0.025 threshold; SC p=0.0565). Table 11 shows no post-correction significance for TAM/TM. The 'higher-quality' claim rests primarily on qualitative inspection of test organization in §5.2.1. While this is acceptable in an exploratory study, the abstract presents it as a decisive finding. The claim should be qualified, or the qualitative analysis should be presented as the primary support.
minor comments (5)
  1. [§2.1] Typo: 'Antropic' should be 'Anthropic'.
  2. [§4.2.2] Incomplete parenthetical: 'MetaGPT X (MGX 5' is missing the closing parenthesis.
  3. [Table 8] For the TM metric, the reported effect size is -0.04. Epsilon-squared is non-negative by definition; report the absolute value or another appropriate measure.
  4. [§4.2.1] The phrase 'controlled pre-experimental study' is oxymoronic; the static-group comparison is a pre-experimental design, not a controlled experiment. Consider rewording.
  5. [§4.2.2, Table 6] 'ChatGPT 5' is used inconsistently; elsewhere the paper refers to OpenAI's GPT-3.5 Turbo. Please align model names.

Circularity Check

0 steps flagged

No significant circularity: the workflow comparison is empirical and self-contained; baseline tests are derived from specifications, not from workflow outputs, and no fitted parameter is renamed as a prediction.

full rationale

The paper's central claims are empirical comparisons among four interaction workflows, not derivations from a fitted or self-referential model. Functional correctness (TPR) is measured against a baseline test suite constructed manually from the feature specification via category-partition testing (Section 4.1, RQ1.2), independently of any workflow output. MCC, coverage, test-structure, and time metrics are likewise defined from standard measures and collected from logs or runtime. No parameter is fitted to a subset of data and then 'predicted' on adjacent data; the agentic and fully-automated workflows are repeated executions, and the human conditions are a controlled pre-experiment. The only self-citations (the authors' preliminary feasibility study and replication package) are descriptive and are not used as evidence for any claimed result. The main methodological weakness, that the agentic condition used ChatGPT-5/Claude Sonnet 4 on MetaGPT X while the collaborative/fully-automated conditions used GPT-3.5 Turbo (Section 4.2), is a construct-validity or confounding threat, not a circularity: it does not make any measured outcome equal to an input by construction. The derivation chain is therefore self-contained; no circular step can be exhibited.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

No new physical entities; the study's inputs are workflow designs, prompts, and evaluation metrics. The max-repetition threshold is the only hand-set numeric constant.

free parameters (1)
  • Max repetitions per iteration = 5
    Ad-hoc cap on re-prompting the CLLM within one TDD iteration (§4.3); affects iteration counts and the definition of 'assertion rework'.
axioms (5)
  • domain assumption TDD is a representative 'highly intensive and demanding' development process for extrapolation to other processes
    Defined in §4 opening; used to justify choosing TDD as the experimental context.
  • domain assumption GPT-3.5 Turbo is an adequate CLLM for the collaborative and fully-automated workflows, and comparing it with GPT-5/Claude-4 in the agentic workflow is meaningful
    §4.2 states GPT-3.5 Turbo for the former and GPT-5/Claude-4 for the agentic runs; this assumption is the weakest link.
  • domain assumption Baseline tests derived from category-partition testing constitute a valid correctness oracle
    §4.1 RQ1.2; all TPR/SCB/BCB values depend on this oracle.
  • domain assumption Static-Group Comparison permits causal attribution of treatment X (collaborative workflow)
    §4.2.1 design lacks a pretest; group equivalence is not established, only random assignment is claimed.
  • domain assumption Self-reported completion time is an accurate measure of process efficiency
    RQ3, Table 4: Time is self-reported by participants in solo/collaborative workflows, with participants P6/P8 reporting >40 min despite the 40-min limit.

pith-pipeline@v1.3.0-alltime-deepseek · 33481 in / 10676 out tokens · 111318 ms · 2026-08-01T04:49:36.404527+00:00 · methodology

0 comments
read the original abstract

Context: Conversational Large Language Models (CLLMs) can automatically generate code by collaborating with users through natural language. However, poor collaboration can lead to poor quality output. Objective: This exploratory study aims to investigate how humans and CLLMs can collaborate as peers through vibe coding, an approach that integrates principles from prompt engineering, agile design, and human-AI co-creation to enhance collaboration. Method: We designed four interaction models representing different collaboration patterns in the software development process: the solo model (human-only development), the collaborative model (human-CLLM collaboration), the fully automated model (development autonomously performed by a CLLM), and the agentic model (development autonomously performed by the MetaGPT~X platform). Based on these models, we implemented corresponding Test-Driven Development (TDD) workflows using structured prompts and Python scripts. We then conducted a controlled pre-experimental study with TDD professionals to compare the solo and collaborative workflows. In addition, we performed repeated exploratory executions of fully automated and agentic workflows on the same development tasks to obtain complementary evidence. Results: Our findings suggest that the choice of interaction model should depend on the development objective. Agentic workflows are best suited for rapid development and functionally correct production code but may introduce additional implementation complexity. However, they may also introduce additional implementation decisions that are not explicitly required by the functional specifications, resulting in untested decision points. In contrast, collaborative workflows produce higher-quality, better-organized test suites. Conclusions: Our work explored how...

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

101 extracted references · 6 canonical work pages

  1. [1]

    Empirical Software Engineering , number =

    Baltes, Sebastian and Ralph, Paul , url =. Empirical Software Engineering , number =

  2. [2]

    The goal question metric approach , author=

  3. [3]

    2022 , publisher=

    Test driven development: By example , author=. 2022 , publisher=

    2022

  4. [4]

    2023 , publisher=

    Deep learning: Foundations and concepts , author=. 2023 , publisher=

    2023

  5. [5]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and W...

    2020

  6. [6]

    Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace , year=

    Butler, Jenna and Suh, Jina and Haniyur, Sankeerti and Hadley, Constance , booktitle=. Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace , year=

  7. [7]

    2015 , publisher=

    Experimental and quasi-experimental designs for research , author=. 2015 , publisher=

    2015

  8. [8]

    2011 , address =

    Causevic, Adnan and Sundmark, Daniel and Punnekkat, Sasikumar , booktitle =. 2011 , address =

    2011

  9. [9]

    2020 , organization =

    Ghafari, Mohammad and Gross, Timm and Fucci, Davide and Felderer, Michael , booktitle =. 2020 , organization =

    2020

  10. [10]

    and Li, Tianle and Li, Dacheng and Zhu, Banghua and Zhang, Hao and Jordan, Michael I

    Chiang, Wei-Lin and Zheng, Lianmin and Sheng, Ying and Angelopoulos, Anastasios N. and Li, Tianle and Li, Dacheng and Zhu, Banghua and Zhang, Hao and Jordan, Michael I. and Gonzalez, Joseph E. and Stoica, Ion , title =. Proceedings of the 41st International Conference on Machine Learning , url=. 2024 , publisher =

    2024

  11. [11]

    Devlin, Jacob and Chang, Ming Wei and Lee, Kenton and Toutanova, Kristina , booktitle =

  12. [12]

    IEEE Software , number =

    Ebert, Christof and Louridas, Panos , url =. IEEE Software , number =

  13. [13]

    2023 , note=

    ERCEA , title=. 2023 , note=

    2023

  14. [14]

    , journal=

    Fakhoury, Sarah and Naik, Aaditya and Sakkas, Georgios and Chakraborty, Saikat and Lahiri, Shuvendu K. , journal=. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation , year=

  15. [15]

    , booktitle =

    Fan, Angela and Gokkaya, Beliz and Harman, Mark and Lyubarskiy, Mitya and Sengupta, Shubho and Yoo, Shin and Zhang, Jie M. , booktitle =. 2023 , address =

    2023

  16. [16]

    IEEE Transactions on Software Engineering , number =

    Fucci, Davide and Erdogmus, Hakan and Turhan, Burak and Oivo, Markku and Juristo, Natalia , url =. IEEE Transactions on Software Engineering , number =

  17. [17]

    Enhancing Code Generation for Low-Resource Languages: No Silver Bullet , year=

    Giagnorio, Alessandro and Martin-Lopez, Alberto and Bavota, Gabriele , booktitle=. Enhancing Code Generation for Low-Resource Languages: No Silver Bullet , year=

  18. [18]

    2016 , note =

    Ian Goodfellow and Yoshua Bengio and Aaron Courville , title=. 2016 , note =

    2016

  19. [19]

    IEEE Software , number =

    Karac, Itir and Turhan, Burak , url =. IEEE Software , number =

  20. [20]

    Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering , pages =

    Li, Junjie and Sangalay, Aseem and Cheng, Cheng and Tian, Yuan and Yang, Jinqiu , title =. Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering , pages =. 2024 , isbn =

    2024

  21. [21]

    J. T. Liang and C. Yang and B. A. Myers , booktitle =. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges , year =

  22. [22]

    2024 , address =

    Liu, Zhe and Chen, Chunyang and Wang, Junjie and Chen, Mengzhuo and Wu, Boyu and Che, Xing and Wang, Dandan and Wang, Qing , booktitle =. 2024 , address =

    2024

  23. [23]

    IEEE Transactions on Software Engineering , number =

    Liu, Zhijie and Tang, Yutian and Luo, Xiapu and Zhou, Yuming and Zhang, Liang Feng , url =. IEEE Transactions on Software Engineering , number =

  24. [24]

    ACM Comput

    Liu, Pengfei and Yuan, Weizhe and Fu, Jinlan and Jiang, Zhengbao and Hayashi, Hiroaki and Neubig, Graham , title =. ACM Comput. Surv. , month = jan, articleno =. 2023 , issue_date =

    2023

  25. [25]

    2024 , booktitle =

    Mastropaolo, Antonio and Ciniselli, Matteo and Pascarella, Luca and Tufano, Rosalia and Aghajani, Emad and Bavota, Gabriele , url =. 2024 , booktitle =

    2024

  26. [26]

    ACM Computing Surveys , month =

    Min, Bonan and Ross, Hayley and Sulem, Elior and Veyseh, Amir Pouran Ben and Nguyen, Thien Huu and Sainz, Oscar and Agirre, Eneko and Heintz, Ilana and Roth, Dan , url =. ACM Computing Surveys , month =

  27. [27]

    and Bui, Nghi D

    Nguyen, Minh Huynh and Phan Chau, Thang and Nguyen, Phong X. and Bui, Nghi D. Q. , booktitle=. AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology , year=

  28. [28]

    2025 , issue_date =

    Zhang, Sai and Xing, Zhenchang and Guo, Ronghui and Xu, Fangzhou and Chen, Lei and Zhang, Zhaoyuan and Zhang, Xiaowang and Feng, Zhiyong and Zhuang, Zhiqiang , title =. 2025 , issue_date =. doi:10.1145/3702987 , journal =

    work page doi:10.1145/3702987 2025

  29. [29]

    , journal=

    Shin, Yonghee and Meneely, Andrew and Williams, Laurie and Osborne, Jason A. , journal=. Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities , year=

  30. [30]

    2026 , eprint=

    Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent , author=. 2026 , eprint=

    2026

  31. [31]

    Tutorials in quantitative Methods for Psychology , volume=

    The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution , author=. Tutorials in quantitative Methods for Psychology , volume=

  32. [32]

    Wasserstein and Allen L

    Ronald L. Wasserstein and Allen L. Schirm and Nicole A. Lazar , title =. The American Statistician , volume =. 2019 , publisher =. doi:10.1080/00031305.2019.1583913 , URL =

    Pith/arXiv arXiv 2019

  33. [33]

    H. B. Mann and D. R. Whitney , journal =. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , urldate =

  34. [34]

    Bird, Christian and Ford, Denae and Zimmermann, Thomas and Forsgren, Nicole and Kalliamvakou, Eirini and Lowdermilk, Travis and Gazit, Idan , title =. Commun. ACM , month = may, pages =. 2023 , issue_date =. doi:10.1145/3589996 , abstract =

    doi:10.1145/3589996 2023

  35. [35]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    MaintainCoder: Maintainable Code Generation Under Dynamic Requirements , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  36. [36]

    Individual Comparisons by Ranking Methods

    Wilcoxon, Frank. Individual Comparisons by Ranking Methods. Breakthroughs in Statistics: Methodology and Distribution. 1992. doi:10.1007/978-1-4612-4380-9_16

    doi:10.1007/978-1-4612-4380-9_16 1992

  37. [37]

    Generative AI for Test Driven Development: Preliminary Results

    Mock, Moritz and Melegati, Jorge and Russo, Barbara. Generative AI for Test Driven Development: Preliminary Results. Agile Processes in Software Engineering and Extreme Programming -- Workshops. 2025

    2025

  38. [38]

    2025 , url =

    Mock, Moritz and Russo, Barbara , title =. 2025 , url =

    2025

  39. [39]

    2024 , booktitle=

    MetaGPT: Meta programming for a multi-agent collaborative framework , author=. 2024 , booktitle=

    2024

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Camel: Communicative agents for" mind" exploration of large language model society , author=. Advances in Neural Information Processing Systems , volume=. 2023 , address=

    2023

  41. [41]

    2023 , eprint=

    Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents , author=. 2023 , eprint=

    2023

  42. [42]

    Agentic Coding: Fundamentals and Practical Implications of Agentic AI , author=

    Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI , author=. 2025 , eprint=

    2025

  43. [43]

    2024 , isbn =

    Corso, Vincenzo and Mariani, Leonardo and Micucci, Daniela and Riganelli, Oliviero , title =. 2024 , isbn =

    2024

  44. [44]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , series =

    Guo, Qi and Cao, Junming and Xie, Xiaofei and Liu, Shangqing and Li, Xiaohong and Chen, Bihuan and Peng, Xin , title =. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , series =. 2024 , isbn =

    2024

  45. [45]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , series =

    Du, Xueying and Liu, Mingwei and Wang, Kaixin and Wang, Hanlin and Liu, Junwei and Chen, Yixuan and Feng, Jiayi and Sha, Chaofeng and Peng, Xin and Lou, Yiling , title =. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , series =. 2024 , isbn =

    2024

  46. [46]

    Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation , url =

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and ZHANG, LINGMING , booktitle =. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation , url =

  47. [47]

    2021 , eprint=

    Program Synthesis with Large Language Models , author=. 2021 , eprint=

    2021

  48. [48]

    Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , pages =

    Mathews, Noble Saji and Nagappan, Meiyappan , title =. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , pages =. 2024 , isbn =

    2024

  49. [49]

    2023 , publisher =

    Vaidya, Jaideep and Asif, Hafiz , title =. 2023 , publisher =. doi:10.1109/MSPEC.2023.10177044 , journal =

    arXiv 2023

  50. [50]

    2026 , isbn =

    Tholander, Jakob and Jonsson, Martin , title =. 2026 , isbn =. doi:10.1145/3772318.3791847 , booktitle =

    arXiv 2026

  51. [51]

    ArXiv , year=

    On Faithfulness and Factuality in Abstractive Summarization , author=. ArXiv , year=

  52. [52]

    Ostrand, T. J. and Balcer, M. J. , title =. Commun. ACM , month = jun, pages =. 1988 , issue_date =. doi:10.1145/62959.62964 , abstract =

    arXiv 1988

  53. [53]

    Exploratory Test-Driven Development Study with ChatGPT in Different Scenarios

    Pancher, Juliano Cesar and Melegati, Jorge and Guerra, Eduardo Martins. Exploratory Test-Driven Development Study with ChatGPT in Different Scenarios. Agile Processes in Software Engineering and Extreme Programming. 2025

    2025

  54. [54]

    2023 , eprint=

    The Impact of AI on Developer Productivity: Evidence from GitHub Copilot , author=. 2023 , eprint=

    2023

  55. [55]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  56. [56]

    Kruskal and W

    William H. Kruskal and W. Allen Wallis , title =. Journal of the American Statistical Association , volume =. 1952 , publisher =. doi:10.1080/01621459.1952.10483441 , URL =

    arXiv 1952

  57. [57]

    and Harman, Mark and Wang, Meng , title =

    Ouyang, Shuyin and Zhang, Jie M. and Harman, Mark and Wang, Meng , title =. 2025 , issue_date =. doi:10.1145/3697010 , journal =

    doi:10.1145/3697010 2025

  58. [58]

    Proceedings of the 1st International Workshop on Large Language Models for Code , pages =

    Piya, Sanyogita and Sullivan, Allison , title =. Proceedings of the 1st International Workshop on Large Language Models for Code , pages =. 2024 , isbn =

    2024

  59. [59]

    2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) , address=

    Piya, Sanyogita and Samadi, Anahita and Sullivan, Allison , title=. 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) , address=. 2025 , volume=

    2025

  60. [60]

    IT Professional , month =

    Sasaki, Yuya and Washizaki, Hironori and Li, Jialong and Yoshioka, Nobukazu and Ubayashi, Naoyasu and Fukazawa, Yoshiaki , url =. IT Professional , month =

  61. [61]

    2024 , eprint=

    LLMs are Imperfect, Then What? An Empirical Study on LLM Failures in Software Engineering , author=. 2024 , eprint=

    2024

  62. [62]

    Tufano, Michele and Drain, Dawn and Svyatkovskiy, Alexey and Sundaresan, Neel , booktitle =

  63. [63]

    Transforming Software Development with Generative AI: Empirical Insights on Collaboration and Workflow

    Ulfsnes, Rasmus and Moe, Nils Brede and Stray, Viktoria and Skarpen, Marianne. Transforming Software Development with Generative AI: Empirical Insights on Collaboration and Workflow. Generative AI for Effective Software Development. 2024

    2024

  64. [64]

    Attention is All you Need , volume =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , volume =

  65. [65]

    Software Testing With Large Language Models: Survey, Landscape, and Vision , year=

    Wang, Junjie and Huang, Yuchao and Chen, Chunyang and Liu, Zhe and Wang, Song and Wang, Qing , journal=. Software Testing With Large Language Models: Survey, Landscape, and Vision , year=

  66. [66]

    ACM Transactions on Software Engineering and Methodology , month =

    Weyssow, Martin and Zhou, Xin and Kim, Kisub and Lo, David and Sahraoui, Houari , url =. ACM Transactions on Software Engineering and Methodology , month =. 2308.10462 , issn =

    Pith/arXiv arXiv

  67. [67]

    , title =

    White, Jules and Fu, Quchen and Hays, Sam and Sandborn, Michael and Olea, Carlos and Gilbert, Henry and Elnashar, Ashraf and Spencer-Smith, Jesse and Schmidt, Douglas C. , title =. Proceedings of the 30th Conference on Pattern Languages of Programs , articleno =. 2023 , isbn =

    2023

  68. [68]

    ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design

    White, Jules and Hays, Sam and Fu, Quchen and Spencer-Smith, Jesse and Schmidt, Douglas C. ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design. Generative AI for Effective Software Development. 2024

    2024

  69. [69]

    o st, Martin and Ohlsson, Magnus C. and Regnell, Bj \

    Wohlin, Claes and Runeson, Per and H \"o st, Martin and Ohlsson, Magnus C. and Regnell, Bj \"o rn and Wessl \'e n, Anders. Empirical Strategies. Springer Nature Link. 2012

    2012

  70. [70]

    ACM Trans

    Yang, Jingfeng and Jin, Hongye and Tang, Ruixiang and Han, Xiaotian and Feng, Qizhang and Jiang, Haoming and Zhong, Shaochen and Yin, Bing and Hu, Xia , title =. ACM Trans. Knowl. Discov. Data , month =. 2024 , issue_date =

    2024

  71. [71]

    Students , author=

    Further Evidence on a Controversial Topic about Human-Based Experiments: Professionals vs. Students , author=. 2025 , eprint=

    2025

  72. [72]

    Are Students Representatives of Professionals in Software Engineering Experiments? , year=

    Salman, Iflaah and Misirli, Ayse Tosun and Juristo, Natalia , booktitle=. Are Students Representatives of Professionals in Software Engineering Experiments? , year=

  73. [73]

    Empirical Softw

    Kitchenham, Barbara and Madeyski, Lech , title =. Empirical Softw. Engg. , month = aug, numpages =. 2024 , issue_date =

    2024

  74. [74]

    2024 , note=

    Massimo Del Vecchio and Dario Fabbri and Elena Marchisa , title=. 2024 , note=

    2024

  75. [75]

    How Effective are LLMs for Data Science Coding? A Controlled Experiment , year=

    Nascimento, Nathalia and Guimaraes, Everton and Chintakunta, Sai Sanjna and Boominathan, Santhosh Anitha , booktitle=. How Effective are LLMs for Data Science Coding? A Controlled Experiment , year=

  76. [76]

    Evaluation of Generative AI Models in Python Code Generation: A Comparative Study , year=

    Palla, Dominik and Slaby, Antonin , journal=. Evaluation of Generative AI Models in Python Code Generation: A Comparative Study , year=

  77. [77]

    Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code , year =

    Shin, Jiho and Tang, Clark and Mohati, Tahmineh and Nayebi, Maleknaz and Wang, Song and Hemmati, Hadi , booktitle =. Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code , year =

  78. [78]

    Empirical Softw

    Mailach, Alina and Gorgosch, Dominik and Siegmund, Norbert and Siegmund, Janet , title =. Empirical Softw. Engg. , month = nov, numpages =. 2024 , issue_date =

    2024

  79. [79]

    and Tahir, A

    Fawzy, A. and Tahir, A. and Blincoe, K. , year=. Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook - a Grey Literature Review , url =

  80. [80]

    and Hill, B.M

    Champion, K. and Hill, B.M. , title=. Empirical Software Engineering , Volume=. 2025 , url =

    2025

Showing first 80 references.