Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Exploring the Challenges and Opportunities of AI-assisted Codebase Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Codebase AI assistants disappoint users: a 16-developer study finds low satisfaction and six recurring failure modes.

desk verdict First real user study of codebase-level AI assistants, with a genuinely useful challenge taxonomy—but the abstract's 77% dissatisfaction figure doesn't reproduce from the paper's own table. read the letter →

arxiv 2508.07966 v1 pith:JFG4V6NI submitted 2025-08-11 cs.SE cs.AI

Philipp Eibl , Sadra Sabouri , Souti Chattopadhyay This is my paper
classification cs.SEcs.AI
keywords codebaseAIassistantscodegenerationuserstudydevelopersatisfactionpromptingbehaviorLLMusabilitychallenges
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies a new class of AI coding tools that generate entire codebases from a natural-language prompt, rather than suggesting snippets. The authors ran a counterbalanced user study with 16 student and professional developers using either GitHub Copilot or GPT-Engineer on create and edit tasks, then interviewed them. They found that users varied widely in what they put in prompts, often omitting requirements they had stated aloud, and rated the resulting codebases low on satisfaction (mean 2.8, median 3 out of 5). Dissatisfaction centered on missing functionality, poor code quality, and communication problems. The paper derives six usability challenges and five work-flow adoption barriers, then compares them with the advertised capabilities of 21 commercial codebase assistants to point toward design changes.

What carries the argument

The central empirical mechanism is the think-aloud user study with counterbalanced task assignment, where 'prompt' refers to the natural-language specification the user writes that serves as the sole interface between developer and codebase assistant. The paper's argument runs through the gap between what participants verbalized as requirements before writing a prompt and what they actually typed: in 20 of 48 tasks the written prompt omitted at least one verbally stated requirement, in 13 of 16 participants. This omission, together with thematic coding of satisfaction justifications (inter-rater agreement κ=0.87) and a comparative capability survey of 21 CBAs, carries the paper's central cla

What would settle it

Run the same task set with a condition where a neutral observer, who cannot hear the think-aloud, independently lists the requirements a participant intends; compare that list with the written prompt. If the omission rate does not differ from a condition where participants write prompts without verbalizing, the omission finding would be an artifact of the think-aloud method.

Watch

Extended reading notes

Core claim

The paper claims that codebase-level AI assistants (CBAs), despite generating whole repositories from text prompts, frequently fail to meet developer expectations, and that the failure is not merely a model-quality issue but a misalignment between how developers express intent and how CBAs process it. In 48 prompted tasks, participants' satisfaction averaged 2.8 on a 5-point scale; only about half of outputs met expectations. Missing functionality was the most cited dissatisfaction factor (77% of mentions), followed by poor code quality (42%) and communication problems (25%). The authors identify six challenges: missing or blank code, inadequate two-way communication, ignored existing code c

Load-bearing premise

The claim that participants omitted requirements from their prompts assumes that think-aloud verbalizations directly reflect the requirements they intended to include, rather than exploratory thoughts or context they never expected the assistant to act on.

Editorial extensions

If this is right

  • If the central claim is right, CBA quality cannot be improved by model scale alone; the prompt interface and feedback loop must change.
  • Tools should adopt guided or scaffolded prompting, asking clarifying questions before generation instead of expecting a complete prompt up front.
  • CBAs should share their implementation plans, verify outputs against stated requirements, and present changes as diffs or pull requests with explanations, to reduce the loss-of-control barrier.
  • Supporting the user in debugging and repairing partially correct output—rather than only generating—would address the most frequent dissatisfaction factors (missing functionality and poor code quality).
  • Adoption in professional settings will remain limited until legal and privacy concerns, such as code ownership and data confidentiality, are resolved by organizational policy, not just tool features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 20-out-of-48 omission rate may understate real-world omission because the lab setting used pre-written task descriptions; in authentic workflows, requirements are often tacit, so the communication gap could be larger than measured.
  • The positive correlation between imperative prompt tone and satisfaction (r=0.31) suggests a testable extension: CBA prompt templates that nudge users toward directive, checklist-style phrasing could be evaluated as an intervention against omission-driven failures.
  • The paper's challenge taxonomy implies a concrete benchmark design: future CBAs could be scored on whether they detect that a user's spoken requirement is absent from the prompt, rather than only on code correctness.
  • The authors note the study captured only first prompts per task; iterative multi-round prompting, a natural next step, may reveal whether the low satisfaction is a first-attempt artifact or a persistent alignment problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a mixed-methods user study (n=16) in which graduate students and professional developers used either GitHub Copilot or GPT-Engineer to create or edit small codebases while thinking aloud, followed by an interview. The authors analyze prompts, satisfaction ratings, and qualitative feedback to derive six challenges users face with codebase-level AI assistants (CBAs) and five barriers to adoption; they then survey 21 commercial CBAs and map seven capability dimensions onto the identified user needs. The main empirical claims are that satisfaction with generated codebases is low (mean 2.8/5), that functionality problems are the most common dissatisfaction driver, and that inadequate communication and ignored context are prevalent challenges.

Significance. If the findings hold, this is a useful early empirical map of a rapidly growing but understudied class of AI coding tools. The study has notable strengths: a think-aloud protocol, open coding with a reported inter-rater agreement (kappa=0.87), a counterbalanced task design, and a systematic survey of 21 commercial CBAs that yields a concrete capability taxonomy (R-CTX, PL-SHARE, S-VER, PROACT, PR, TEACH, ASK). The qualitative six-challenge/five-barrier framework and the design recommendations are plausible and actionable. However, the quantitative layer of the paper is substantially weaker than the qualitative core: some headline numbers are not reproducible from the reported tables, and several statistical reports are internally inconsistent. These issues would need to be corrected before the paper's quantitative claims can be trusted.

major comments (3)
  1. [Abstract and §V, Table III] The percentages in the abstract (77% functionality, 42% poor code quality, 25% communication issues) are not reproducible from the data presented. Table III reports 83 mention counts across all factors; Functionality is 25/83 ≈ 30% of mentions, not 77%. The body text uses a different metric for unmet functionality: 'In 16 of the 48 tasks (33%)' (§V, Functionality paragraph). No definition of 'instances' is given, and 'poor code quality' and 'communication issues' are not mapped to any Table III category. Because these percentages are the headline quantitative result, the authors should either recompute them with an explicit denominator (mentions, tasks, participants, or negative-only mentions) and a clear mapping to the coded themes, or remove/strongly qualify them. As written, the abstract's central quantitative message is unsupported.
  2. [§V, Table III] There is a direct numeric inconsistency in the reported correlation for 'Met func. req.': Table III lists 0.34* while the text states 'the strongest positive correlation with satisfaction (r = 0.45, p = 0.0015)'. This is not a rounding difference. The authors should correct the value and also define the asterisk in Table III's caption; it appears to denote significance but is never explained.
  3. [§VI, ANOVA and post-hoc tests] The reported omnibus ANOVA result F(1,11)=3.46, p=0.0004 is internally inconsistent: for F(1,11)=3.46 the p-value is approximately 0.09, not 0.0004. Either the test statistic, degrees of freedom, or p-value is misreported. In addition, the post-hoc t-tests appear to be uncorrected for multiple comparisons; with the stated factors and 10 evaluative themes, a single p=0.034 for Correctness would not survive a standard correction. The authors should report the full ANOVA table (including both factors and the interaction), the post-hoc procedure, and adjusted p-values, or drop the claim that students and developers differ significantly.
minor comments (5)
  1. [§V, Executability paragraph and Table III] The text says 'Nine participants expressed that code that is not running...' but Table III lists Executability counts as 9/9/7, i.e., 7 participants. Please align the text with the table.
  2. [§IV, RQ1a prompt types] The taxonomy labels two groups as 'Type II' (Behavior-focused and Functionality-focused). The third should be 'Type III' to avoid confusion.
  3. [§VI, Inadequate Communication] The claim that 20/48 tasks omitted verbally stated requirements assumes that think-aloud verbalizations are direct evidence of the requirements participants intended to include in the prompt. Some verbalizations may be exploratory thoughts or context-setting rather than intended prompt content. The authors should acknowledge this assumption and, if possible, triangulate with participants' post-hoc evaluations, which they already partially do.
  4. [Table IV and References] The /terminal row lists GPT-Engineer with citation [36], but reference [36] is Lovable; the GPT-Engineer reference is [5]. Please correct the citation mapping.
  5. [§V, chi-square test] The chi-square test on satisfaction distributions across task types and CBAs is performed on a very small sample (n=48 ratings, with expected counts likely below 5). The authors should either justify the test's validity more carefully or present it as purely descriptive.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: user study derives challenges inductively; the abstract's 77% figure is a reporting inconsistency, not a circular derivation.

full rationale

This paper reports an observational user study. Its contributions (six challenges, five adoption barriers, satisfaction scores) are obtained by thematic coding of transcripts, task logs, and ratings; they are not fitted parameters, equations, or predictions derived from other fitted quantities. There is no step in which an input is defined in terms of an output, no fitted value renamed as a prediction, and no uniqueness theorem or ansatz imported via citation. The authors cite their own prior work ([18], [78]), but those citations support peripheral framing (trust dynamics, double-sided clarification, study design precedents) rather than the load-bearing empirical derivation; the challenges and barriers stand on the reported data. The only notable quantitative concern is that the abstract's '77% of instances' for functionality dissatisfaction cannot be reproduced from Table III (Functionality has 25 mentions out of 83 total, about 30%), and the 42% and 25% figures similarly lack defined denominators. That is a reproducibility/reporting issue, not circularity, because those figures are not derived from Table III by construction. No self-consistency or derivation chain reduces the paper's findings to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The central claims rest on the validity of qualitative coding, the representativeness of the small sample and author-designed tasks, and the correctness of reported statistics. No free parameters are fitted; the paper's 'quantitative' results are observational. The main risk is that the reported correlations and p-values contain internal inconsistencies that could mislead readers.

assumptions (5)
  • domain assumption Think-aloud verbalizations correspond to intended task requirements.
    Section VI uses the contrast between what participants said aloud and what they wrote to conclude that 20/48 prompts omitted verbally stated requirements. This assumes users' spoken words are an accurate proxy for their intended prompt content.
  • domain assumption Open-coding themes with negotiated agreement and kappa=0.87 are valid.
    Section III-C describes inductive coding and inter-rater reliability; the resulting categories (satisfaction factors, challenges, barriers) are treated as objective patterns.
  • domain assumption Small convenience sample is sufficient for theme saturation.
    Section IX acknowledges the small sample and notes that findings are 'consistent with prior work', relying on qualitative richness rather than statistical power.
  • domain assumption The author-designed tasks and codebases are representative of real CBA use.
    Section III-B states tasks were crafted by the authors; Section IX says artificial codebases 'do not fully capture developers' naturalistic interactions', yet the study draws general conclusions from them.
  • domain assumption Statistical test assumptions are met (independence, expected cell counts).
    Section V applies chi-square and Section VI applies ANOVA to the small task-level data; the paper cites [27] for chi-square sample-size requirements, but does not verify all assumptions, and one reported p-value is implausible.
invented entities (2)
  • CBA capability dimensions (R-CTX, PL-SHARE, S-VER, PROACT, PR, TEACH, ASK)
    purpose: Organize the feature survey of 21 tools and map to user challenges.
    The dimensions are inductively derived from tool advertisements by the authors; they are not a validated standard taxonomy.
  • Six CBA challenges and five adoption barriers
    purpose: Conceptual categories for observed usability problems and workflow deterrents.
    Emerged from qualitative coding of participant data; they summarize the data but are not independently tested hypotheses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Challenges and Opportunities of AI-assisted Codebase Generation." pith.science (2026). https://pith.science/paper/JFG4V6NI

@misc{pith2026250807966,
  author       = {Pith},
  title        = {Pith review of: Exploring the Challenges and Opportunities of AI-assisted Codebase Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFG4V6NI}},
  note         = {Machine review of arXiv:2508.07966}
}
read the original abstract

Recent AI code assistants have significantly improved their ability to process more complex contexts and generate entire codebases based on a textual description, compared to the popular snippet-level generation. These codebase AI assistants (CBAs) can also extend or adapt codebases, allowing users to focus on higher-level design and deployment decisions. While prior work has extensively studied the impact of snippet-level code generation, this new class of codebase generation models is relatively unexplored. Despite initial anecdotal reports of excitement about these agents, they remain less frequently adopted compared to snippet-level code assistants. To utilize CBAs better, we need to understand how developers interact with CBAs, and how and why CBAs fall short of developers' needs. In this paper, we explored these gaps through a counterbalanced user study and interview with (n = 16) students and developers working on coding tasks with CBAs. We found that participants varied the information in their prompts, like problem description (48% of prompts), required functionality (98% of prompts), code structure (48% of prompts), and their prompt writing process. Despite various strategies, the overall satisfaction score with generated codebases remained low (mean = 2.8, median = 3, on a scale of one to five). Participants mentioned functionality as the most common factor for dissatisfaction (77% of instances), alongside poor code quality (42% of instances) and communication issues (25% of instances). We delve deeper into participants' dissatisfaction to identify six underlying challenges that participants faced when using CBAs, and extracted five barriers to incorporating CBAs into their workflows. Finally, we surveyed 21 commercial CBAs to compare their capabilities with participant challenges and present design opportunities for more efficient and useful CBAs.

Figures

Figures reproduced from arXiv: 2508.07966 by the authors.

Figure 1
Figure 1. An overview of our methodology. Each participant was assigned to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of satisfaction scores by task type, task length, and CBA. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Security Considerations for Multi-agent Systems

    cs.CR 2026-03 unverdicted novelty 6.0 of 10

    No existing AI security framework covers a majority of the 193 identified multi-agent system threats in any category, with OWASP Agentic Security Initiative achieving the highest overall coverage at 65.3%.

  2. Towards More Empathic Programming Environments: An Experimental Empathic AI-Enhanced IDE

    cs.SE 2026-04 unverdicted novelty 4.0 of 10

    Pilot study of an empathic AI IDE found no significant gains in learning or workload over standard AI tools, with only greater perceived help in error correction.

Reference graph

Works this paper leans on

82 extracted references · 52 canonical work pages · cited by 2 Pith papers

  1. [1]

    Overflow, Jun 2022

    S. Overflow, Jun 2022. [Online]. Available: https://survey.stackoverflow.co/2023/

    2022

  2. [2]

    [Online]

    GitHub, Jun 2022. [Online]. Available: https://github.com/features/copilot

    2022

  3. [3]

    Longrope: Extending llm context window beyond 2 million tokens,

    Y . Ding, L. L. Zhang, C. Zhang, Y . Xu, N. Shang, J. Xu, F. Yang, and M. Yang, “Longrope: Extending llm context window beyond 2 million tokens,” arXiv preprint arXiv:2402.13753 , 2024

    arXiv 2024

  4. [4]

    Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,

    Y . Zhu, X. Wang, J. Chen, S. Qiao, Y . Ou, Y . Yao, S. Deng, H. Chen, and N. Zhang, “Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,” World Wide Web, vol. 27, no. 5, p. 58, 2024

    work page 2024

  5. [5]

    Gpt engineer: Specify what you want, get code,

    A. Osika, “Gpt engineer: Specify what you want, get code,” Jun 2023. [Online]. Available: https://github.com/AntonOsika/gpt-engineer

    work page 2023

  6. [6]

    Devin: First ai software engineer,

    C. Labs, “Devin: First ai software engineer,” 2024. [Online]. Available: https://www.cognition-labs.com/

    work page 2024

  7. [7]

    Productivity assessment of neural code completion,

    A. Ziegler, E. Kalliamvakou, S. Simister, G. Sittampalam, A. Li, A. Rice, D. Rifkin, and E. Aftandilian, “Productivity assessment of neural code completion,” 2022. [Online]. Available: https://arxiv.org/abs/2205.06537

    arXiv 2022

  8. [8]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,”

    work page

Show all 82 references
  1. [9]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, pp. 21 558–21 572, 2023

    2023

  2. [10]

    Grounded Copilot: How Programmers Interact with Code-Generating Models,

    S. Barke, M. B. James, and N. Polikarpova, “Grounded Copilot: How Programmers Interact with Code-Generating Models,” Oct. 2022, arXiv:2206.15000 [cs]. [Online]. Available: http://arxiv.org/abs/2206.15000

    2022 arXiv

  3. [11]

    How far are ai-powered programming assistants from meeting developers’ needs?

    X. Tan, X. Long, X. Ni, Y . Zhu, J. Jiang, and L. Zhang, “How far are ai-powered programming assistants from meeting developers’ needs?”

  4. [12]

    Developer experiences with a contextualized ai coding assistant: Usability, expectations, and outcomes,

    G. Pinto, C. de Souza, T. Rocha, I. Steinmacher, A. de Souza, and E. Monteiro, “Developer experiences with a contextualized ai coding assistant: Usability, expectations, and outcomes,” 2023. [Online]. Available: https://arxiv.org/abs/2311.18452

    2023 arXiv

  5. [13]

    Codea11y: Making ai coding assistants useful for accessible web development,

    P. Mowar, Y .-H. Peng, J. Wu, A. Steinfeld, and J. P. Bigham, “Codea11y: Making ai coding assistants useful for accessible web development,”

  6. [14]

    Evaluating the usability and functionality of intelligent source code completion assistants: A comprehensive review,

    T. Hli ˇs, L. ˇCetina, T. Berani ˇc, and L. Pavli ˇc, “Evaluating the usability and functionality of intelligent source code completion assistants: A comprehensive review,” Applied Sciences , vol. 13, no. 24, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/24/13061

    2023

  7. [15]

    Discovering the syntax and strategies of natural language programming with generative language models,

    E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry, “Discovering the syntax and strategies of natural language programming with generative language models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , s...

    2022

  8. [16]

    Reading between the lines: Modeling user behavior and costs in ai-assisted programming,

    H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz, “Reading between the lines: Modeling user behavior and costs in ai-assisted programming,” in Proceedings of the CHI Conference on Human Factors in Computing Systems , ser. CHI ’24. New York, NY , USA: Association for Computin...

    2024

  9. [17]

    Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,

    A. Sergeyuk, Y . Golubev, T. Bryksin, and I. Ahmed, “Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,” Information and Software Technology, vol. 178, p. 107610, Feb. 2025. [Online]. Available: http://dx.doi.org/10.1016/j.infsof.2024.107610

    2025

  10. [18]

    Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications ,

    S. Sabouri, P. Eibl, X. Zhou, M. Ziyadi, N. Medvidovic, L. Lindemann, and S. Chattopadhyay, “ Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications ,” in 2025 IEEE/ACM 47th Inter- national Conference on Software Engineering (ICSE) . IEEE Computer So...

    2025

  11. [19]

    Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools,

    C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit, “Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools,” Queue, vol. 20, no. 6, p. 35–57, jan 2023. [Online]. Available: https://doi.org/10.1...

    2023 doi

  12. [20]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.00515

    2024 arXiv

  13. [21]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770 , 2023

    2023 arXiv

  14. [22]

    Stall+: Boosting llm-based repository-level code completion with static analysis,

    J. Liu, Y . Chen, M. Liu, X. Peng, and Y . Lou, “Stall+: Boosting llm-based repository-level code completion with static analysis,” arXiv preprint arXiv:2406.10018 , 2024. [Online]. Available: https://arxiv.org/abs/2406.10018

    2024 arXiv

  15. [23]

    Hierarchical context pruning: Optimizing real-world code completion with repository-level pretrained code llms,

    L. Zhang, Y . Li, J. Li, X. Xia, J. Yang, R. Luo, M. Wang, L. Chen, J. Liu, and M. Yang, “Hierarchical context pruning: Optimizing real-world code completion with repository-level pretrained code llms,” arXiv preprint arXiv:2406.18294 , 2024. [Online]. Available: https://arxiv...

    2024 arXiv

  16. [24]

    Repofusion: Training code models to understand your repository,

    D. Shrivastava, D. Kocetkov, H. de Vries, D. Bahdanau, and T. Scholak, “Repofusion: Training code models to understand your repository,” arXiv preprint arXiv:2306.10998, 2023. [Online]. Available: https://arxiv.org/abs/2306.10998

    2023 arXiv

  17. [25]

    Supplementary Material,

    Anonymous, “Supplementary Material,” 9 2024. [Online]. Available: https://figshare.com/s/a13d7a90f13f2fcc2105

    2024

  18. [26]

    Unfamiliar finetuning examples control how language models hallucinate,

    K. Kang, E. Wallace, C. Tomlin, A. Kumar, and S. Levine, “Unfamiliar finetuning examples control how language models hallucinate,” arXiv preprint arXiv:2403.05612, 2024

    2024 arXiv

  19. [27]

    An investigation of the restraints with respect to sample size commonly imposed on the use of the chi-square statistic,

    J. T. Roscoe and J. A. Byars, “An investigation of the restraints with respect to sample size commonly imposed on the use of the chi-square statistic,” Journal of the American Statistical Association , vol. 66, no. 336, pp. 755–759, 1971

    1971

  20. [28]

    Ethical aspects of chatgpt in software engineering research,

    M. A. Akbar, A. A. Khan, and P. Liang, “Ethical aspects of chatgpt in software engineering research,” IEEE Transactions on Artificial Intelligence, 2023

    2023

  21. [29]

    Developer perspectives on licens- ing and copyright issues arising from generative ai for coding,

    T. Stalnaker, N. Wintersgill, O. Chaparro, L. A. Heymann, M. Di Penta, D. M. German, and D. Poshyvanyk, “Developer perspectives on licens- ing and copyright issues arising from generative ai for coding,” arXiv preprint arXiv:2411.10877, 2024

    2024 arXiv

  22. [30]

    [Online]

    GitHub, Jun 2021. [Online]. Available: https://github.com/features/copilot

    2021

  23. [31]

    [Online]

    Cursor, Dec 2024. [Online]. Available: https://docs.cursor.com/chat/agent

    2024

  24. [32]

    [Online]

    Cline, 2024. [Online]. Available: https://docs.cline.bot/

    2024

  25. [33]

    [Online]

    Windsurf, 2024. [Online]. Available: https://windsurf.com/editor

    2024

  26. [34]

    [Online]

    Softgen, 2023. [Online]. Available: https://docs.cline.bot/

    2023

  27. [35]

    [Online]

    Pear, 2025. [Online]. Available: https://trypear.ai/docs

    2025

  28. [36]

    [Online]

    Lovable, Apr 2025. [Online]. Available: https://docs.lovable.dev/introduction

    2025

  29. [37]

    [Online]

    Aider, 2020. [Online]. Available: https://aider.chat/docs/

    2020

  30. [38]

    [Online]

    Codebuff, 2024. [Online]. Available: https://www.codebuff.com/docs/help

    2024

  31. [39]

    Next, Apr 2024

    G. Next, Apr 2024. [Online]. Available: https://githubnext.com/projects/copilot-workspace

    2024

  32. [40]

    [Online]

    Databutton, 2021. [Online]. Available: https://docs.databutton.com/getting-started/meet-the-databutton-ai-agent

    2021

  33. [41]

    [Online]

    Replit, 2016. [Online]. Available: https://docs.replit.com/getting- started/intro-replit

    2016

  34. [42]

    [Online]

    Base44, 2024. [Online]. Available: https://docs.base44.com/

    2024

  35. [43]

    [Online]

    Qodo, 2024. [Online]. Available: https://docs.qodo.ai/qodo- documentation

    2024

  36. [44]

    [Online]

    Srcbook, 2024. [Online]. Available: https://getmocha.com/

    2024

  37. [45]

    [Online]

    Pythagora, Jun 2025. [Online]. Available: https://www.pythagora.ai/docs

    2025

  38. [46]

    [Online]

    Bolt, 2025. [Online]. Available: https://docs.boltai.com/docs

    2025

  39. [47]

    [Online]

    V0, 2023. [Online]. Available: https://v0.dev/docs

    2023

  40. [48]

    [Online]

    Webdraw, Nov 2024. [Online]. Available: https://docs.webdraw.com/introduction

    2024

  41. [49]

    [Online]

    Tempo, 2024. [Online]. Available: https://www.tempo.new/

    2024

  42. [50]

    On the impacts of contexts on repository-level code generation,

    N. L. Hai, D. M. Nguyen, and N. D. Bui, “On the impacts of contexts on repository-level code generation,” arXiv preprint arXiv:2406.11927 , 2024

    2024 arXiv

  43. [51]

    Large language model-aware in-context learning for code generation,

    J. Li, C. Tao, J. Li, G. Li, Z. Jin, H. Zhang, Z. Fang, and F. Liu, “Large language model-aware in-context learning for code generation,” ACM Transactions on Software Engineering and Methodology , 2023

    2023

  44. [52]

    Make your llm fully utilize the context,

    S. An, Z. Ma, Z. Lin, N. Zheng, J.-G. Lou, and W. Chen, “Make your llm fully utilize the context,”Advances in Neural Information Processing Systems, vol. 37, pp. 62 160–62 188, 2024

    2024

  45. [53]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , vol. 2, p. 1, 2023

    2023 arXiv

  46. [54]

    Self-planning code generation with large language models,

    X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 7, pp. 1–30, 2024

    2024

  47. [55]

    A plan-based intelligent assistant that supports the software development,

    K. E. Huff and V . R. Lesser, “A plan-based intelligent assistant that supports the software development,” in Proceedings of the third ACM SIGSOFT/SIGPLAN software engineering symposium on Practical soft- ware development environments, 1988, pp. 97–106

    1988

  48. [56]

    Planning agents,

    J. L. Pollock, “Planning agents,” in Foundations of rational agency . Springer, 1999, pp. 53–79

    1999

  49. [57]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in neural information processing systems , vol. 36, pp. 11 809–11 822, 2023

    2023

  50. [58]

    Generation probabilities are not enough: Improving error highlighting for ai code suggestions,

    H. Vasconcelos, G. Bansal, A. Fourney, Q. V . Liao, and J. W. Vaughan, “Generation probabilities are not enough: Improving error highlighting for ai code suggestions,” in HCAI Workshop at NeurIPS , 2022

    2022

  51. [59]

    The formai dataset: Generative ai in software security through the lens of formal verification,

    N. Tihanyi, T. Bisztray, R. Jain, M. A. Ferrag, L. C. Cordeiro, and V . Mavroeidis, “The formai dataset: Generative ai in software security through the lens of formal verification,” in Proceedings of the 19th International Conference on Predictive Models and Data Analytics in ...

    2023

  52. [60]

    Can llms formally reason as abstract interpreters for program analysis?

    J. L. Mitchell, B. H. Kim, C. Zhou, and C. Wang, “Can llms formally reason as abstract interpreters for program analysis?” arXiv preprint arXiv:2503.12686, 2025

    2025

  53. [61]

    A self-improving coding agent,

    M. Robeyns, M. Szummer, and L. Aitchison, “A self-improving coding agent,” arXiv preprint arXiv:2504.15228 , 2025

    2025 arXiv

  54. [62]

    Towards understanding sycophancy in language models,

    M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston et al., “Towards understanding sycophancy in language models,”arXiv preprint arXiv:2310.13548, 2023

    2023 arXiv

  55. [63]

    Vibe coding: programming through conversa- tion with artificial intelligence,

    A. Sarkar and I. Drosos, “Vibe coding: programming through conversa- tion with artificial intelligence,” arXiv preprint arXiv:2506.23253, 2025

    2025

  56. [64]

    The gotchas of ai coding and vibe coding. it’s all about support and maintenance,

    S. H. Maes, “The gotchas of ai coding and vibe coding. it’s all about support and maintenance,” 2025

    2025

  57. [65]

    Ensuring the maintainability and supportability of

    ——, “Ensuring the maintainability and supportability of” vibe-coded” software systems: A framework for bridging intuition and engineering rigor,” OSF, 2025

    2025

  58. [66]

    Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization,

    Y . Ishibashi and Y . Nishimura, “Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization,” arXiv preprint arXiv:2404.02183 , 2024

    2024 arXiv

  59. [67]

    Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,

    S. Hamer, M. d’Amorim, and L. Williams, “Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,” in 2024 IEEE Security and Privacy Workshops (SPW). IEEE, 2024, pp. 87–94

    2024

  60. [68]

    An insight into the pull requests of github,

    M. M. Rahman and C. K. Roy, “An insight into the pull requests of github,” in Proceedings of the 11th working conference on mining software repositories, 2014, pp. 364–367

    2014

  61. [69]

    Does code quality affect pull request acceptance? an empirical study,

    V . Lenarduzzi, V . Nikkola, N. Saarim ¨aki, and D. Taibi, “Does code quality affect pull request acceptance? an empirical study,” Journal of Systems and Software , vol. 171, p. 110806, 2021

    2021

  62. [70]

    Comparing and merging files with gnu diff and patch,

    D. MacKenzie, P. Eggert, and R. Stallman, “Comparing and merging files with gnu diff and patch,” Network Theory Ltd , vol. 4, pp. 23–25, 2002

    2002

  63. [71]

    Review and analysis of software development team communication research,

    J. F. Defranco and P. A. Laplante, “Review and analysis of software development team communication research,” IEEE Transactions on Professional Communication, vol. 60, no. 2, pp. 165–182, 2017

    2017

  64. [72]

    The effective- ness of pair programming: A meta-analysis,

    J. E. Hannay, T. Dyb ˚a, E. Arisholm, and D. I. Sjøberg, “The effective- ness of pair programming: A meta-analysis,” Information and software technology, vol. 51, no. 7, pp. 1110–1122, 2009

    2009

  65. [73]

    Make your tools sparkle with trust: The picse framework for trust in software tools,

    B. Johnson, C. Bird, D. Ford, N. Forsgren, and T. Zimmermann, “Make your tools sparkle with trust: The picse framework for trust in software tools,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 202...

    2023

  66. [74]

    Clarifygpt: Empowering llm-based code generation with intention clarification,

    F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “Clarifygpt: Empowering llm-based code generation with intention clarification,” arXiv preprint arXiv:2310.10996 , 2023

    2023 arXiv

  67. [75]

    Understanding the hawthorne effect,

    P. Sedgwick and N. Greenwood, “Understanding the hawthorne effect,” Bmj, vol. 351, 2015

    2015

  68. [76]

    Think-aloud protocol,

    R. J ¨a¨askel¨ainen, “Think-aloud protocol,” in Handbook of translation studies: volume 1 . John Benjamins publishing company, 2012, pp. 371–373

    2012

  69. [77]

    Developers’ information seeking in question & answer websites through a gender lens,

    A. Sedhain, V . Diwanji, H. Solomon, S. Leon, and S. K. Kuttal, “Developers’ information seeking in question & answer websites through a gender lens,” Journal of Computer Languages, vol. 79, p. 101267, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pi...

    2024

  70. [78]

    Supporting code comprehension via annotations: Right information at the right time and place,

    M. Adeli, N. Nelson, S. Chattopadhyay, H. Coffey, A. Henley, and A. Sarma, “Supporting code comprehension via annotations: Right information at the right time and place,” in 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) , 2020, pp. 1–10

    2020

  71. [79]

    The patchworks code editor: toward faster navigation with less code arranging and fewer navigation mistakes,

    A. Z. Henley and S. D. Fleming, “The patchworks code editor: toward faster navigation with less code arranging and fewer navigation mistakes,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ser. CHI ’14. New York, NY , USA: Association for Comp...

    2014

  72. [2023]

    Available: https://arxiv.org/abs/2303.17125

    [Online]. Available: https://arxiv.org/abs/2303.17125

    arXiv

  73. [2024]

    Available: https://arxiv.org/abs/2404.12000

    [Online]. Available: https://arxiv.org/abs/2404.12000

    arXiv

  74. [2025]

    Available: https://arxiv.org/abs/2502.10884

    [Online]. Available: https://arxiv.org/abs/2502.10884

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.