Pith. sign in

REVIEW 3 major objections 5 minor 23 references

A systematic review of 47 sources finds that vibe coding is not one-shot prompting but an iterative loop of intent, generation, evaluation, and refinement, with productivity gains that hinge on the surrounding evaluation and governance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Vibe coding evidence from 47 sources describes an intent-driven, iterative evaluation loop whose productivity benefits are conditional on review and validation practices.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Well-documented MLR whose headline "iterative loop" claim is partly a product of the search string and codebook — treat that finding as provisional rather than established. the 3 major comments →

arxiv 2607.21652 v1 pith:BYIXMGYL submitted 2026-07-22 cs.SE

Vibe Coding in Software Development: A Multivocal Literature Review

classification cs.SE
keywords vibe codingmultivocal literature reviewlarge language modelsAI-assisted software developmenthuman-AI collaborationcode generationdeveloper role shiftsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This review synthesizes 28 peer-reviewed and 19 practitioner sources to establish what vibe coding actually is and how it behaves in software development. Its central claim is that vibe coding is consistently described as an iterative generation–evaluation–revision loop, not a one-shot prompt-to-code activity, and that the real bottleneck is the strength of the evaluation loop around generated code rather than generation speed. The review finds that developer work shifts from writing code to specifying intent, supervising output, and validating results, and that reported productivity gains, appearing in 45% of sources, are conditional on expertise, task type, and verification practices. Evidence is strongest for prototyping and UI work and weakest for production, data-intensive, and safety-critical settings, and tool visibility does not imply demonstrated effectiveness. A sympathetic reader should care because this reframes a hype term as an engineering workflow with identifiable controls and testable failure modes.

Core claim

On its own terms, the paper's central discovery is that the scattered literature on vibe coding converges on a single process shape: the developer states intent in natural language, an LLM generates or revises code, the developer evaluates the result by running, inspecting, or testing it, and then refines the next prompt based on observed behavior. Across 43 of 47 sources, validation and evaluation pipelines (72%) and chat-based iterative loops (63%) are the dominant workflow patterns, and across 36 sources the practice is described most often as an intent-driven socio-technical practice (69%) and as natural-language-to-code prompting (67%). The review labels this an iterative control system

What carries the argument

The central object is the vibe coding loop: a feedback-driven cycle of intent specification, LLM generation, developer evaluation, and prompt refinement, with validation and testing inside the loop rather than optional additions. The paper's conceptual framework adds four interacting layers—workflow, role and experience, outcomes, and risk and governance—moderated by usage context and tool ecosystem, and attaches an evidence-strength label (strong, moderate, weak, emerging) to each finding. That combination of loop, layered model, and evidence weighting is what carries the argument.

Load-bearing premise

The conclusion that vibe coding is 'consistently' iterative rests on a search string and eligibility rules that deliberately excluded literature using broader terms such as 'LLM-assisted programming,' 'AI coding assistant,' and 'agentic coding'; if that excluded literature contains many one-shot or pure-delegation accounts, the consistency claim would overstate the evidence.

What would settle it

A concrete falsifier: re-run the review's search with the deliberately excluded terms ('LLM-assisted programming,' 'AI coding assistant,' 'agentic coding') and count how many of the additionally retrieved sources describe vibe-coding-adjacent work as one-shot, fully delegated generation. If a substantial number do, the claim that the literature 'consistently' describes an iterative loop is falsified. A complementary experiment: compare matched teams using vibe coding with and without enforced test-and-review pipelines; if defect rates and maintainability do not differ, the claim that outcomes

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Vibe coding should be managed as an engineering workflow: teams that adopt it without automated tests, review routines, and validation pipelines are likely trading short-term speed for long-term defect and maintenance burden.
  • Reported productivity gains are most credible for prototyping and UI/front-end work; claims of gains in production, data-intensive, or safety-critical settings should not be assumed until studied.
  • Safeguards such as human-in-the-loop review and validation pipelines are consistently recommended, but since their effectiveness is under-tested, they should be treated as hypotheses to evaluate rather than proven controls.
  • Session-level dynamics—momentum breaks, context drift, and repetitive prompt–patch cycles—are real but under-measured, pointing to a need for instrumented long-horizon studies.
  • The review's evidence-strength labels provide a practical map: strong claims are limited to the iterative-loop description and short-term productivity; everything else is moderate, weak, or emerging.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the iterative-control-system framing is right, a testable extension is to instrument real development sessions and check whether teams with stronger evaluation loops (CI gates, test coverage, structured review) show lower escaped-defect rates and better maintainability than teams that iterate without them.
  • The review's exclusion of broader terms like 'LLM-assisted programming' and 'agentic coding' suggests a conservative lower bound: one-shot or pure-delegation characterizations may exist in that excluded literature, so the 'consistently iterative' claim is best read as applying to the vibe-coding-specific corpus.
  • The novice/expert split implies a design consequence: tools and training should target verification skill (how to evaluate and test generated code) rather than only prompt skill, because overtrust is most dangerous for the novice users the practice is said to empower.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript reports a multivocal literature review (MLR) of 47 sources (28 peer-reviewed, 19 grey) on vibe coding, following Garousi et al.'s MLR guidelines. The review poses eight research questions covering definitions, workflows, developer-role shifts, outcomes, risks/safeguards, usage contexts, tools, and open challenges. The central claim is that vibe coding is consistently described in the literature as an iterative generation–evaluation–refinement loop rather than a one-shot activity, and that developer work shifts from writing code toward specification, supervision, and validation. The paper contributes a documented protocol, a replication package, an evidence-strength labeling scheme, a conceptual framework, and a comparison with prior reviews by Ge et al., Ray, and Fawzy et al.

Significance. If the central claim holds, this is a useful early consolidation of a fast-moving, practitioner-driven topic. The review's methodological transparency is a genuine strength: the search counts, inclusion/exclusion criteria, quality and credibility checklists, item-level scores, traceability from findings to source excerpts, and the public replication package exceed what is typical for an MLR in this area. The explicit cross-stream triangulation, especially in RQ5, and the careful denominator reporting are also commendable. However, the headline conclusion that vibe coding is 'iterative rather than one-shot' is, as argued below, not currently falsifiable given the protocol's search vocabulary and coding scheme. The paper is therefore valuable as a documented synthesis but needs protocol-level revision before its central interpretive claim can be accepted.

major comments (3)
  1. [Section 3.3 / Table 18 / Section 2.2.4] The RQ2 codebook contains no code for one-shot generation, pure delegation, or no-evaluation acceptance. Every code in Table 18 (validation pipelines, chat-based iterative loops, iterate–prompt–patch, etc.) presupposes multi-turn or validation activity. Takeaway 2's contrast—'iterative generation–evaluation–refinement loop rather than a one-shot code-generation activity'—is therefore not an empirical result of the synthesis; it is built into the extraction categories. The abstract itself states that vibe coding 'is often framed as one-shot prompting,' yet no RQ2 code records that framing and the Results section never counts or reconciles it. To make the central claim testable, the authors should add codes such as 'one-shot / direct generation without iterative refinement' and 'acceptance without evaluation,' re-code the corpus, and report how many sources characterize vibe coding that wa
  2. [Section 2.2.1 / Section 5.1] The Stage B search string includes 'prompt-based development*,' 'conversational programming,' and 'natural language programming,' terms that semantically presuppose iterative prompting or conversation, while deliberately excluding 'LLM-assisted programming,' 'AI pair programming,' 'AI coding assistant,' and 'agentic coding.' This is not merely a precision/recall trade-off; it is a directional sampling choice that enriches the corpus for iterative descriptions and removes a large body of literature that could contain one-shot or pure-delegation characterizations. Section 5.1 acknowledges reduced recall but not this directional bias. Because the headline claim is that the evidence 'consistently' describes an iterative loop, the authors should either broaden the search vocabulary to include the excluded terms, or provide a sensitivity analysis showing that the excluded literature would not
  3. [Section 2.3 / Section 5.4] The evidence-strength labels are assigned by the first author alone, without formal independent double-coding or an inter-rater agreement measure. Section 5.4 discloses this, but the label 'strong' attached to Takeaway 2 is used to support the central claim, and the underlying percentages (72%, 63%) are the product of a single coder's application of a codebook that lacks a one-shot category. For a review whose stated contribution is an 'evidence-weighted account,' this is a load-bearing reliability risk. I recommend independent coding of a sample of sources against the revised codebook, with agreement reported, or at minimum a clearer pre-specified rule for how source counts, quality bands, and consistency translate into strong/moderate/weak/emerging labels.
minor comments (5)
  1. [Figure 2] The phrase 'White Screening' appears in the figure; this should read 'Title Screening' or similar.
  2. [Section 3.3] The sentence 'Validation and testing sit inside this loop rather than as optional additions' is presented as a synthesis claim, but the supporting Table 18 reports only mentions of validation patterns. Consider softening to 'are frequently described as sitting inside this loop' to avoid overstating the strength of the evidence.
  3. [Section 6] In the discussion of Ge et al., the sentence beginning 'Based on the information reported, however, its search...' is grammatically incomplete. Please revise.
  4. [Section 3.9] Table 28 reports 'Security concerns including unsafe code...' at 50% (10 of 20 RQ8-contributing sources). The subsequent sentence says 'A similar proportion of studies (30%, 6 sources)'—'similar proportion' is misleading; consider 'The next most frequent challenges...'.
  5. [Section 2.2.2] The reported snowballing yield is very low (2 peer-reviewed and 3 grey additions). This is acknowledged, but the reader would benefit from a brief statement of how many references were checked in forward vs. backward snowballing, since the current description gives only total candidate counts.

Circularity Check

0 steps flagged

No significant circularity; iterative-loop finding rests on independent source excerpts, with a minor deductive-coding caveat.

full rationale

This is a multivocal literature review, not a derivation with fitted parameters or equations. The core claim that vibe coding is an iterative generation–evaluation–refinement loop is supported by verbatim excerpts from primary sources (e.g., WL15: 'iteratively conversing with a large language model'; GL8: 'iterative prompting and strategic direction') and every finding is traceable to source IDs. The only near-cyclic element is methodological: RQ2's rationale lists 'iterative generation and testing loops' as an example workflow, and the RQ-derived codebook (Table 18) contains iterative/loop codes but no explicit 'one-shot generation' code, so the 'rather than one-shot' contrast is not directly evidenced by the extraction scheme. The search string also includes 'conversational programming' and 'prompt-based development*' while excluding 'AI coding assistant' and 'agentic coding,' which could bias the corpus toward iterative descriptions. However, the paper's own limitations sections acknowledge reduced recall and coding subjectivity (Section 5.1, 5.4), and this is a standard deductive-coding structure rather than a result-forcing equivalence: the positive iterative evidence is independently present in the quoted sources, and the conclusion does not reduce by construction to the search or codebook. No load-bearing self-citations or imported uniqueness theorems appear. Accordingly, no circular step meets the evidentiary bar for flagging.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters or invented entities; the review's claims rest on the completeness and coding of the 47-source corpus and on the validity of the MLR instruments. The main risks are search recall, single-coder extraction, and treating mention frequency as evidence strength.

axioms (4)
  • domain assumption The Stage B search string and snowballing captured a representative sample of the vibe-coding literature.
    Section 2.2.1 and Section 5.1 acknowledge that broader terms such as 'LLM-assisted programming' and 'agentic coding' were deliberately excluded, which can reduce recall and bias the corpus toward sources that frame vibe coding as a distinct iterative workflow.
  • domain assumption Frequency of mention across sources is a meaningful basis for evidence-strength labels.
    Section 2.3 and Section 5.8 define evidence strength by number of sources, quality bands, and cross-stream consistency, while warning that counts measure mentions, not real-world prevalence. The central conclusions lean on these counts.
  • domain assumption A single-coder thematic synthesis with sampled senior review is adequate to avoid systematic bias.
    Section 2.3 and Section 5.4 state the first author performed screening, QA, extraction, coding, and evidence-strength assignment; no formal inter-rater agreement was measured. The evidence-strength labels and thematic boundaries depend on this assumption.
  • domain assumption Garousi et al. MLR guidelines and the Dybå–Dingsøyr checklist are valid instruments for this domain.
    The quality and credibility tiers used throughout Section 2 and Section 3 rest on these instruments being appropriate for assessing software-engineering and grey-literature sources.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Vibe Coding in Software Development: A Multivocal Literature Review." pith.science (2026). https://pith.science/paper/BYIXMGYL

@misc{pith2026260721652,
  author       = {Pith},
  title        = {Pith review of: Vibe Coding in Software Development: A Multivocal Literature Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYIXMGYL}},
  note         = {Machine review of arXiv:2607.21652}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vibe coding is a software development practice in which developers state intent in natural language and large language models generate code. It is often framed as one-shot prompting, but the evidence describes an intent-driven, iterative workflow whose outcomes depend on how generated code is evaluated and governed. Knowledge of how vibe coding is defined, practiced, and governed is scattered across academic and practitioner sources, and, to our knowledge, existing reviews have not yet integrated both evidence streams. We conducted a multivocal literature review of peer-reviewed and grey literature following established guidelines. Searches spanned 2022 to October 2025. After screening, credibility assessment, and snowballing, 47 sources were retained (28 peer-reviewed and 19 grey) and analyzed through descriptive mapping and thematic synthesis across eight research questions. Vibe coding is consistently described as an iterative generation-evaluation-revision loop rather than a one-shot activity, and developer work shifts from writing code towards specification, supervision, and validation. Short-term productivity and time-to-prototype gains are reported in 21 of 47 sources (45%), while evidence on maintainability, long-term quality, and safeguard effectiveness remains limited. Evidence is strongest for prototyping and user-interface work and weakest for production, data-intensive, and safety-critical use, and tool visibility does not imply effectiveness. This is one of the first reviews to integrate peer-reviewed and grey literature on vibe coding under a single documented protocol. Future work should evaluate safeguard effectiveness, study session-level dynamics and long-term maintainability, and test vibe coding in production, data-intensive, and safety-critical settings.

Figures

Figures reproduced from arXiv: 2607.21652 by Jussi Rasku, Kai-Kristian Kemell, Mika Saari, Muhammad Waseem, Pekka Abrahamsson, Shahbaz Siddeeq.

Figure 1
Figure 1. Figure 1: The vibe coding loop. The developer states the intent in natural language; one or more LLMs generate the code; the developer evaluates it by running and inspecting the result; and the developer refines the next prompt based on what they observe. The loop repeats until the goal is met. Review of the generated code line by line is often reduced or skipped, which sets vibe coding apart from ordinary AI-assist… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MLR review process [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Database-level reduction of the peer-reviewed corpus across screening stages. The record count falls from the 540 initial records through title and abstract screening, deduplication, and full-text assessment. This figure complements the overall review process in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Conceptual framework for vibe coding in software development, synthesized from the thematic analysis of 47 primary sources. (a) The process view captures the temporal dynamics of a development session, specifically how the iterative loop operates. (b) The structural view decomposes the phenomenon into interconnected layers that shape and are shaped by the loop, with each layer grounded in the corresponding… view at source ↗
Figure 5
Figure 5. Figure 5: Demographic distribution of peer-reviewed and grey-literature sources by publication year, publication type, and source category [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Fundamental skills for effective vibe coding, as described by two practitioner grey-literature sources (Huang [GL6] and Sapkota et al. [GL8]). This is a practitioner￾derived lens rather than a central framework of this review, and it is only partially supported by the wider evidence base [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Thinking layers, based on the practitioner accounts of Huang [GL6] and Sapkota et al. [GL8]. Shown as a practitioner￾derived illustration, not as a validated framework. Such knowledge helps constrain the solution space and guide technology choices, and it supports using LLMs to compare framework alternatives against project requirements. Checkpoints (version control discipline) Checkpoints are described as… view at source ↗
Figure 8
Figure 8. Figure 8: Layered vibe-coding workflow architecture, adapted from Ray [14]. The developer (left) composes a natural￾language prompt in the Prompt Box. The request passes through the Tool, Prompt-Engineering, and Integration layers before reaching the LLM Provider. Generated code is assem￾bled by the Stack Engine, deployed through the Deployment & Operations layer, and monitored via the Feedback & Observability layer… view at source ↗
Figure 9
Figure 9. Figure 9: Relationship between RQ3.3 categories and reported development outcomes (RQ4). Cell values represent the number of sources reporting each outcome for the corresponding category. (a) Audience dimension: novice-oriented sources emphasize productivity, while expert-oriented sources show a more balanced distribution across outcome types. (b) Task-domain dimension: UI and front-end tasks dominate the evidence b… view at source ↗
Figure 10
Figure 10. Figure 10: Tool ecosystem categories for vibe coding, with representative platforms. Takeaway 10 Copilot and chat-based assistants are the most vis￾ible tools across both evidence streams. AI-native IDEs and agentic tools appear more prominently in grey literature but still lack strong peer-reviewed evaluation. These patterns reflect visibility in the literature rather than demonstrated effectiveness. Ev￾idence stre… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 1 canonical work pages

  1. [1]

    Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50

    D. Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50

  2. [2]

    S. I. Ross, F. Martinez, S. Houde, M. Muller, J. D. Weisz, The programmer’s assistant: Conversational interaction with a large lan- guage model for software development, in: Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 491–514

  3. [3]

    J.Sauvola,S.Tarkoma,M.Klemettinen,J.Riekki,D.Doermann,Fu- tureofsoftwaredevelopmentwithgenerativeai,AutomatedSoftware Engineering 31 (1) (2024) 26

  4. [4]

    Karpathy, There’s a new kind of coding i call ‘vibe coding’..., (February 2025)

    A. Karpathy, There’s a new kind of coding i call ‘vibe coding’..., (February 2025). URLhttps://x.com/karpathy/status/1886192184808149383?lang=en

  5. [5]

    S.Peng,E.Kalliamvakou,P.Cihon,M.Demirer,Theimpactofaion developer productivity: Evidence from github copilot, arXiv preprint arXiv:2302.06590 (2023)

  6. [6]

    Mastropaolo, L

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scal- abrino, R. Oliveto, G. Bavota, On the robustness of code gener- ation techniques: An empirical study on github copilot, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2149–2160

  7. [7]

    Akhoroz, C

    M. Akhoroz, C. Yildirim, Conversational ai as a coding assistant: Understandingprogrammers’interactionswithandexpectationsfrom large language models for coding, arXiv preprint arXiv:2503.16508 (2025)

  8. [8]

    URLhttps://www.linkedin.com/pulse/state-vibe-coding-tools-may -2025-nufar-gaspar-x1znf/

    N.Gaspar,Thestateofvibecodingtools(may2025),LinkedIn(May 2025). URLhttps://www.linkedin.com/pulse/state-vibe-coding-tools-may -2025-nufar-gaspar-x1znf/

  9. [9]

    Huang, Vibe coding fundamentals in 33 minutes, YouTube (May 2025)

    T. Huang, Vibe coding fundamentals in 33 minutes, YouTube (May 2025). URLhttps://youtu.be/iLCDSY2XX7E S.Siddeeq et al.:Preprint submitted to ElsevierPage 34 of 35 Multivocal Review of Vibe Coding

  10. [10]

    Harkar, What is vibe coding?, IBM (2025)

    S. Harkar, What is vibe coding?, IBM (2025). URLhttps://www.ibm.com/think/topics/vibe-coding

  11. [11]

    vibe coding

    H. Chowdhury, J. Mann, Silicon valley’s next act: Bringing “vibe coding” to the world, Business Insider (2025)

  12. [12]

    Edwards, Will the future of software development run on vibes?, Ars Technica (March 2025)

    B. Edwards, Will the future of software development run on vibes?, Ars Technica (March 2025). URLhttps://arstechnica.com/ai/2025/03/is-vibe-coding-with-a i-gnarly-or-reckless-maybe-some-of-both/

  13. [13]

    Y. Ge, L. Mei, Z. Duan, T. Li, Y. Zheng, Y. Wang, L. Wang, J. Yao, T. Liu, Y. Cai, et al., A survey of vibe coding with large language models, arXiv preprint arXiv:2510.12399 (2025)

  14. [14]

    P. P. Ray, A review on vibe coding: Fundamentals, state-of-the-art, challenges and future directions, Authorea Preprints (2025)

  15. [15]

    Fawzy, A

    A. Fawzy, A. Tahir, K. Blincoe, Vibe coding in practice: Motiva- tions,challenges,andafutureoutlook–agreyliteraturereview,arXiv preprint arXiv:2510.00328 (2025)

  16. [16]

    V.Garousi,M.Felderer,M.V.Mäntylä,Guidelinesforincludinggrey literature and conducting multivocal literature reviews in software engineering, Information and software technology 106 (2019) 101– 121

  17. [17]

    B.Kitchenham,S.Charters,etal.,Guidelinesforperformingsystem- atic literature reviews in software engineering (2007)

  18. [18]

    V. R. Basili, G. Caldiera, H. D. Rombach, The goal question metric approach, in: Encyclopedia of Software Engineering, Vol. 1, John Wiley & Sons, 1994, pp. 528–532

  19. [19]

    C.Wohlin,Guidelinesforsnowballinginsystematicliteraturestudies andareplicationinsoftwareengineering,in:Proceedingsofthe18th international conference on evaluation and assessment in software engineering, 2014, pp. 1–10

  20. [20]

    T. Dybå, T. Dingsøyr, Empirical studies of agile software devel- opment: A systematic review, Information and software technology 50 (9-10) (2008) 833–859

  21. [21]

    D. S. Cruzes, T. Dybå, Recommended steps for thematic synthesis in software engineering, in: 2011 International Symposium on Empir- ical Software Engineering and Measurement (ESEM), IEEE, 2011, pp. 275–284

  22. [22]

    A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Des- marais, Z. M. J. Jiang, Github copilot ai pair programmer: Asset or liability?, Journal of Systems and Software 203 (2023) 111734

  23. [23]

    Siddeeq, M

    S. Siddeeq, M. Waseem, K.-K. Kemell, M. Saari, J. Rasku, P. Abra- hamsson, Vibe coding in software development: A multivocal litera- ture review (Jul. 2026).doi:10.5281/zenodo.21490083. URLhttps://doi.org/10.5281/zenodo.21490083 S.Siddeeq et al.:Preprint submitted to ElsevierPage 35 of 35

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.