REVIEW 3 major objections 5 minor 23 references
A systematic review of 47 sources finds that vibe coding is not one-shot prompting but an iterative loop of intent, generation, evaluation, and refinement, with productivity gains that hinge on the surrounding evaluation and governance.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Vibe coding evidence from 47 sources describes an intent-driven, iterative evaluation loop whose productivity benefits are conditional on review and validation practices.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection Well-documented MLR whose headline "iterative loop" claim is partly a product of the search string and codebook — treat that finding as provisional rather than established. the 3 major comments →
Vibe Coding in Software Development: A Multivocal Literature Review
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On its own terms, the paper's central discovery is that the scattered literature on vibe coding converges on a single process shape: the developer states intent in natural language, an LLM generates or revises code, the developer evaluates the result by running, inspecting, or testing it, and then refines the next prompt based on observed behavior. Across 43 of 47 sources, validation and evaluation pipelines (72%) and chat-based iterative loops (63%) are the dominant workflow patterns, and across 36 sources the practice is described most often as an intent-driven socio-technical practice (69%) and as natural-language-to-code prompting (67%). The review labels this an iterative control system
What carries the argument
The central object is the vibe coding loop: a feedback-driven cycle of intent specification, LLM generation, developer evaluation, and prompt refinement, with validation and testing inside the loop rather than optional additions. The paper's conceptual framework adds four interacting layers—workflow, role and experience, outcomes, and risk and governance—moderated by usage context and tool ecosystem, and attaches an evidence-strength label (strong, moderate, weak, emerging) to each finding. That combination of loop, layered model, and evidence weighting is what carries the argument.
Load-bearing premise
The conclusion that vibe coding is 'consistently' iterative rests on a search string and eligibility rules that deliberately excluded literature using broader terms such as 'LLM-assisted programming,' 'AI coding assistant,' and 'agentic coding'; if that excluded literature contains many one-shot or pure-delegation accounts, the consistency claim would overstate the evidence.
What would settle it
A concrete falsifier: re-run the review's search with the deliberately excluded terms ('LLM-assisted programming,' 'AI coding assistant,' 'agentic coding') and count how many of the additionally retrieved sources describe vibe-coding-adjacent work as one-shot, fully delegated generation. If a substantial number do, the claim that the literature 'consistently' describes an iterative loop is falsified. A complementary experiment: compare matched teams using vibe coding with and without enforced test-and-review pipelines; if defect rates and maintainability do not differ, the claim that outcomes
If this is right
- Vibe coding should be managed as an engineering workflow: teams that adopt it without automated tests, review routines, and validation pipelines are likely trading short-term speed for long-term defect and maintenance burden.
- Reported productivity gains are most credible for prototyping and UI/front-end work; claims of gains in production, data-intensive, or safety-critical settings should not be assumed until studied.
- Safeguards such as human-in-the-loop review and validation pipelines are consistently recommended, but since their effectiveness is under-tested, they should be treated as hypotheses to evaluate rather than proven controls.
- Session-level dynamics—momentum breaks, context drift, and repetitive prompt–patch cycles—are real but under-measured, pointing to a need for instrumented long-horizon studies.
- The review's evidence-strength labels provide a practical map: strong claims are limited to the iterative-loop description and short-term productivity; everything else is moderate, weak, or emerging.
Where Pith is reading between the lines
- If the iterative-control-system framing is right, a testable extension is to instrument real development sessions and check whether teams with stronger evaluation loops (CI gates, test coverage, structured review) show lower escaped-defect rates and better maintainability than teams that iterate without them.
- The review's exclusion of broader terms like 'LLM-assisted programming' and 'agentic coding' suggests a conservative lower bound: one-shot or pure-delegation characterizations may exist in that excluded literature, so the 'consistently iterative' claim is best read as applying to the vibe-coding-specific corpus.
- The novice/expert split implies a design consequence: tools and training should target verification skill (how to evaluate and test generated code) rather than only prompt skill, because overtrust is most dangerous for the novice users the practice is said to empower.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports a multivocal literature review (MLR) of 47 sources (28 peer-reviewed, 19 grey) on vibe coding, following Garousi et al.'s MLR guidelines. The review poses eight research questions covering definitions, workflows, developer-role shifts, outcomes, risks/safeguards, usage contexts, tools, and open challenges. The central claim is that vibe coding is consistently described in the literature as an iterative generation–evaluation–refinement loop rather than a one-shot activity, and that developer work shifts from writing code toward specification, supervision, and validation. The paper contributes a documented protocol, a replication package, an evidence-strength labeling scheme, a conceptual framework, and a comparison with prior reviews by Ge et al., Ray, and Fawzy et al.
Significance. If the central claim holds, this is a useful early consolidation of a fast-moving, practitioner-driven topic. The review's methodological transparency is a genuine strength: the search counts, inclusion/exclusion criteria, quality and credibility checklists, item-level scores, traceability from findings to source excerpts, and the public replication package exceed what is typical for an MLR in this area. The explicit cross-stream triangulation, especially in RQ5, and the careful denominator reporting are also commendable. However, the headline conclusion that vibe coding is 'iterative rather than one-shot' is, as argued below, not currently falsifiable given the protocol's search vocabulary and coding scheme. The paper is therefore valuable as a documented synthesis but needs protocol-level revision before its central interpretive claim can be accepted.
major comments (3)
- [Section 3.3 / Table 18 / Section 2.2.4] The RQ2 codebook contains no code for one-shot generation, pure delegation, or no-evaluation acceptance. Every code in Table 18 (validation pipelines, chat-based iterative loops, iterate–prompt–patch, etc.) presupposes multi-turn or validation activity. Takeaway 2's contrast—'iterative generation–evaluation–refinement loop rather than a one-shot code-generation activity'—is therefore not an empirical result of the synthesis; it is built into the extraction categories. The abstract itself states that vibe coding 'is often framed as one-shot prompting,' yet no RQ2 code records that framing and the Results section never counts or reconciles it. To make the central claim testable, the authors should add codes such as 'one-shot / direct generation without iterative refinement' and 'acceptance without evaluation,' re-code the corpus, and report how many sources characterize vibe coding that wa
- [Section 2.2.1 / Section 5.1] The Stage B search string includes 'prompt-based development*,' 'conversational programming,' and 'natural language programming,' terms that semantically presuppose iterative prompting or conversation, while deliberately excluding 'LLM-assisted programming,' 'AI pair programming,' 'AI coding assistant,' and 'agentic coding.' This is not merely a precision/recall trade-off; it is a directional sampling choice that enriches the corpus for iterative descriptions and removes a large body of literature that could contain one-shot or pure-delegation characterizations. Section 5.1 acknowledges reduced recall but not this directional bias. Because the headline claim is that the evidence 'consistently' describes an iterative loop, the authors should either broaden the search vocabulary to include the excluded terms, or provide a sensitivity analysis showing that the excluded literature would not
- [Section 2.3 / Section 5.4] The evidence-strength labels are assigned by the first author alone, without formal independent double-coding or an inter-rater agreement measure. Section 5.4 discloses this, but the label 'strong' attached to Takeaway 2 is used to support the central claim, and the underlying percentages (72%, 63%) are the product of a single coder's application of a codebook that lacks a one-shot category. For a review whose stated contribution is an 'evidence-weighted account,' this is a load-bearing reliability risk. I recommend independent coding of a sample of sources against the revised codebook, with agreement reported, or at minimum a clearer pre-specified rule for how source counts, quality bands, and consistency translate into strong/moderate/weak/emerging labels.
minor comments (5)
- [Figure 2] The phrase 'White Screening' appears in the figure; this should read 'Title Screening' or similar.
- [Section 3.3] The sentence 'Validation and testing sit inside this loop rather than as optional additions' is presented as a synthesis claim, but the supporting Table 18 reports only mentions of validation patterns. Consider softening to 'are frequently described as sitting inside this loop' to avoid overstating the strength of the evidence.
- [Section 6] In the discussion of Ge et al., the sentence beginning 'Based on the information reported, however, its search...' is grammatically incomplete. Please revise.
- [Section 3.9] Table 28 reports 'Security concerns including unsafe code...' at 50% (10 of 20 RQ8-contributing sources). The subsequent sentence says 'A similar proportion of studies (30%, 6 sources)'—'similar proportion' is misleading; consider 'The next most frequent challenges...'.
- [Section 2.2.2] The reported snowballing yield is very low (2 peer-reviewed and 3 grey additions). This is acknowledged, but the reader would benefit from a brief statement of how many references were checked in forward vs. backward snowballing, since the current description gives only total candidate counts.
Circularity Check
No significant circularity; iterative-loop finding rests on independent source excerpts, with a minor deductive-coding caveat.
full rationale
This is a multivocal literature review, not a derivation with fitted parameters or equations. The core claim that vibe coding is an iterative generation–evaluation–refinement loop is supported by verbatim excerpts from primary sources (e.g., WL15: 'iteratively conversing with a large language model'; GL8: 'iterative prompting and strategic direction') and every finding is traceable to source IDs. The only near-cyclic element is methodological: RQ2's rationale lists 'iterative generation and testing loops' as an example workflow, and the RQ-derived codebook (Table 18) contains iterative/loop codes but no explicit 'one-shot generation' code, so the 'rather than one-shot' contrast is not directly evidenced by the extraction scheme. The search string also includes 'conversational programming' and 'prompt-based development*' while excluding 'AI coding assistant' and 'agentic coding,' which could bias the corpus toward iterative descriptions. However, the paper's own limitations sections acknowledge reduced recall and coding subjectivity (Section 5.1, 5.4), and this is a standard deductive-coding structure rather than a result-forcing equivalence: the positive iterative evidence is independently present in the quoted sources, and the conclusion does not reduce by construction to the search or codebook. No load-bearing self-citations or imported uniqueness theorems appear. Accordingly, no circular step meets the evidentiary bar for flagging.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption The Stage B search string and snowballing captured a representative sample of the vibe-coding literature.
- domain assumption Frequency of mention across sources is a meaningful basis for evidence-strength labels.
- domain assumption A single-coder thematic synthesis with sampled senior review is adequate to avoid systematic bias.
- domain assumption Garousi et al. MLR guidelines and the Dybå–Dingsøyr checklist are valid instruments for this domain.
Cite this review
Pith. "Pith review of Vibe Coding in Software Development: A Multivocal Literature Review." pith.science (2026). https://pith.science/paper/BYIXMGYL
@misc{pith2026260721652,
author = {Pith},
title = {Pith review of: Vibe Coding in Software Development: A Multivocal Literature Review},
year = {2026},
howpublished = {\url{https://pith.science/paper/BYIXMGYL}},
note = {Machine review of arXiv:2607.21652}
}
read the original abstract
Vibe coding is a software development practice in which developers state intent in natural language and large language models generate code. It is often framed as one-shot prompting, but the evidence describes an intent-driven, iterative workflow whose outcomes depend on how generated code is evaluated and governed. Knowledge of how vibe coding is defined, practiced, and governed is scattered across academic and practitioner sources, and, to our knowledge, existing reviews have not yet integrated both evidence streams. We conducted a multivocal literature review of peer-reviewed and grey literature following established guidelines. Searches spanned 2022 to October 2025. After screening, credibility assessment, and snowballing, 47 sources were retained (28 peer-reviewed and 19 grey) and analyzed through descriptive mapping and thematic synthesis across eight research questions. Vibe coding is consistently described as an iterative generation-evaluation-revision loop rather than a one-shot activity, and developer work shifts from writing code towards specification, supervision, and validation. Short-term productivity and time-to-prototype gains are reported in 21 of 47 sources (45%), while evidence on maintainability, long-term quality, and safeguard effectiveness remains limited. Evidence is strongest for prototyping and user-interface work and weakest for production, data-intensive, and safety-critical use, and tool visibility does not imply effectiveness. This is one of the first reviews to integrate peer-reviewed and grey literature on vibe coding under a single documented protocol. Future work should evaluate safeguard effectiveness, study session-level dynamics and long-term maintainability, and test vibe coding in production, data-intensive, and safety-critical settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50
D. Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50
2024
-
[2]
S. I. Ross, F. Martinez, S. Houde, M. Muller, J. D. Weisz, The programmer’s assistant: Conversational interaction with a large lan- guage model for software development, in: Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 491–514
2023
-
[3]
J.Sauvola,S.Tarkoma,M.Klemettinen,J.Riekki,D.Doermann,Fu- tureofsoftwaredevelopmentwithgenerativeai,AutomatedSoftware Engineering 31 (1) (2024) 26
2024
-
[4]
Karpathy, There’s a new kind of coding i call ‘vibe coding’..., (February 2025)
A. Karpathy, There’s a new kind of coding i call ‘vibe coding’..., (February 2025). URLhttps://x.com/karpathy/status/1886192184808149383?lang=en
arXiv 2025
-
[5]
S.Peng,E.Kalliamvakou,P.Cihon,M.Demirer,Theimpactofaion developer productivity: Evidence from github copilot, arXiv preprint arXiv:2302.06590 (2023)
Pith/arXiv arXiv 2023
-
[6]
Mastropaolo, L
A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scal- abrino, R. Oliveto, G. Bavota, On the robustness of code gener- ation techniques: An empirical study on github copilot, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2149–2160
2023
-
[7]
M. Akhoroz, C. Yildirim, Conversational ai as a coding assistant: Understandingprogrammers’interactionswithandexpectationsfrom large language models for coding, arXiv preprint arXiv:2503.16508 (2025)
Pith/arXiv arXiv 2025
-
[8]
URLhttps://www.linkedin.com/pulse/state-vibe-coding-tools-may -2025-nufar-gaspar-x1znf/
N.Gaspar,Thestateofvibecodingtools(may2025),LinkedIn(May 2025). URLhttps://www.linkedin.com/pulse/state-vibe-coding-tools-may -2025-nufar-gaspar-x1znf/
2025
-
[9]
Huang, Vibe coding fundamentals in 33 minutes, YouTube (May 2025)
T. Huang, Vibe coding fundamentals in 33 minutes, YouTube (May 2025). URLhttps://youtu.be/iLCDSY2XX7E S.Siddeeq et al.:Preprint submitted to ElsevierPage 34 of 35 Multivocal Review of Vibe Coding
2025
-
[10]
Harkar, What is vibe coding?, IBM (2025)
S. Harkar, What is vibe coding?, IBM (2025). URLhttps://www.ibm.com/think/topics/vibe-coding
2025
-
[11]
vibe coding
H. Chowdhury, J. Mann, Silicon valley’s next act: Bringing “vibe coding” to the world, Business Insider (2025)
2025
-
[12]
Edwards, Will the future of software development run on vibes?, Ars Technica (March 2025)
B. Edwards, Will the future of software development run on vibes?, Ars Technica (March 2025). URLhttps://arstechnica.com/ai/2025/03/is-vibe-coding-with-a i-gnarly-or-reckless-maybe-some-of-both/
2025
-
[13]
Y. Ge, L. Mei, Z. Duan, T. Li, Y. Zheng, Y. Wang, L. Wang, J. Yao, T. Liu, Y. Cai, et al., A survey of vibe coding with large language models, arXiv preprint arXiv:2510.12399 (2025)
arXiv 2025
-
[14]
P. P. Ray, A review on vibe coding: Fundamentals, state-of-the-art, challenges and future directions, Authorea Preprints (2025)
2025
- [15]
-
[16]
V.Garousi,M.Felderer,M.V.Mäntylä,Guidelinesforincludinggrey literature and conducting multivocal literature reviews in software engineering, Information and software technology 106 (2019) 101– 121
2019
-
[17]
B.Kitchenham,S.Charters,etal.,Guidelinesforperformingsystem- atic literature reviews in software engineering (2007)
2007
-
[18]
V. R. Basili, G. Caldiera, H. D. Rombach, The goal question metric approach, in: Encyclopedia of Software Engineering, Vol. 1, John Wiley & Sons, 1994, pp. 528–532
1994
-
[19]
C.Wohlin,Guidelinesforsnowballinginsystematicliteraturestudies andareplicationinsoftwareengineering,in:Proceedingsofthe18th international conference on evaluation and assessment in software engineering, 2014, pp. 1–10
2014
-
[20]
T. Dybå, T. Dingsøyr, Empirical studies of agile software devel- opment: A systematic review, Information and software technology 50 (9-10) (2008) 833–859
2008
-
[21]
D. S. Cruzes, T. Dybå, Recommended steps for thematic synthesis in software engineering, in: 2011 International Symposium on Empir- ical Software Engineering and Measurement (ESEM), IEEE, 2011, pp. 275–284
2011
-
[22]
A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Des- marais, Z. M. J. Jiang, Github copilot ai pair programmer: Asset or liability?, Journal of Systems and Software 203 (2023) 111734
2023
-
[23]
S. Siddeeq, M. Waseem, K.-K. Kemell, M. Saari, J. Rasku, P. Abra- hamsson, Vibe coding in software development: A multivocal litera- ture review (Jul. 2026).doi:10.5281/zenodo.21490083. URLhttps://doi.org/10.5281/zenodo.21490083 S.Siddeeq et al.:Preprint submitted to ElsevierPage 35 of 35
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.