{"id":"ca47d567-b22a-4aa7-8184-c20d2e0347cf","arxiv_id":"2507.23120","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vibe modeling proposes using LLM-driven agents to create software models rather than code, then generating the final software with deterministic model-to-code tools.","lead":"This paper introduces 'vibe modeling', a development approach where AI agents turn natural language descriptions into software models, and then deterministic tools generate the code from those models. A smart generalist should read it to understand a proposed middle path between fully AI-generated code and traditional model-driven engineering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central reliability claim depends on non-expert users validating LLM-produced models, an empirical premise asserted without evidence and left open by Section 6.1 and Section 2's prior negative results.","rationale":"The reader's weakest assumption identifies the same premise: that non-experts can validate LLM-produced models well enough for deterministic code generation to yield reliable software. The paper is a vision paper, so absence of an empirical evaluation is not by itself a flaw; however, the wording 'If the model is good, we know the code will be good' elevates an untested hypothesis to a guarantee. The paper's own Section 6.1 admits model-quality evaluation is unsolved, and Section 2's cited studies show LLM modeling errors occur precisely in abstraction decisions that non-experts are poorly equipped to assess. This is not a disagreement with consensus; it is a correctness risk in the central argument: the claimed separation from vibe coding depends on validation reliability, and no evidence is provided for it. A conditional verdict remains appropriate because the concept is coherent and the proposed MCP infrastructure is a reasonable first step, but the reliability claim should be explicitly reframed as a hypothesis pending the proposed user study. No new objection beyond the reader's was found.","tokens_in":9106,"tokens_out":3534,"duration_ms":43402,"concrete_test":"Run a controlled study: recruit non-expert domain users and give each a requirements specification for a moderately complex system plus an LLM-generated model in which the experimenters have injected M semantic errors (wrong inheritance, missing association, ambiguous requirement). Give a separate cohort the corresponding vibe-coded source with equivalent injected bugs. Measure error detection rates and the fraction of participants who approve the artifact. If model-validation detection is not significantly better than code-review detection, or if approval rates stay high despite injected errors, the 'understandable output' advantage and the reliability guarantee are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 claims a decisive advantage over vibe coding: models are 'understandable output' that a user can validate 'even without coding expertise,' and deterministic code generation then ensures 'If the model is good, we know the code will be good.' The load-bearing premise is that a non-expert conversational participant can correctly judge model adequacy, and that a validated model suffices for reliability. This premise is not established. Prior results cited in Section 2 show LLM-generated models fail exactly where abstraction is needed (e.g., inheritance vs. attributes, association classes); those are the errors a non-expert is least likely to notice. Section 6.1 lists 'how to evaluate the quality of the models inferred by the agents' as an open challenge. Without a model-quality criterion, 'good' in the paper's central sentence is circular: the guarantee holds only if the model is already correct, and the hard problem is determining correctness. If non-expert validation is unreliable, vibe modeling inherits vibe coding's blind-trust problem while adding a modeling step; the claimed advantage over vibe coding and the 'reliable complex systems' outcome both collapse.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes 'vibe modeling' as a development approach in which an LLM-based agent interacts conversationally with a domain expert to produce a conceptual model, and the final system is then generated by deterministic, rule-based code generators. Section 3 defines the concept and argues for two advantages over vibe coding: models are more understandable and therefore validatable by non-programmers, and deterministic generation makes code quality depend only on model quality. Section 4 extends the idea to multi-agent collaboration; Section 5 proposes using the Model Context Protocol (MCP) to connect agents to modeling platforms and reports a small prototype MCP server for the BESSER low-code platform; Section 6 lists open challenges. The paper is explicitly a position/vision paper, with no empirical evaluation of the proposed workflow.","tokens_in":9302,"tokens_out":5995,"duration_ms":65591,"significance":"As a research agenda, the paper names a real gap: the modeling community has studied LLM-based model inference mostly as one-shot prompting, while the code-generation community has adopted vibe coding without addressing reliability. The proposal to separate conversational model construction from deterministic generation is a plausible and potentially valuable synthesis. The paper also deserves credit for being candid in Section 6 about open challenges, including model-quality evaluation, agent training data, and traceability. However, the central claim that vibe modeling yields 'reliable complex systems' for non-experts is not established: it rests on the assumptions that non-experts can validate LLM-produced models and that a good model is sufficient for reliable code. These assumptions are exactly the ones left open in Section 6.1. The paper is therefore more a proposal than a demonstrated result, and its strengths are in framing and infrastructure ideas rather than evidence.","major_comments":[{"comment":"The 'Reliable code-generation' advantage is load-bearing but is stated as 'If the model is good, we know the code will be good,' and the paper provides no operational criterion for 'good.' Section 6.1 explicitly lists 'How to evaluate the quality of the models inferred by the agents?' as an open challenge, and Section 6.3 only proposes storing a confidence score, not a validation method. As written, the reliability guarantee holds only when the hard part of the problem (knowing that the model is correct) has already been solved. I recommend reformulating this advantage as a research hypothesis and adding to Section 6.1 a concrete sketch of candidate quality criteria, for example consistency checking, scenario-based validation, or comparison against a manually built reference model for a benchmark domain.","section":"Section 3"},{"comment":"The 'Understandable output' advantage is in tension with the evidence the paper itself cites in Section 2. Camara et al. [8] found that LLM-generated UML class diagrams had the worst results exactly when abstraction was required (using inheritance instead of attributes, or creating association classes), and several cited works ([12,29,31]) report that identifying relationships among classes remains challenging. These are precisely the kinds of errors that a non-expert is least likely to detect by inspection. The section should either weaken the claim that non-experts can validate the output 'even without coding expertise,' or propose concrete validation aids (model smells, automated consistency checks, or executable examples generated from the model) that would make such validation plausible.","section":"Section 3"},{"comment":"The paper asserts that code-generation templates need only be verified 'once and for all,' but for reliable complex systems, generator correctness is itself a nontrivial assurance problem. The paper does not indicate what 'verified once and for all' means (certified compilers, test suites, formal verification, or something weaker). This matters because the reliability claim shifts all residual risk onto the model, and the manuscript should either specify the intended level of rigor or soften the claim to avoid the impression that deterministic generation by itself guarantees dependable code.","section":"Section 3"},{"comment":"The BESSER MCP server example in Listing 1.1 only exposes a function that creates a new, empty DomainModel and returns it serialized. This does not demonstrate the core vibe-modeling loop shown in Figure 1, which consists of conversational model construction, validation, refinement, and eventual code generation. I suggest either extending the prototype to illustrate at least one complete refinement-and-validation cycle, or explicitly labeling the current implementation as a first building block and stating that the remaining steps are future work. The current wording ('Example of creating a new model via a service exposed by the BESSER MCP Server') is accurate but does not support the broader feasibility claims made elsewhere in the paper.","section":"Section 5"}],"minor_comments":[{"comment":"The phrase 'at the expenses of' should be 'at the expense of.'","section":"Abstract"},{"comment":"In the sentence beginning 'Therefore, We propose to use MCP...,' the 'W' in 'We' should be lowercase.","section":"Section 5"},{"comment":"The phrase 'the limitations of LLMS' should be 'the limitations of LLMs.'","section":"Section 6.2"},{"comment":"The introductory phrase 'Aspects like:' is followed by a bullet list of sentence fragments; consider rewriting each bullet as a full question to make the open challenges easier to reference and discuss.","section":"Section 6.1"},{"comment":"The keyword 'Low-modeling' does not appear in the body of the paper; either introduce the term in the text or remove it from the keyword list.","section":"Keywords"},{"comment":"The phrase 'the more the merrier' is colloquial for a technical paper; consider rephrasing to something like 'a larger number of diverse agents may lead to a better final model.'","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"This is a single-author vision paper with no empirical validation. It could be a good fit for a position-paper track or a new-ideas session at a modeling or software-engineering venue, and the author is transparent about open challenges. For a full archival paper, the central claims about reliability and non-expert validation need to be reframed as testable hypotheses, and the proof-of-concept section needs to be extended or more carefully scoped. The self-citation pattern is not inappropriate for a research agenda rooted in the author's prior work on BESSER and low-code, but the paper should avoid implying that the proposed infrastructure is already validated beyond the trivial example shown."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a good vision paper. The author takes the scattered work on LLM-for-modeling, adds the agentic conversation step, and pairs it with deterministic low-code code generation. That integration — 'vibe modeling' — is genuinely a new framing, even if every component has been published before. The paper is clearly written, the related work is current, and the roadmap in Section 6 is unusually candid about what is unknown.\n\nWhat it does well: it correctly identifies that vibe coding's weakness is the blind trust in generated code, and that models are a more inspectable intermediate artifact. The MCP-based infrastructure proposal is practical and concrete, with a small but real code snippet from the BESSER MCP server. The literature review covers the main results (Cámara, Fill, Chaaben, Kourani) and highlights the abstraction failures.\n\nWhere it gets soft: the central claim 'If the model is good, we know the code will be good' (Section 3) is doing more work than the paper admits. Determinism in code generation only pushes the reliability question upstream to model quality. The paper itself lists 'how to evaluate the quality of the models inferred by the agents' as an open challenge in 6.1, so the sentence is at best premature. The 'understandable output' argument assumes non-experts can validate models; Section 2's own evidence says LLM errors cluster at abstractions like inheritance and association classes — exactly the mistakes a non-expert is least likely to see. So the advantage over vibe coding is plausible, not demonstrated. To be fair, the paper does label these as open issues in the roadmap, but the Section 3 claims read as established advantages rather than hypotheses.\n\nThe stress-test note's circularity point is fair in spirit: 'good' is only meaningful with a model-quality oracle, and no oracle is proposed. The paper needs to acknowledge that the whole approach hinges on this validation problem, not just list it as a future challenge.\n\nWho this is for: MDE researchers, low-code platform folks, and anyone thinking about agentic LLM workflows in software engineering. It deserves a serious referee if the venue is a vision/new-ideas track, with revisions that hedge the reliability claims and add a preliminary feasibility sketch. I'd encourage engagement — the concept is timely and worth arguing about.","headline":"A clear, honest vision paper that names a real integration opportunity, but the reliability promise rests on an untested assumption about non-expert model validation; worth a serious referee with revisions.","tokens_in":9821,"tokens_out":3621,"would_cite":true,"duration_ms":39892,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces 'vibe modeling' as a development method where an LLM converses with the user to produce validated software models, and deterministic rule-based generators then turn those models into running code.","keywords":["Vibe Modeling","Low-modeling","Low-code","DSL","Artificial Intelligence","Model-driven","Vibe Coding"],"falsifier":"A controlled study in which non-expert users validate LLM-generated models and then the generated systems are tested: if a substantial fraction of accepted models lead to failing or unsafe generated code, the claim that vibe modeling yields reliable software would be refuted.","tokens_in":8890,"feed_emoji":"🤖","tokens_out":5801,"duration_ms":60791,"temperature":0.7,"pith_summary":"This paper introduces 'vibe modeling' as a software development method that sits between vibe coding and classical model-driven engineering. Instead of asking an LLM to write code directly, the user converses with an agent that produces and refines a formal model of the system; once the model is validated, rule-based generators compile it into the final software. The author argues this gives the speed and natural-language accessibility of vibe coding while keeping the understandability and reliability of model-based development. The paper lays out the development process, collaborative multi-agent extensions, the infrastructure needed, and a roadmap of open challenges. A sympathetic reader would care because it proposes a concrete way to make AI-assisted development safer for non-experts and to revive the role of modeling in modern toolchains.","feed_headline":"Move over vibe coding: let AI build the models","feed_subtitle":"LLM-drafted models plus deterministic code generation could put reliable software in non-experts' hands.","key_machinery":"The carrying mechanism is the two-stage vibe modeling pipeline: a conversational LLM agent, guided by a domain expert, iteratively produces and validates a model, and then a deterministic rule-based generator compiles that model into code. The Model Context Protocol (MCP) is the supporting infrastructure: it standardizes how agents discover and call modeling-platform services, so any agent can create, validate, and manipulate models without embedding the modeling stack. The paper also introduces the notion that models carry uncertainty and traceability, with confidence scores and change proposals attached to model elements.","core_discovery":"The paper's central claim is that 'vibe modeling' — building software by conversing with an LLM trained for modeling rather than coding, then generating the final system with deterministic rule-based code generators — can combine the strengths of AI and model-driven engineering. The author argues that because the LLM's output is a model rather than code, users can understand and validate it without deep programming expertise, and because code generation is deterministic, a validated model reliably yields good code. The paper further claims this makes vibe modeling useful to both technical and non-technical experts and positions it as a model-driven alternative to vibe coding. It supports the proposal with a description of the iterative human-agent modeling loop and a proof-of-concept integration of a modeling platform with an agent through the Model Context Protocol.","pith_inferences":["A natural extension the paper does not run: a head-to-head benchmark comparing vibe modeling against vibe coding on defect density, maintainability, and how well users catch errors in the artifact they review.","The paper's traceability requirement implies that modeling platforms will need new versioning and provenance standards for model elements; this is left to future work.","If non-experts can validate models reliably, vibe modeling could carry low-code development into domains like citizen data collection where users have no coding training; the paper does not discuss such applications."],"forward_implications":["Non-programmers could build reliable systems by validating a model instead of trusting generated code they cannot read.","Because code generation is deterministic, regenerating the system after a model change would not require re-verifying the generated code each time.","Vibe modeling would restore conceptual modeling to a place in the AI-assisted development workflow, countering the trend toward direct code generation.","Standardizing modeling services as MCP servers would let any LLM agent work with any modeling platform, avoiding vendor-specific agent integrations.","In collaborative settings, a community of specialized agents could produce better models than a single agent, with a consensus mechanism selecting among competing proposals."],"supporting_citations":[{"why":"It provides evidence that zero-shot prompting can produce UML class diagrams from text, grounding the claim that LLMs can draft models.","marker":"[8]"},{"why":"It shows GPT-4 can create ER, BPMN, and Heraklit models from examples, supporting the feasibility of LLM-based model inference.","marker":"[15]"},{"why":"It demonstrates few-shot prompting for concept recommendation in domain modeling, another basis for the proposed conversational model creation.","marker":"[10]"},{"why":"It proposes a human-in-the-loop approach that refines LLM-generated process models with user feedback, the direct precursor of the vibe modeling loop.","marker":"[20]"},{"why":"It supplies the open-source low-code platform used for the proof-of-concept MCP server in the paper.","marker":"[1]"},{"why":"It establishes the idea of executable UML, the grounding for the claim that models can be deterministically compiled into running systems.","marker":"[22]"},{"why":"It provides a dataset for machine learning in model-driven engineering, which the paper cites as a starting point for training specialized modeling agents.","marker":"[21]"},{"why":"It surveys multi-agent LLM systems, supporting the collaborative vibe modeling extension.","marker":"[16]"}],"fun_headline_variants":["Vibe modeling: LLMs draft models, not code","Model-first AI: vibe modeling for reliable systems","Vibe modeling: AI designs the model, rules write the code","Vibe modeling: LLM-generated models, rule-based code","Vibe modeling: AI for models, rules for code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that an LLM agent can, through conversation, produce a model accurate enough for a non-expert to validate, and that a validated model is enough for deterministic code generation to yield reliable software.","fun_headline_variants_meta":{"raw":{"variants":["Vibe modeling: LLMs draft models, not code","Model-first AI: vibe modeling for reliable systems","Vibe modeling: AI designs the model, rules write the code","Vibe modeling: LLM-generated models, rule-based code","Vibe modeling: AI for models, rules for code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3385,"prompt_tokens":858,"completion_tokens":2527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2445}},"tokens_in":474,"tokens_out":2527,"duration_ms":22621,"temperature":1.0,"reasoning_tokens":2445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:01:13.861541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study in which non-expert users validate LLM-generated models and then the generated systems are tested: if a substantial fraction of accepted models lead to failing or unsafe generated code, the claim that vibe modeling yields reliable software would be refuted.","supporting_citations":[{"cited_title":"In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A","cited_arxiv_id":null,"evidence_quote":"It proposes a human-in-the-loop approach that refines LLM-generated process models with user feedback, the direct precursor of the vibe modeling loop."},{"cited_title":"Addison-Wesley Professional (2002)","cited_arxiv_id":null,"evidence_quote":"It establishes the idea of executable UML, the grounding for the claim that models can be deterministically compiled into running systems."}],"review_version":1}