Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Vibe Modeling: Challenges and Opportunities

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces 'vibe modeling' as a development method where an LLM converses with the user to produce validated software models, and deterministic rule-based generators then turn those models into running code.

desk verdict A clear, honest vision paper that names a real integration opportunity, but the reliability promise rests on an untested assumption about non-expert model validation; worth a serious referee with revisions. read the letter →

arxiv 2507.23120 v1 pith:XWOCT4J4 submitted 2025-07-30 cs.SE

Jordi Cabot This is my paper · ORCID
classification cs.SE
keywords VibeModelingLow-modelingLow-codeDSLArtificialIntelligenceModel-drivenCoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces 'vibe modeling' as a software development method that sits between vibe coding and classical model-driven engineering. Instead of asking an LLM to write code directly, the user converses with an agent that produces and refines a formal model of the system; once the model is validated, rule-based generators compile it into the final software. The author argues this gives the speed and natural-language accessibility of vibe coding while keeping the understandability and reliability of model-based development. The paper lays out the development process, collaborative multi-agent extensions, the infrastructure needed, and a roadmap of open challenges. A sympathetic reader would care because it proposes a concrete way to make AI-assisted development safer for non-experts and to revive the role of modeling in modern toolchains.

What carries the argument

The carrying mechanism is the two-stage vibe modeling pipeline: a conversational LLM agent, guided by a domain expert, iteratively produces and validates a model, and then a deterministic rule-based generator compiles that model into code. The Model Context Protocol (MCP) is the supporting infrastructure: it standardizes how agents discover and call modeling-platform services, so any agent can create, validate, and manipulate models without embedding the modeling stack. The paper also introduces the notion that models carry uncertainty and traceability, with confidence scores and change proposals attached to model elements.

What would settle it

A controlled study in which non-expert users validate LLM-generated models and then the generated systems are tested: if a substantial fraction of accepted models lead to failing or unsafe generated code, the claim that vibe modeling yields reliable software would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that 'vibe modeling' — building software by conversing with an LLM trained for modeling rather than coding, then generating the final system with deterministic rule-based code generators — can combine the strengths of AI and model-driven engineering. The author argues that because the LLM's output is a model rather than code, users can understand and validate it without deep programming expertise, and because code generation is deterministic, a validated model reliably yields good code. The paper further claims this makes vibe modeling useful to both technical and non-technical experts and positions it as a model-driven alternative to vibe coding. It supports the proposal with a description of the iterative human-agent modeling loop and a proof-of-concept integration of a modeling platform with an agent through the Model Context Protocol.

Load-bearing premise

The approach assumes that an LLM agent can, through conversation, produce a model accurate enough for a non-expert to validate, and that a validated model is enough for deterministic code generation to yield reliable software.

Editorial extensions

If this is right

  • Non-programmers could build reliable systems by validating a model instead of trusting generated code they cannot read.
  • Because code generation is deterministic, regenerating the system after a model change would not require re-verifying the generated code each time.
  • Vibe modeling would restore conceptual modeling to a place in the AI-assisted development workflow, countering the trend toward direct code generation.
  • Standardizing modeling services as MCP servers would let any LLM agent work with any modeling platform, avoiding vendor-specific agent integrations.
  • In collaborative settings, a community of specialized agents could produce better models than a single agent, with a consensus mechanism selecting among competing proposals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run: a head-to-head benchmark comparing vibe modeling against vibe coding on defect density, maintainability, and how well users catch errors in the artifact they review.
  • The paper's traceability requirement implies that modeling platforms will need new versioning and provenance standards for model elements; this is left to future work.
  • If non-experts can validate models reliably, vibe modeling could carry low-code development into domains like citizen data collection where users have no coding training; the paper does not discuss such applications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes 'vibe modeling' as a development approach in which an LLM-based agent interacts conversationally with a domain expert to produce a conceptual model, and the final system is then generated by deterministic, rule-based code generators. Section 3 defines the concept and argues for two advantages over vibe coding: models are more understandable and therefore validatable by non-programmers, and deterministic generation makes code quality depend only on model quality. Section 4 extends the idea to multi-agent collaboration; Section 5 proposes using the Model Context Protocol (MCP) to connect agents to modeling platforms and reports a small prototype MCP server for the BESSER low-code platform; Section 6 lists open challenges. The paper is explicitly a position/vision paper, with no empirical evaluation of the proposed workflow.

Significance. As a research agenda, the paper names a real gap: the modeling community has studied LLM-based model inference mostly as one-shot prompting, while the code-generation community has adopted vibe coding without addressing reliability. The proposal to separate conversational model construction from deterministic generation is a plausible and potentially valuable synthesis. The paper also deserves credit for being candid in Section 6 about open challenges, including model-quality evaluation, agent training data, and traceability. However, the central claim that vibe modeling yields 'reliable complex systems' for non-experts is not established: it rests on the assumptions that non-experts can validate LLM-produced models and that a good model is sufficient for reliable code. These assumptions are exactly the ones left open in Section 6.1. The paper is therefore more a proposal than a demonstrated result, and its strengths are in framing and infrastructure ideas rather than evidence.

major comments (4)
  1. [Section 3] The 'Reliable code-generation' advantage is load-bearing but is stated as 'If the model is good, we know the code will be good,' and the paper provides no operational criterion for 'good.' Section 6.1 explicitly lists 'How to evaluate the quality of the models inferred by the agents?' as an open challenge, and Section 6.3 only proposes storing a confidence score, not a validation method. As written, the reliability guarantee holds only when the hard part of the problem (knowing that the model is correct) has already been solved. I recommend reformulating this advantage as a research hypothesis and adding to Section 6.1 a concrete sketch of candidate quality criteria, for example consistency checking, scenario-based validation, or comparison against a manually built reference model for a benchmark domain.
  2. [Section 3] The 'Understandable output' advantage is in tension with the evidence the paper itself cites in Section 2. Camara et al. [8] found that LLM-generated UML class diagrams had the worst results exactly when abstraction was required (using inheritance instead of attributes, or creating association classes), and several cited works ([12,29,31]) report that identifying relationships among classes remains challenging. These are precisely the kinds of errors that a non-expert is least likely to detect by inspection. The section should either weaken the claim that non-experts can validate the output 'even without coding expertise,' or propose concrete validation aids (model smells, automated consistency checks, or executable examples generated from the model) that would make such validation plausible.
  3. [Section 3] The paper asserts that code-generation templates need only be verified 'once and for all,' but for reliable complex systems, generator correctness is itself a nontrivial assurance problem. The paper does not indicate what 'verified once and for all' means (certified compilers, test suites, formal verification, or something weaker). This matters because the reliability claim shifts all residual risk onto the model, and the manuscript should either specify the intended level of rigor or soften the claim to avoid the impression that deterministic generation by itself guarantees dependable code.
  4. [Section 5] The BESSER MCP server example in Listing 1.1 only exposes a function that creates a new, empty DomainModel and returns it serialized. This does not demonstrate the core vibe-modeling loop shown in Figure 1, which consists of conversational model construction, validation, refinement, and eventual code generation. I suggest either extending the prototype to illustrate at least one complete refinement-and-validation cycle, or explicitly labeling the current implementation as a first building block and stating that the remaining steps are future work. The current wording ('Example of creating a new model via a service exposed by the BESSER MCP Server') is accurate but does not support the broader feasibility claims made elsewhere in the paper.
minor comments (6)
  1. [Abstract] The phrase 'at the expenses of' should be 'at the expense of.'
  2. [Section 5] In the sentence beginning 'Therefore, We propose to use MCP...,' the 'W' in 'We' should be lowercase.
  3. [Section 6.2] The phrase 'the limitations of LLMS' should be 'the limitations of LLMs.'
  4. [Section 6.1] The introductory phrase 'Aspects like:' is followed by a bullet list of sentence fragments; consider rewriting each bullet as a full question to make the open challenges easier to reference and discuss.
  5. [Keywords] The keyword 'Low-modeling' does not appear in the body of the paper; either introduce the term in the text or remove it from the keyword list.
  6. [Section 4] The phrase 'the more the merrier' is colloquial for a technical paper; consider rephrasing to something like 'a larger number of diverse agents may lead to a better final model.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a definitional, position-style proposal with no fitted parameters, quantitative predictions, or derivation chain that collapses into its own inputs.

full rationale

The paper introduces 'vibe modeling' as a proposed development process; it does not derive a quantitative result from data, from a fitted value, or from an imported uniqueness theorem. The claimed 'reliable code-generation' advantage is explicitly conditional ('If the model is good, we know the code will be good'), and the paper itself lists model-quality evaluation as an open challenge in Section 6.1, so the guarantee is a stated premise rather than a circularly validated conclusion. The proof-of-concept MCP server for BESSER is an implementation sketch; the citation to BESSER is contextual and does not carry the argument. Other author self-citations (e.g., low-modeling, Collaboro, ModelSET) appear in related-work or roadmap context and are not load-bearing. There is no equation, renamed result, or fitted input whose output equals the input, so no circular step can be quoted under the required standard; the untested premise that non-experts can validate models is an evidence/correctness concern, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The central proposal rests on the feasibility of LLM-driven model inference and on deterministic generation from validated models. Neither is measured; the paper itself delegates most of the needed evidence to future work.

assumptions (3)
  • domain assumption An LLM tuned for modeling, after conversational refinement, can produce models that are good enough for non-expert validation.
    Section 3 and Section 6.1; the paper lists how to train and evaluate such agents as open challenges, so this is assumed for the central process.
  • domain assumption Rule-based deterministic code generation from a good model yields good code without per-generation checks.
    Section 3 second bullet; this assumes the code-generation templates are verified once and for all, with no residual risk.
  • domain assumption Non-technical users can validate models more easily than code.
    Section 3 first bullet; plausible but unmeasured.
invented entities (2)
  • vibe modeling
    purpose: To describe a workflow where LLM agents produce models, then deterministic generators produce code.
    Concept introduced in this paper; no empirical validation, only a proof-of-concept in progress.
  • modeling agents
    purpose: To perform conversational model inference and refinement on behalf of users.
    Described in Sections 4 and 6.1; no implementation beyond the BESSER MCP example.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vibe Modeling: Challenges and Opportunities." pith.science (2026). https://pith.science/paper/XWOCT4J4

@misc{pith2026250723120,
  author       = {Pith},
  title        = {Pith review of: Vibe Modeling: Challenges and Opportunities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWOCT4J4}},
  note         = {Machine review of arXiv:2507.23120}
}
read the original abstract

There is a pressing need for better development methods and tools to keep up with the growing demand and increasing complexity of new software systems. New types of user interfaces, the need for intelligent components, sustainability concerns, ... bring new challenges that we need to handle. In the last years, model-driven engineering (MDE) has been key to improving the quality and productivity of software development, but models themselves are becoming increasingly complex to specify and manage. At the same time, we are witnessing the growing popularity of vibe coding approaches that rely on Large Language Models (LLMs) to transform natural language descriptions into running code at the expenses of code vulnerabilities, scalability issues and maintainability concerns. In this paper, we introduce the concept of \textit{vibe modeling} as a novel approach to integrate the best of both worlds (AI and MDE) to speed up the development of reliable complex systems. We outline the key concepts of vibe modeling and highlight the opportunities and open challenges it presents for the future of modeling.

Figures

Figures reproduced from arXiv: 2507.23120 by the authors.

Figure 1
Figure 1. The vibe modeling process as part of a low-code architecture. The question [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview of the AI Agent community in the Mosaico EU project [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. MCP for modeling interactions [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Multi-agent and multi-tool collaboration facilitated by the use of MCP [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example of creating a new model via a service exposed by the BESSER [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages

  1. [8]

    Cámara, J., Troya, J., Burgueño, L., Vallecillo, A.: On the assessment of generative AI in modeling tasks: an experience report with chatgpt and UML. Softw. Syst. Model. 22(3), 781–793 (2023). https://doi.org/10.1007/s10270-023-01105-5

    doi:10.1007/s10270-023-01105-5 2023

  2. [1]

    In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A

    Alfonso, I., Conrardy, A.D., Sulejmani, A., Nirumand, A., Haq, F.U., Gomez- Vazquez, M., Sottet, J., Cabot, J.: Building BESSER: an open-source low-code platform. In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A. (eds.) Enterprise, Business-Process and Information Systems Modeling - 25th International Confer- ence, BPMDS 2024, and 29th International Co...

    2024

  3. [2]

    35(5),108–114(2018)

    Booch,G.:Thehistoryofsoftwareengineering.IEEESoftw. 35(5),108–114(2018). https://doi.org/10.1109/MS.2018.3571234

    arXiv 2018

  4. [3]

    Burgueño, L., Cabot, J., Wimmer, M., Zschaler, S.: Guest editorial to the theme section on ai-enhanced model-driven engineering. Softw. Syst. Model.21(3), 963– 965 (2022). https://doi.org/10.1007/s10270-022-00988-0

    doi:10.1007/s10270-022-00988-0 2022

  5. [4]

    CoRRabs/2405.18539 (2024)

    Burgueño, L., Ruscio, D.D., Sahraoui, H.A., Wimmer, M.: The past, present, and future of automation in model-driven engineering. CoRRabs/2405.18539 (2024). https://doi.org/10.48550/ARXIV.2405.18539

    arXiv doi:10.48550/arxiv.2405.18539 2024

  6. [5]

    In: Fill, H., Mayo, F.J.D., van Sinderen, M., Maciaszek, L.A

    Cabot, J.: Low-modeling of software systems. In: Fill, H., Mayo, F.J.D., van Sinderen, M., Maciaszek, L.A. (eds.) Software Technologies - 18th Interna- tional Conference, ICSOFT 2023, Rome, Italy, July 10-12, 2023, Revised Se- lected Papers. Communications in, vol. 2104, pp. 19–28. Springer (2023). https://doi.org/10.1007/978-3-031-61753-9_2

    doi:10.1007/978-3-031-61753-9_2 2023

  7. [6]

    Self-published, 1st edn

    Cabot, J.: The low-code handbook: Learn how to unlock faster and better soft- ware development with low-code solutions. Self-published, 1st edn. (October 2024), https://lowcode-book.com/

    2024

  8. [7]

    IEEE Softw

    Cabot, J., Clarisó, R.: Low code for smart software development. IEEE Softw. 40(1), 89–93 (2023). https://doi.org/10.1109/MS.2022.3211352

    arXiv 2023

Show all 32 references
  1. [9]

    ACM Trans

    Cañizares, P.C., López-Morales, J.M., Pérez-Soler, S., Guerra, E., de Lara, J.: Measuring and clustering heterogeneous chatbot designs. ACM Trans. Softw. Eng. Methodol. 33(4), 90:1–90:43 (2024). https://doi.org/10.1145/3637228

    2024 doi

  2. [10]

    In: 2023 IEEE/ACM 45th In- ternational Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)

    Chaaben, M.B., Burgueño, L., Sahraoui, H.: Towards using Few-Shot Prompt Learning for Automating Model Completion. In: 2023 IEEE/ACM 45th In- ternational Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). pp. 7–12. IEEE, Melbourne, Australia (May 2...

    2023

  3. [11]

    In: 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS)

    Chen, K., Yang, Y., Chen, B., Hernández López, J.A., Mussbacher, G., Varró, D.: AutomatedDomainModelingwithLargeLanguageModels:AComparativeStudy. In: 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS). pp. 162–172. IEEE, Väst...

    2023

  4. [12]

    arXiv preprint arXiv:2410.09854 (2024)

    Chen, R., Shen, J., He, X.: A model is not built by a single prompt: Llm-based domain modeling with question decomposition. arXiv preprint arXiv:2410.09854 (2024)

    2024 arXiv

  5. [13]

    CoRR abs/2404.09919 (2024)

    d’Aloisio, G., Sipio, C.D., Marco, A.D., Ruscio, D.D.: How fair are we? from conceptualization to automated assessment of fairness definitions. CoRR abs/2404.09919 (2024). https://doi.org/10.48550/ARXIV.2404.09919

    2024 doi arXiv

  6. [14]

    In: Grabis, J., Vos, T.E.J., Escalona, M.J., Pastor, O

    Daoudi, N., Alfonso, I., Cabot, J.: Modelling neural network models. In: Grabis, J., Vos, T.E.J., Escalona, M.J., Pastor, O. (eds.) Research Challenges in Informa- tion Science - 19th International Conference, RCIS 2025, Seville, Spain, May 20- 23, 2025, Proceedings, Part II. ...

    2025 doi

  7. [15]

    Enterprise Modelling Vibe Modeling: Challenges and Opportunities 13 and Information Systems Architectures (EMISAJ) pp

    Fill, H.G., Fettke, P., Köpke, J.: Conceptual Modeling and Large Language Mod- els: Impressions From First Experiments With ChatGPT. Enterprise Modelling Vibe Modeling: Challenges and Opportunities 13 and Information Systems Architectures (EMISAJ) pp. 3:1–15 Pages (Apr 2023). ...

    2023 doi

  8. [16]

    arXiv preprint arXiv:2402.01680 (2024)

    Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N.V., Wiest, O., Zhang, X.: Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024)

    2024 arXiv

  9. [17]

    ACM Trans

    He, J., Treude, C., Lo, D.: Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Trans. Softw. Eng. Methodol. 34(5) (May 2025). https://doi.org/10.1145/3712003

    2025 doi

  10. [18]

    Hutchinson, J.E., Whittle, J., Rouncefield, M.: Model-driven engineering practices in industry: Social, organizational and managerial factors that lead to success or failure. Sci. Comput. Program. 89, 144–161 (2014). https://doi.org/10.1016/j.scico.2013.03.017

    2014 doi

  11. [19]

    PeerJ Computer Science 2, e84 (2016)

    Izquierdo, J.L.C., Cabot, J.: Collaboro: a collaborative (meta) modeling tool. PeerJ Computer Science 2, e84 (2016)

    2016

  12. [20]

    In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: Process modeling with large language models. In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A. (eds.) Enterprise, Business-Process and Information Systems Modeling. pp. 229–

  13. [21]

    López, J.A.H., Izquierdo, J.L.C., Cuadrado, J.S.: Modelset: a dataset for machine learning in model-driven engineering. Softw. Syst. Model.21(3), 967–986 (2022). https://doi.org/10.1007/s10270-021-00929-3

    2022 doi

  14. [22]

    Addison-Wesley Professional (2002)

    Mellor, S.J., Balcer, M.J.: Executable UML: a foundation for model-driven archi- tecture. Addison-Wesley Professional (2002)

    2002

  15. [23]

    In: Egyed, A., Wimmer, M., Chechik, M., Combemale, B

    Morales, S., Clarisó, R., Cabot, J.: A DSL for testing llms for fairness and bias. In: Egyed, A., Wimmer, M., Chechik, M., Combemale, B. (eds.) Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Lan- guages and Systems, MODELS 2024, Linz, Aus...

    2024

  16. [24]

    Planas, E., Daniel, G., Brambilla, M., Cabot, J.: Towards a model-driven approach for multiexperience AI-based user interfaces. Softw. Syst. Model.20(4), 997–1009 (2021)

    2021

  17. [25]

    In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)

    Rahimi, M., Guo, J.L., Kokaly, S., Chechik, M.: Toward requirements spec- ification for machine-learned components. In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). pp. 241–244 (2019). https://doi.org/10.1109/REW.2019.00049

    2019

  18. [26]

    Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., Tonella, P.: Testing machine learning based systems: a systematic mapping. Empir. Softw. Eng. 25(6), 5193–5254 (2020). https://doi.org/10.1007/s10664-020-09881-0

    2020 doi

  19. [27]

    Rocco, J.D., Ruscio, D.D., Sipio, C.D., Nguyen, P.T., Rubei, R.: On the use of large language models in model-driven engineering. Softw. Syst. Model.24(3), 923–948 (2025). https://doi.org/10.1007/S10270-025-01263-8

    2025 doi

  20. [28]

    Ruscio, D.D., Kolovos, D.S., de Lara, J., Pierantonio, A., Tisi, M., Wimmer, M.: Low-code development and model-driven engineering: Two sides of the same coin? Softw. Syst. Model. 21(2), 437–446 (2022). https://doi.org/10.1007/S10270-021- 00970-2

    2022 doi

  21. [29]

    In: Maass, W., Han, H., Yasar, H., Multari, N.J

    Silva, J., Ma, Q., Cabot, J., Kelsen, P., Proper, H.A.: Application of the tree- of-thoughts framework to llm-enabled domain modeling. In: Maass, W., Han, H., Yasar, H., Multari, N.J. (eds.) Conceptual Modeling - 43rd International Conference, ER 2024, Pittsburgh, PA, USA, Oct...

    2024 doi

  22. [30]

    Troya, J., Moreno, N., Bertoa, M.F., Vallecillo, A.: Uncertainty representation in software models: a survey. Softw. Syst. Model. 20(4), 1183–1213 (2021). https://doi.org/10.1007/s10270-020-00842-1

    2021 doi

  23. [31]

    In: 2024 IEEE International Conference on Software Services Engineering (SSE)

    Wang, B., Wang, C., Liang, P., Li, B., Zeng, C.: How llms aid in uml modeling: an exploratory study with novice analysts. In: 2024 IEEE International Conference on Software Services Engineering (SSE). pp. 249–257. IEEE (2024)

    2024

  24. [244]

    Springer Nature Switzerland, Cham (2024)

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.