pith. sign in

arxiv: 2606.24721 · v1 · pith:LGOF7XOUnew · submitted 2026-06-23 · 💻 cs.SE

Two-Level vs. Multi-Level Modelling: An Empirical Study of Cascading Maintenance Burden

Yuhong Fu , Weixing Zhang , Bowen Jiang , Haowei Cheng , Georg Grossmann , Karamjit Kaur , Matt Selway , Markus Stumptner This is my paper

Pith reviewed 2026-06-25 23:12 UTC · model grok-4.3

classification 💻 cs.SE
keywords multi-level modellingtwo-level modellingco-evolutionmodel-driven engineeringsoftware maintenanceempirical studymutation testingconsistency checking
0
0 comments X

The pith

Multi-level modelling yields fewer post-change inconsistencies and smaller modification footprints than two-level modelling in equivalent evolution scenarios.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether unifying domain knowledge in a single level reduces the cascading updates required when core definitions evolve. Two-level modelling splits knowledge across metamodel and model artefacts that must stay consistent, creating repeated co-evolution work. Multi-level modelling collapses these layers into one structure. The study builds semantically equivalent scenarios in both styles from a published corpus, applies identical mutations, and measures resulting inconsistencies plus edit size through automated checks and pre-registered tests. A blinded mapping protocol and positive controls are used to limit construction bias.

Core claim

The paper establishes that MLM's structural unification yields fewer post-change inconsistencies and a smaller modification footprint than 2LM for semantically equivalent evolution scenarios, demonstrated via a pre-registered, mutation-based comparison that applies identical changes to paired artefacts and evaluates outcomes with automated consistency checking.

What carries the argument

The blinded mapping protocol that produces semantically equivalent MLM counterparts from curated 2LM co-evolution scenarios, enabling direct paired comparison of inconsistency counts and modification footprints under identical mutations.

If this is right

  • MLM adoption in MDE projects can lower the measured cost of keeping artefacts consistent after definition changes.
  • The reusable benchmarking protocol allows direct comparison of other modelling paradigms on the same co-evolution metrics.
  • Automated consistency checking becomes a practical way to quantify maintenance burden across modelling styles.
  • Domains with frequent core-definition changes gain a concrete basis for preferring one structural organisation over another.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same paired-mutation design could be applied to compare MLM against other unification techniques such as aspect-oriented or view-based modelling.
  • If the reduction holds, tool builders could prioritise MLM support for domains where model evolution frequency is high.
  • The outcome variables (inconsistency count and footprint size) could serve as benchmarks for future language-design choices that affect co-evolution.

Load-bearing premise

The MLM versions constructed from the 2LM corpus are semantically equivalent to the originals and the mapping protocol plus controls remove bias in how equivalence and mutations are applied.

What would settle it

A replication study in which the pre-registered hypothesis tests show no statistically significant reduction in inconsistencies or modification size for the MLM versions after the same mutations.

read the original abstract

When a core definition changes, every dependent artefact must be updated, a cascading problem central to software maintenance. In Model-Driven Engineering (MDE), the dominant two-level modelling (2LM) paradigm fragments domain knowledge across metamodel and model artefacts that must be kept mutually consistent, making co-evolution a persistent source of inconsistencies and effort. Multi-level modelling (MLM) unifies these artefacts and is claimed to reduce co-evolution burden, but this has not been tested in a controlled, paired comparison against 2LM. We hypothesise that MLM's structural unification yields fewer post-change inconsistencies and a smaller modification footprint than 2LM for semantically equivalent evolution scenarios. To test this, we present a pre-registered, mutation-based empirical comparison of co-evolution behaviour in both paradigms. From a curated corpus of published 2LM co-evolution scenarios, we construct semantically equivalent MLM counterparts, apply identical evolution mutations to both, and measure outcomes through automated consistency checking and pre-registered hypothesis tests. Positive controls and a blinded mapping protocol guard against bias. This design provides the first empirical framework for assessing whether paradigm-level structural choices affect cascading maintenance burden, operationalising co-evolution burden as two automatically measurable outcome variables and delivering a reusable benchmarking protocol for replication and extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents a pre-registered, mutation-based empirical study comparing cascading maintenance burden between two-level modelling (2LM) and multi-level modelling (MLM). From a curated corpus of published 2LM co-evolution scenarios, the authors construct semantically equivalent MLM counterparts via a blinded mapping protocol with positive controls, apply identical evolution mutations to both paradigms, and measure post-mutation inconsistencies and modification footprint using automated consistency checking and pre-registered hypothesis tests. The central hypothesis is that MLM's structural unification produces fewer inconsistencies and a smaller modification footprint than 2LM for equivalent scenarios.

Significance. If the results hold, the work would supply the first controlled, paired empirical evidence on whether paradigm-level structural choices affect co-evolution burden in MDE, operationalising the outcome via two automatically measurable variables and delivering a reusable benchmarking protocol. The pre-registered design, external corpus, automated checks, and positive controls are genuine strengths that raise the evidential bar above typical claims in the area.

major comments (2)
  1. [Methods] Methods (mapping protocol subsection): The claim that MLM counterparts are 'semantically equivalent' to the original 2LM scenarios is load-bearing for the paired comparison, yet the description supplies only high-level statements about the blinded protocol and positive controls. No concrete criteria are given for verifying preservation of instance-level semantics, conformance relations, or cross-artefact invariants; without these, measured differences could arise from construction artefacts rather than the 2LM-vs-MLM distinction.
  2. [Results] Results (hypothesis-test tables): The abstract states that automated consistency checking and pre-registered tests are used, but the manuscript must report the exact operational definitions of 'inconsistency' and 'modification footprint' (including how mutations are rendered identical across paradigms) and the raw counts or effect sizes; otherwise the statistical claims cannot be evaluated for robustness against the equivalence assumption.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'semantically equivalent evolution scenarios' is repeated without a forward reference to the precise equivalence criteria that will be defined later; a parenthetical pointer would improve readability.
  2. [Methods] The paper should include a short table summarising the corpus size, number of scenarios, and mutation types to allow quick assessment of statistical power.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognising the strengths of the pre-registered design, external corpus, automated checks, and positive controls. We address each major comment below and will revise the manuscript accordingly to improve clarity and transparency.

read point-by-point responses

  1. Referee: [Methods] Methods (mapping protocol subsection): The claim that MLM counterparts are 'semantically equivalent' to the original 2LM scenarios is load-bearing for the paired comparison, yet the description supplies only high-level statements about the blinded protocol and positive controls. No concrete criteria are given for verifying preservation of instance-level semantics, conformance relations, or cross-artefact invariants; without these, measured differences could arise from construction artefacts rather than the 2LM-vs-MLM distinction.

    Authors: We agree that the mapping protocol description is currently high-level and that explicit verification criteria are needed to support the semantic-equivalence claim. In the revised manuscript we will expand the Mapping Protocol subsection with concrete, operational criteria (derived from the positive controls) that specify how instance-level semantics, conformance relations, and cross-artefact invariants are checked and preserved during the blinded mapping. These additions will allow readers to confirm that observed differences stem from the 2LM-vs-MLM distinction rather than mapping artefacts. revision: yes

  2. Referee: [Results] Results (hypothesis-test tables): The abstract states that automated consistency checking and pre-registered tests are used, but the manuscript must report the exact operational definitions of 'inconsistency' and 'modification footprint' (including how mutations are rendered identical across paradigms) and the raw counts or effect sizes; otherwise the statistical claims cannot be evaluated for robustness against the equivalence assumption.

    Authors: We concur that the manuscript should provide explicit operational definitions and supporting data for full evaluability. The revised version will include a new subsection that states the precise definitions of inconsistency (as flagged by the automated checker) and modification footprint (as the minimal set of post-mutation changes), together with the protocol used to render mutations identical across paradigms. Raw counts and effect sizes will be added to the hypothesis-test tables (or supplied as supplementary material) so that readers can assess robustness against the equivalence assumption. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical comparison

full rationale

The paper is a pre-registered empirical study that curates an external corpus of published 2LM co-evolution scenarios, constructs MLM counterparts via a blinded mapping protocol with positive controls, applies identical mutations, and measures post-change inconsistencies and modification footprint through automated checks and hypothesis tests. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citation chains appear in the abstract or described method. The central claim rests on independent, falsifiable measurements against an external corpus rather than reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The comparison rests on two domain assumptions: that a curated corpus of published 2LM scenarios is representative of typical MDE maintenance burdens, and that semantically equivalent MLM counterparts can be constructed without introducing structural bias. No free parameters or invented entities are introduced.

axioms (2)
  • domain assumption The curated corpus of published 2LM co-evolution scenarios is representative of typical maintenance burdens in MDE.
    The study selects this corpus as the basis for constructing paired MLM scenarios and generalising findings.
  • ad hoc to paper Semantically equivalent MLM counterparts can be constructed from 2LM scenarios without introducing structural advantages or disadvantages.
    This equivalence is required for the paired comparison to isolate the effect of the modelling paradigm.

pith-pipeline@v0.9.1-grok · 5784 in / 1298 out tokens · 23476 ms · 2026-06-25T23:12:58.386971+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references

  1. [1]

    Programs, life cycles, and laws of software evolution,

    M. Lehman, “Programs, life cycles, and laws of software evolution,” Proceedings of the IEEE, vol. 68, no. 9, pp. 1060–1076, 1980

    1980

  2. [2]

    Quantifying schema evolution,

    D. I. K. Sjøberg, “Quantifying schema evolution,”Inf. Softw. Technol., vol. 35, no. 1, pp. 35–44, 1993

    1993

  3. [3]

    How do developers react to API deprecation?: the case of a smalltalk ecosystem,

    R. Robbes, M. Lungu, and D. R ¨othlisberger, “How do developers react to API deprecation?: the case of a smalltalk ecosystem,” inSIGSOFT 2012. ACM, 2012, p. 56

    2012

  4. [4]

    How do apis evolve? A story of refactoring,

    D. Dig and R. E. Johnson, “How do apis evolve? A story of refactoring,” J. Softw. Maintenance Res. Pract., vol. 18, no. 2, pp. 83–107, 2006

    2006

  5. [5]

    Guest editor’s introduction: Model-driven engineering,

    D. C. Schmidt, “Guest editor’s introduction: Model-driven engineering,” Computer, vol. 39, no. 2, pp. 25–31, 2006

    2006

  6. [6]

    Brambilla, J

    M. Brambilla, J. Cabot, and M. Wimmer,Model-Driven Software Engineering in Practice, Second Edition, ser. Synthesis Lectures on Software Engineering. Morgan & Claypool Publishers, 2017

    2017

  7. [7]

    Automating co- evolution in model-driven engineering,

    A. Cicchetti, D. D. Ruscio, R. Eramo, and A. Pierantonio, “Automating co- evolution in model-driven engineering,” inECOC 2008. IEEE Computer Society, 2008, pp. 222–231

    2008

  8. [8]

    COPE - automating coupled evolution of metamodels and models,

    M. Herrmannsdoerfer, S. Benz, and E. J¨ urgens, “COPE - automating coupled evolution of metamodels and models,” inProc. of ECOOP 2009, ser. LNCS. Springer, 2009, pp. 52–76

    2009

  9. [9]

    A semi-automatic maintenance and co-evolution of OCL constraints with (meta)model evolution,

    D. E. Khelladi, R. Bendraou, R. Hebig, and M. Gervais, “A semi-automatic maintenance and co-evolution of OCL constraints with (meta)model evolution,”J. Syst. Softw., vol. 134, pp. 242–260, 2017

    2017

  10. [10]

    When and how to use multilevel modelling,

    J. de Lara, E. Guerra, and J. S. Cuadrado, “When and how to use multilevel modelling,”ACM Trans. Softw. Eng. Methodol., vol. 24, no. 2, pp. 12:1– 12:46, 2014

    2014

  11. [11]

    Multilevel modeling - toward a new paradigm of conceptual modeling and information systems design,

    U. Frank, “Multilevel modeling - toward a new paradigm of conceptual modeling and information systems design,”Bus. Inf. Syst. Eng., vol. 6, no. 6, pp. 319–337, 2014

    2014

  12. [12]

    A conceptual framework for large-scale ecosystem interoper- ability and industrial product lifecycles,

    M. Selway, M. Stumptner, W. Mayer, A. Jordan, G. Grossmann, and M. Schrefl, “A conceptual framework for large-scale ecosystem interoper- ability and industrial product lifecycles,”Data Knowl. Eng., vol. 109, pp. 85–111, 2017

    2017

  13. [13]

    Multi-level risk modelling for interoperability of risk information,

    Y. Fu, G. Grossmann, K. Kaur, M. Selway, and M. Stumptner, “Multi-level risk modelling for interoperability of risk information,” inProc. of IN4PL

  14. [14]

    SCITEPRESS, 2022, pp. 242–249

    2022

  15. [15]

    Towards the integration of multi-level and multi-view modelling for interoperability,

    ——, “Towards the integration of multi-level and multi-view modelling for interoperability,” inMODELS 2023 Companion (Proc. MULTI Workshop). IEEE, 2023, pp. 679–688

    2023

  16. [16]

    Sup- porting meta-model-based language evolution and rapid prototyping with automated grammar transformation,

    W. Zhang, J. Holtmann, D. Str¨ uber, R. Hebig, and J.-P. Stegh¨ofer, “Sup- porting meta-model-based language evolution and rapid prototyping with automated grammar transformation,”Journal of Systems and Software, vol. 214, p. 112069, 2024

    2024

  17. [17]

    Modelling a warehouse with SLICER: A contribution to the MULTI warehouse challenge,

    Y. Fu, M. Selway, G. Grossmann, K. Kaur, and M. Stumptner, “Modelling a warehouse with SLICER: A contribution to the MULTI warehouse challenge,” inMODELS 2024 Companion (Proc. MULTI Workshop). ACM, 2024, pp. 828–837

    2024

  18. [18]

    Flexible deep modeling with melanee,

    C. Atkinson and R. Gerbig, “Flexible deep modeling with melanee,” in Modellierung 2016, 2.-4. M ¨arz 2016, Karlsruhe - Workshopband, ser. LNI. GI, 2016, pp. 117–122

    2016

  19. [19]

    Multecore: Combining the best of fixed-level and multilevel metamodelling,

    F. Mac ´ıas, A. Rutle, and V. Stolz, “Multecore: Combining the best of fixed-level and multilevel metamodelling,” inMODELS 2016 Companion (Proc. of MULTI Workshop), ser. CEUR Workshop Proceedings. CEUR- WS.org, 2016, pp. 66–75

    2016

  20. [20]

    Deeptelos: Multi-level modeling with most general instances,

    M. A. Jeusfeld and B. Neumayr, “Deeptelos: Multi-level modeling with most general instances,” inProc. of ER 2016, ser. LNCS, 2016, pp. 198– 211

    2016

  21. [21]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,”Biometrics, vol. 1, pp. 196–202, 1945. [Online]. Available: https://api.semanticscholar. org/CorpusID:53662922

    1945

  22. [22]

    An extensive catalog of operators for the coupled evolution of metamodels and models,

    M. Herrmannsdoerfer, S. Vermolen, and G. Wachsmuth, “An extensive catalog of operators for the coupled evolution of metamodels and models,” inSLE 2010, ser. LNCS. Springer, 2010, pp. 163–182

    2010

  23. [23]

    Language evolution in practice: The history of GMF,

    M. Herrmannsdoerfer, D. Ratiu, and G. Wachsmuth, “Language evolution in practice: The history of GMF,” inSLE 2009, ser. LNCS. Springer, 2009, pp. 3–22

    2009

  24. [24]

    Approaches to co-evolution of metamodels and models: A survey,

    R. Hebig, D. E. Khelladi, and R. Bendraou, “Approaches to co-evolution of metamodels and models: A survey,”IEEE Trans. Software Eng., vol. 43, no. 5, pp. 396–414, 2017

    2017

  25. [25]

    On the impact significance of metamodel evolution in MDE,

    L. Iovino, A. Pierantonio, and I. Malavolta, “On the impact significance of metamodel evolution in MDE,”J. Object Technol., vol. 11, no. 3, pp. 3: 1–33, 2012

    2012

  26. [26]

    Development and evolution of xtext- based dsls on github: an empirical investigation,

    W. Zhang, D. Str¨ uber, and R. Hebig, “Development and evolution of xtext- based dsls on github: an empirical investigation,”Empirical Software Engineering, vol. 31, no. 3, p. 48, 2026

    2026

  27. [27]

    Conflict management for multi-level models in collaborative modelling environ- ments,

    Y. Fu, G. Grossmann, K. Kaur, M. Selway, and M. Stumptner, “Conflict management for multi-level models in collaborative modelling environ- ments,” inMODELS 2025 Companion. IEEE, 2025, pp. 502–511

    2025

  28. [28]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: http://www.jstor.org/stable/4615733

    arXiv 1979

  29. [29]

    Cohen,Statistical Power Analysis for the Behavioral Sciences, 2nd ed

    J. Cohen,Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988

    1988

  30. [30]

    A systematic review of effect size in software engineering experiments,

    V. B. Kampenes, T. Dyb˚a, J. E. Hannay, and D. I. K. Sjøberg, “A systematic review of effect size in software engineering experiments,”Inf. Softw. Technol., vol. 49, no. 11-12, pp. 1073–1086, 2007

    2007

  31. [31]

    Lever- aging llms to support co-evolution between definitions and instances of textual dsls: A systematic evaluation,

    W. Zhang, B. Jiang, Y. Fu, A. Koziolek, R. Hebig, and D. Str¨ uber, “Lever- aging llms to support co-evolution between definitions and instances of textual dsls: A systematic evaluation,”arXiv preprint arXiv:2602.11904, 2026. The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports

    arXiv 2026