Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An extended reference architecture for ML-enabled systems is proposed as the first step toward measuring their architectural complexity.

desk verdict A clean, honest research proposal whose only concrete result is a two-subsystem extension of an existing MLOps reference architecture, with a validation claim that is asserted but not shown. read the letter →

arxiv 2506.08153 v1 pith:KXHTSKLT submitted 2025-06-09 cs.SE cs.AIcs.LG

Renato Cordeiro Ferreira (1 , 2 , 3 , 4) ((1) University of São Paulo , (2) Jheronimus Academy of Data Science , (3) Technical University of Eindhoven , (4) Tilburg University) This is my paper
classification cs.SEcs.AIcs.LG
keywords ML-enabledsystemssoftwarecomplexitymetricsreferencearchitectureMLOpsdataacquisitioncontinuousdeliveryarchitecturalmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the complexity of machine-learning-enabled systems is architectural and can be managed by measuring it. Its concrete contribution is an extension of an existing MLOps reference architecture with two new subsystems: data acquisition and continuous delivery. The resulting six-subsystem model is meant to describe any ML-enabled system so that complexity metrics can later be attached to its architectural parts. If this step holds, it gives researchers and practitioners a shared frame for locating where complexity enters these systems, and a foundation for a fuller metrics-based model.

What carries the argument

The key object is the extended reference architecture: a diagram that maps an ML-enabled system's components into applications/services, pipelines, and storage, arranged into six subsystems and connected by execution-flow and data-flow arrows. The addition of DATA ACQUISITION and CONTINUOUS DELIVERY is the machinery that makes the model more complete than its base version. This architecture carries the argument because it provides the fixed inventory of architectural places where complexity metrics can later be collected.

What would settle it

Map the components of a diverse set of production ML-enabled systems onto the six subsystems and search for components that fit no subsystem or for data or execution flows the architecture cannot represent; any unplaceable component or unrepresentable flow would break the completeness claim.

Watch

Extended reading notes

Core claim

The paper's central claim is modest but load-bearing: a reference architecture that groups ML-enabled systems into six subsystems can describe these systems well enough to support future metric collection. The architecture distinguishes applications and services from pipelines and data storage, and links them through execution flow and data flow. The paper takes a previously published reference architecture and adds two subsystems, DATA ACQUISITION and CONTINUOUS DELIVERY, arguing that data-collection components were missing and some delivery components were ungrouped in the original. It then states that this extended architecture will be used as the descriptive foundation for a metrics-oriented model of ML-system complexity.

Load-bearing premise

The load-bearing premise is that the extended reference architecture is complete enough that every component where complexity can emerge in an ML-enabled system falls inside one of its six subsystems; if a system has important parts that fit nowhere, the later metric-collection model built on it will miss them.

Editorial extensions

If this is right

  • If the extended architecture is accepted, future work can describe an ML-enabled system by mapping its components onto the six subsystems before deciding what to measure.
  • Complexity metrics can be attached to architectural elements such as pipelines, storage, and services, rather than to code alone, which would capture data- and model-related complexity that code metrics miss.
  • The two added subsystems put data collection and continuous delivery on equal footing with training, serving, and monitoring, making them explicit targets for complexity analysis.
  • The architecture gives the future metrics-oriented model a repeatable structure, so metric collection can be reproduced across different systems and case studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The next testable step is whether metrics assigned to each subsystem actually separate simple from complex ML-enabled systems; the paper does not yet show such a separation.
  • If data acquisition is truly a distinct subsystem, then data-source variety, labeling effort, and data-drift handling may each deserve their own complexity metrics, extending the paper's architectural move into operational measures.
  • The continuous-delivery subsystem opens a route to measure deployment frequency, rollback rate, and pipeline automation, connecting architecture-driven complexity to the production-readiness concerns the paper cites.
  • A practical extension would use the six-subsystem architecture as an interview or survey instrument, asking engineering teams which subsystems concentrate complexity and checking whether the resulting metric collections agree.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that the complexity of machine learning-enabled systems (MLES) can be managed through a metrics-oriented architectural model, and presents what it claims is the first step toward that model: an extension of Kumara et al.'s MLOps reference architecture with two additional subsystems, DATA ACQUISITION and CONTINUOUS DELIVERY, shown in Figure 2. The manuscript also outlines a four-step research methodology (systematic literature review, ontology design, exploratory case study, confirmatory case study) that is planned but not yet executed, and lists expected results and threats to validity.

Significance. If the extended reference architecture were properly validated, it could serve as a useful descriptive framework for attaching complexity metrics to MLES components, addressing a recognized gap in the SE4AI literature. The paper correctly identifies that few metrics exist for MLES and that complexity management is an open challenge. However, the manuscript as submitted contains no empirical evidence, no demonstrated metric collection, and no completed case studies; its value is as a research proposal rather than a validated contribution.

major comments (4)
  1. [Section V, paragraph after Figure 2] The only support for the extended reference architecture is the statement, "This opportunity was also used to validate the extended reference architecture represented in Figure 2." No interview protocol, participant count, selection criteria, raw data, or analysis is reported. This is an assertion, not a testable validation. Because the completeness and placement of the six subsystems, especially the two added ones, are load-bearing for any later metric collection, the validation evidence must be presented or the claim must be weakened accordingly.
  2. [Abstract and Section V] The paper claims that the contribution is "an extension of a reference architecture that can describe MLES to collect their metrics," but it never demonstrates the architecture in use. No MLES is described using Figure 2, no metric is attached to any component, and no collection process is shown. The statement that the architecture "will be used to describe systems before collecting their metrics" is a plan, not a result; the central claim is therefore asserted rather than evidenced.
  3. [Title, Abstract, and Section I] The title and abstract promise a "metrics-oriented architectural model" that characterizes complexity, but the manuscript delivers only a reference-architecture extension plus a plan. The metrics themselves, the operationalization process, and the case studies are all deferred to future work (Figure 1 and Section IV). This overstates the achieved scope and should be corrected to match the actual contribution, or the missing components must be included.
  4. [Section V, Figure 2] The necessity and sufficiency of the two added subsystems are not justified. The text says DATA ACQUISITION addresses components "that were not addressed" and CONTINUOUS DELIVERY "associating some components that were not grouped" in Kumara et al., but no criteria are given for why these specific subsystems are the correct or complete additions, nor is there any comparison with other published MLOps reference architectures. A missing or misplaced subsystem would propagate into whatever metrics are later defined, so this gap is load-bearing.
minor comments (5)
  1. [Section VI, ER2] The text says "production systems from industry partners for step 2.1" in two places; this should read "step 2.2" to be consistent with Figure 1 and Section IV.B.2.
  2. [Section IV.A.2] The ontology-design paragraph appears twice in sequence ("Making an ontology results in..." followed by "Making an ontology can be created via..."), which looks like a copy-paste error that should be removed.
  3. [Figure 2] The figure legend is extremely dense and the labels (numbers 1-13, roman numerals I-VII, and letters A-L) are never explained in the text; a table mapping each label to its component name and subsystem would substantially improve readability.
  4. [Section IV.B.2] The subsection heading "Confirmatory Study Case" should be "Confirmatory Case Study" for consistency with standard terminology and the rest of the paper.
  5. [References] Reference [16] is described as "proposing a set of metrics to measure how automated is an MLES," but the actual title suggests a model-driven, metrics-based approach to assessing quality-aspect support in MLOps architectures; the description should be checked against the cited work for accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper proposes an extension of an externally grounded reference architecture and reports future work; no prediction reduces to fitted parameters or self-cited uniqueness claims.

full rationale

The paper's central contribution is Figure 2, an extension of Kumara et al.'s externally published reference architecture with two subsystems. The extension is not derived from the metric-collection goal; it is a proposal grounded in the Kumara et al. architecture and an unreported expert-validation opportunity. RQ1/RQ2 and the case studies are explicitly future work, so no metric is fitted and then predicted. The only potentially self-referential element is the phrase 'This opportunity was also used to validate the extended reference architecture represented in Figure 2' (Section V), which is an unsupported evidence claim rather than a circular reduction: the paper does not define DATA ACQUISITION or CONTINUOUS DELIVERY in terms of the validation, nor does it import a uniqueness theorem from self-authored work. The absence of interview protocol, participant data, and analysis is a validity and evidence concern, not circularity. Accordingly, the paper is self-contained with respect to circularity; score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The paper relies on a prior reference architecture and on a hypothesized relation between metrics and complexity. The two added subsystems are categories, not new entities with independent evidence.

assumptions (2)
  • domain assumption The Kumara et al. reference architecture is a valid and complete basis for describing MLES, except for the two subsystems added.
    The paper builds on this architecture without questioning its validity or completeness beyond the additions. This is a borrowing from prior literature, accepted as ground truth.
  • domain assumption The complexity of MLES can be characterized by software metrics collected over an architectural model.
    This is the overarching research hypothesis, stated in the abstract and research questions, but not yet demonstrated. The entire project depends on this premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems." pith.science (2026). https://pith.science/paper/KXHTSKLT

@misc{pith2026250608153,
  author       = {Pith},
  title        = {Pith review of: A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXHTSKLT}},
  note         = {Machine review of arXiv:2506.08153}
}
read the original abstract

How can the complexity of ML-enabled systems be managed effectively? The goal of this research is to investigate how complexity affects ML-Enabled Systems (MLES). To address this question, this research aims to introduce a metrics-based architectural model to characterize the complexity of MLES. The goal is to support architectural decisions, providing a guideline for the inception and growth of these systems. This paper showcases the first step for creating the metrics-based architectural model: an extension of a reference architecture that can describe MLES to collect their metrics.

Figures

Figures reproduced from arXiv: 2506.08153 by the authors.

Figure 1
Figure 1. summarizes four steps, grouped into two phases, of the methodology proposed for this research. A. Understanding Metrics for Complexity To answer RQ1, this research will start studying the com￾plexity of software systems and metrics. For that, phase 1 has been divided into two steps: (1) to analyze the existing liter￾ature on software systems complexity; and (2) to categorize the metrics related to complexity from th… view at source ↗
Figure 2
Figure 2. Reference Architecture for ML-enabled Systems. There are three types of components in an ML-enabled system. Rectangles represent applications or services, which execute continuously. Stacked rectangles represent pipelines, which execute a task on demand. Lastly, cylinders represent data storage, which may be databases of any type. Components are connected by arrows. Black arrows with a hollow tip illustrate the exec… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems

    cs.SE 2025-06 unverdicted novelty 4.0 of 10

    A research proposal describing a planned metrics-based model for ML-enabled system complexity, illustrated with two architecture diagrams, but containing no computed metrics or validated results.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    The Mythical Man-Month,

    F. P. Brooks, “The Mythical Man-Month,”ACM SIG- PLAN Notices, vol. 10, no. 6, 1975

    work page 1975

  2. [2]

    Ameisen,Building Machine Learning Powered Ap- plications

    E. Ameisen,Building Machine Learning Powered Ap- plications. O’Reilly Media, 2020, pp. 1–257

    work page 2020

  3. [3]

    Software Engineering for Ma- chine Learning: A Case Study,

    S. Amershi et al., “Software Engineering for Ma- chine Learning: A Case Study,”Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE- SEIP 2019, pp. 291–300, May 2019

    work page 2019

  4. [4]

    Machine Learning Systems and Intel- ligent Applications,

    W. C. Benton, “Machine Learning Systems and Intel- ligent Applications,”IEEE Software, vol. 37, no. 4, pp. 43–49, Jul. 2020. [5]Gartner Survey Reveals 80% of Executives Think Au- tomation Can Be Applied to Any Business Decision

    work page 2020

  5. [6]

    Hidden Technical Debt in Machine Learning Systems,

    D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” inAdvances in Neural Information Processing Systems, vol. 2015-January, 2015

    work page 2015

  6. [7]

    A Software Engineering Perspective on Engi- neering Machine Learning Systems: State of the Art and Challenges,

    G. Giray, “A Software Engineering Perspective on Engi- neering Machine Learning Systems: State of the Art and Challenges,”Journal of Systems and Software, vol. 180, p. 111 031, Oct. 2021

    work page 2021

  7. [8]

    Sustainable MLOps: Trends and Challenges,

    D. A. Tamburri, “Sustainable MLOps: Trends and Challenges,”Proceedings - 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2020, pp. 17–23, Sep. 2020

    work page 2020

  8. [9]

    Operationalizing Machine Learning: An Interview Study,

    S. Shankar, R. Garcia, J. M. Hellerstein, and A. G. Parameswaran, “Operationalizing Machine Learning: An Interview Study,” Sep. 2022

    work page 2022

Show all 29 references
  1. [10]

    A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey,

    J. Diaz-De-Arcaya, A. I. Torre-Bastida, G. Z ´arate, R. Mi ˜n´on, and A. Almeida, “A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey,”ACM Computing Surveys, vol. 56, no. 4, 2023

    2023

  2. [11]

    How does machine learning change software development prac- tices?

    Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development prac- tices?”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1857–1871, Sep. 2021

    2021

  3. [12]

    Large-Scale Machine Learning Sys- tems in Real-World Industrial Settings: A Review of Challenges and Solutions,

    L. E. Lwakatare, A. Raj, I. Crnkovic, J. Bosch, and H. H. Olsson, “Large-Scale Machine Learning Sys- tems in Real-World Industrial Settings: A Review of Challenges and Solutions,”Information and Software Technology, vol. 127, 2020

    2020

  4. [13]

    A Survey of Data Quality Requirements That Matter in ML De- velopment Pipelines,

    M. Priestley, F. O’Donnell, and E. Simperl, “A Survey of Data Quality Requirements That Matter in ML De- velopment Pipelines,”Journal of Data and Information Quality, vol. 15, no. 2, 2023

    2023

  5. [14]

    Data Pipeline Quality: Influencing Factors, Root Causes of Data-Related Issues, and Processing Problem Areas for Developers,

    H. Foidl, V . Golendukhina, R. Ramler, and M. Felderer, “Data Pipeline Quality: Influencing Factors, Root Causes of Data-Related Issues, and Processing Problem Areas for Developers,”Journal of Systems and Soft- ware, vol. 207, 2024

    2024

  6. [15]

    Trials and tribulations of developers of intelligent systems: A field study,

    C. Hill, R. Bellamy, T. Erickson, and M. Burnett, “Trials and tribulations of developers of intelligent systems: A field study,”Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC, vol. 2016-November, pp. 162–170, Nov. 2016

    2016

  7. [16]

    A model- driven, metrics-based approach to assessing support for quality aspects in MLOps system architectures,

    S. J. Warnett, E. Ntentos, and U. Zdun, “A model- driven, metrics-based approach to assessing support for quality aspects in MLOps system architectures,”Journal of Systems and Software, vol. 220, p. 112 257, Feb. 2025

    2025

  8. [17]

    Verschuren and H

    P. Verschuren and H. Doorewaard,Designing a Re- search Project: Project Design. 2010

    2010

  9. [18]

    Fenton and J

    N. Fenton and J. Bieman,Software Metrics: A Rigorous and Practical Approach, Third Edition. 2014

    2014

  10. [19]

    Continuous Delivery for Machine Learning,

    D. Sato, A. Wider, and C. Windheuser, “Continuous Delivery for Machine Learning,”Martin Fowler, 2019

    2019

  11. [20]

    Finding a complexity measure for business process models,

    A. M. Latva-Koivisto, “Finding a complexity measure for business process models,”Complexity, 2001

    2001

  12. [21]

    Complexity metrics for process models – A systematic literature review,

    G. Polan ˇciˇc and B. Cegnar, “Complexity metrics for process models – A systematic literature review,”Com- puter Standards and Interfaces, vol. 51, 2017

    2017

  13. [22]

    Systematic Review in Software Engineering,

    B. A. Kitchenham, “Systematic Review in Software Engineering,” pp. 1–2, Sep. 2012

    2012

  14. [23]

    Practices for Managing Machine Learning Products: A Multivocal Literature Review,

    I. Alves, L. A. Leite, P. Meirelles, F. Kon, and C. S. R. Aguiar, “Practices for Managing Machine Learning Products: A Multivocal Literature Review,”IEEE Trans- actions on Engineering Management, vol. 71, 2024

    2024

  15. [24]

    Big data qual- ity metrics for sentiment analysis approaches,

    I. El Alaoui, Y . Gahi, and R. Messoussi, “Big data qual- ity metrics for sentiment analysis approaches,” inACM International Conference Proceeding Series, 2019

    2019

  16. [25]

    Ontology Develop- ment 101: A Guide to Creating Your First Ontology,

    N. F. Noy and D. L. Mcguinness, “Ontology Develop- ment 101: A Guide to Creating Your First Ontology,”

  17. [26]

    Selecting Empirical Methods for Software Engineering Research,

    S. Easterbrook, J. Singer, M. A. Storey, and D. Damian, “Selecting Empirical Methods for Software Engineering Research,”Guide to Advanced Empirical Software En- gineering, pp. 285–311, 2008

    2008

  18. [27]

    SPIRA: Building an Intelligent Sys- tem for Respiratory Insufficiency Detection,

    R. C. Ferreira, D. Gomes, V . Tamae, F. Wernke, and A. Goldman, “SPIRA: Building an Intelligent Sys- tem for Respiratory Insufficiency Detection,”Workshop Brasileiro de Engenharia de Software Inteligente (ISE), vol. 1, pp. 19–22, Oct. 2022

    2022

  19. [28]

    Requirements and Reference Architecture for MLOps: Insights from Industry,

    I. Kumara, R. Arts, D. D. Nucci, R. Kazman, W. J. Van Den Heuvel, and D. A. Tamburri, “Requirements and Reference Architecture for MLOps: Insights from Industry,”Authorea Preprints, Oct. 2023

    2023

  20. [29]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl ´en,Experimentation in Software Engineering. 2012, vol. 9783642290442

    2012

  21. [30]

    MLOps in Practice: Requirements and a Reference Architecture from Industry,

    I. Kumara et al., “MLOps in Practice: Requirements and a Reference Architecture from Industry,”Communica- tions of the ACM, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.