Pith. sign in

REVIEW 4 major objections 6 minor 18 references

A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This research proposal sets out to build a metrics-oriented architectural model that characterizes complexity in machine-learning-enabled systems through software, data, and model dimensions, starting with the SPIRA and Ocean Guard…

desk verdict A clear, honest PhD research proposal whose only concrete artifact is two architecture diagrams; the promised metrics model is not there yet, so treat it as a position paper, not a characterization. read the letter →

arxiv 2506.11295 v1 pith:OWTQNFSS submitted 2025-06-12 cs.SE cs.AIcs.LG

Renato Cordeiro Ferreira (1 , 2 , 3 , 4) ((1) University of São Paulo , (2) Jheronimus Academy of Data Science , (3) Technical University of Eindhoven , (4) Tilburg University) This is my paper
classification cs.SEcs.AIcs.LG
keywords SoftwareMetricsComplexityML-EnabledSystemsMachineLearningEngineeringMLOpsArchitecturalModelCaseStudyMetrics-oriented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ML-enabled systems carry high essential complexity because their architectures must host data processing and model handling alongside regular software concerns, and practitioners report an aversion to that complexity. This PhD proposal plans to make complexity measurable rather than anecdotal: it will survey how practitioners handle complexity, build an ontology of complexity metrics, then use two systems the author knows well—SPIRA, a speech-based respiratory-insufficiency screener, and Ocean Guard, a maritime anomaly detector—as typical cases to develop a metrics-oriented architectural model. The paper's contribution at this stage is the side-by-side architecture representations of the two systems and the design of a two-phase, four-step methodology that would produce the model. If the plan works, architects would be able to see where complexity concentrates in an ML system and act before it blocks production.

What carries the argument

The load-bearing object is the metrics-oriented architectural model, a planned device that would map complexity metrics onto an MLES architecture. It is carried by a shared architecture notation—applications and services as rectangles, on-demand pipelines as stacked rectangles, storage as cylinders, with colored data-flow arrows and subsystem groupings—that lets two different systems be drawn on the same terms. The two typical cases, SPIRA and Ocean Guard, do the empirical work: they are the known codebases on which metric collection will be documented before the process is validated on a larger set of open-source ML products.

What would settle it

Take an unseen ML-enabled system from the open-source dataset of 262 ML products, run the documented metric-collection process on it, and compare the resulting complexity scores with the system's developers' ratings of where complexity hurts them; if the scores disagree with those ratings, or if the collection cannot be performed without the model's creator on hand, the generalizability claim fails.

Watch

Extended reading notes

Core claim

The central claim is that architectural complexity in an ML-enabled system is not an unstructured burden but a multi-dimensional property—spanning software, data, and model components—that can be captured by metrics and operationalized over an architecture description. The paper argues this in the form of a research plan: RQ1 asks which measurable dimensions of complexity exist in MLES architectures, and RQ2 asks how such metrics can be collected in practice. As preliminary results, it offers a common notation and two side-by-side architectural diagrams, SPIRA and Ocean Guard, chosen as typical cases for the exploratory case study. The discovery, on the paper's own terms, is that these two systems and this notation provide a concrete starting point from which the metric collection process and the first version of the metrics-oriented architectural model can be built.

Load-bearing premise

The load-bearing premise is that SPIRA and Ocean Guard, two systems developed by the researcher himself, are typical enough of ML-enabled systems that complexity metrics developed on them will transfer to other systems; the paper's own open questions acknowledge the bias risk from this familiarity.

Editorial extensions

If this is right

  • Complexity in MLES would become decomposable into software, data, and model dimensions, each with its own measurable metrics rather than a single subjective judgment.
  • Architects and engineering teams could use the model to compare architectural options and to locate complexity hotspots that often stop ML systems from reaching production.
  • The model would ship with a documented metric-collection process, enabling other researchers to apply the same measurements to their own systems.
  • The two case studies would produce a first version of the model, and a confirmatory case study on open-source ML repositories would refine it, giving the model a path from two examples to a broader population.
  • The shared architecture notation would become a comparison standard, making side-by-side complexity analysis of different MLES possible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the model succeeds, a natural extension is a complexity budget for MLES—teams could set thresholds per subsystem and alert when a change pushes complexity past the budget, much like performance budgets in web engineering.
  • Beyond the paper: the metric-collection process could be tested retroactively on historical ML projects to see whether measured complexity correlates with production failures, giving the model predictive value the paper does not claim.
  • Beyond the paper: because both selected systems follow MLOps-heavy, microservice-style patterns, the model may over-fit to that style; a testable extension is to apply it to simpler, notebook-centric ML systems and check whether the metrics still make sense.
  • Beyond the paper: the side-by-side diagrams themselves suggest a concrete first comparison—counting component types and data-flow edges in each system could serve as a preliminary proxy for complexity before the formal metric ontology is complete.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript, framed as a PhD research proposal, aims to develop a metrics-oriented architectural model for characterizing complexity in machine-learning-enabled systems (MLES). It poses two research questions: what measurable dimensions of complexity exist in MLES architectures (RQ1), and how such metrics can be operationalized over those architectures (RQ2). The proposed methodology, summarized in Figure 1, consists of four steps across two phases: an industry survey and ontology design to understand complexity metrics, followed by an exploratory case study on two author-built systems (SPIRA and Ocean Guard) and a confirmatory case study on open-source ML repositories. The only concrete results presented are two detailed architecture diagrams, Figures 2 and 3, drawn with an extension of the authors' earlier notation. Section 5 acknowledges open questions, including how to mitigate experimental bias from the author's familiarity with the case studies. The paper's own Section 1 states that it presents a plan, and no complexity metrics are yet defined, computed, or validated.

Significance. If the proposed model is realized and validated, it would address a recognized gap: Giray [9] lists handling complexity as an open challenge in ML engineering, and Shankar et al. [17] report practitioners' aversion to complexity. The manuscript's strengths are its explicit research questions, a clear four-step methodology, and two unusually detailed architecture descriptions that use a notation grounded in prior reference-architecture work [10,11]. The diagrams themselves are substantial descriptive artifacts. However, the current significance is conditional: the paper does not yet provide any metric definitions, quantitative comparisons, or empirical validation, and the title and abstract claim a characterization that the body of the paper does not deliver. The contribution as written is a research plan plus two case-study descriptions, not a validated complexity model.

major comments (4)
  1. [Abstract and Section 4] The title and abstract claim to 'characterize' architectural complexity, but the only results in Section 4 are two architecture figures; no metric is defined, computed, or compared. Figure 1 shows that the metrics-oriented model is future work (steps 1.2, 2.1, 2.2), and Section 1 itself says 'This proposal presents a plan.' The authors should either reframe the contribution as a research proposal and adjust the title accordingly, or add an illustrative instantiation of quantitative complexity indicators on Figures 2 and 3 (for example, component counts, data-flow edge counts, pipeline dependencies, or feedback loops) so that the characterization claim is testable rather than purely promissory.
  2. [Section 3.2] The two 'typical cases' are selected because they are 'currently being developed by the author' and because familiarity 'should allow the author to focus on operationalizing the metric collection.' The paper does not define what makes a case typical, nor does it provide a sampling rationale from any defined population of MLES. Section 5 explicitly raises the resulting bias question without an answer. Because the initial metric set is chosen partly by measurability on these two familiar systems, the resulting model may be an artifact of the two systems rather than a general characterization. Please add a typicality argument (for instance, coverage of the component types in the reference architecture [10,11]) and a concrete mitigation plan (for example, independent or blinded metric selection, preregistration of the metric ontology, or an inter-rater agreement check).
  3. [Section 2 and Section 3.2, Step 2.2] The confirmatory case study's stated goal is to 'validate the robustness of the metric collection process' and to review 'which metrics are worth collecting.' This establishes repeatability or computability, not that the selected metrics measure architectural complexity. No ground truth or external criterion for complexity is defined, so the empirical loop is at risk of circularity: metrics are selected because they are computable on the author's two systems and then 'confirmed' by being computable on other repositories. The manuscript should state a falsifiable success criterion, such as a hypothesized relationship between candidate metrics and expert-assessed complexity, maintenance effort, or defect incidence, and explain how the Step 2.2 data could disconfirm the initial metric selection.
  4. [Section 4 and Figures 2-3] The two diagrams are presented 'side-by-side' but the paper offers no comparison or analysis relating the notation elements to complexity. There is no table contrasting the systems along dimensions such as numbers of services, pipelines, storage components, or data-flow edges, so the reader cannot see what the 'tale of two systems' demonstrates. Please add a comparison table and map diagram elements to candidate complexity metrics, since such a mapping is a necessary link between the descriptive architecture notation and the planned metrics-oriented model.
minor comments (6)
  1. [Abstract and Section 4] There are several language issues: 'theSPIRA' is missing a space in the Abstract, and 'prediagnostic of insufficiency respiratory' in Section 4 should read 'prediagnosis of respiratory insufficiency' or similar.
  2. [Section 3.1] The two paragraphs under 'Ontology Design' repeat almost identical sentences about the ontology focusing on 'different types of metrics [5]' and 'possible quality attributes [12].' Please merge them into one paragraph.
  3. [Section 5] The autobiographical narrative ('This PhD started at University of São Paulo...') is unusual in an archival paper. Consider moving it to an author-positionality statement or appendix and keeping Section 5 focused on threats to validity and planned mitigations.
  4. [Figures 2 and 3] The labels in both figures are very small, and the color-coded data types are difficult to distinguish in grayscale. Please provide high-resolution figures and a legend that uses patterns or textual labels in addition to color.
  5. [References] Reference [6] is the author's own CAIN 2025 paper; self-citations should be explicitly identified, and for [8] and [13], state whether the cited submissions are accepted or under review.
  6. [Section 2] The claim that RQ1 and RQ2 follow the SMART principle is not fully substantiated; for example, the time-bound criterion is not evident in the current wording of either research question. Please clarify or soften the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a research proposal whose planned model is to be induced from literature and case studies; acknowledged self-selection bias is a validity threat, not a tautological derivation.

full rationale

This is a PhD research proposal, not a derivation with fitted values or quantitative predictions. It contains no equations and no parameter fitting. The proposed metrics-oriented architectural model is to be built in Phase 1 from a practitioner survey and ontology design, then operationalized in Phase 2 on two author-developed systems (SPIRA and Ocean Guard) and later on open-source repositories. The reliance on the author's own systems and his own architecture notation [6,7,8,13] is a real threat to external validity, and the paper explicitly acknowledges it in Section 5: 'How to mitigate experimental bias and other threats to validity when creating the metrics-oriented architectural model, considering the familiarity with the case studies?' However, this is a validity concern, not circularity: the paper does not claim that the two systems prove the model; it labels them 'typical cases' for an exploratory study and defers validation to a confirmatory step. The confirmatory step's stated goal is to 'validate the robustness of the metric collection process,' which is weaker than validating that the metrics measure complexity; that is a limitation and an open problem, not a circular reduction. No self-citation is used to prove the central claim, and there is no mechanism by which the model is forced to equal its inputs by construction. Therefore no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted constants or invented physical entities. Its load-bearing assumptions are domain-level premises from software engineering literature and methodological choices about using the author's own systems and notation as the basis for the planned model. The latter two are acknowledged in Section 5 but not yet resolved.

assumptions (4)
  • domain assumption Essential and accidental complexity, as defined by Brooks, applies to ML-enabled systems and can be used as a framing for complexity metrics.
    Section 1 uses Brooks' distinction to motivate the work. It is a reasonable domain framing from prior software engineering literature, not a mathematical axiom.
  • domain assumption MLES complexity decomposes into software, data, and model dimensions that are measurable in the architecture.
    Section 2, RQ1 states MLES 'also have their data and model dimensions [16]'. This is the core premise of RQ1 and is treated as given.
  • ad hoc to paper The architecture notation, an extension of Kumara et al., is sufficient to represent and compare complexity-relevant aspects of SPIRA and Ocean Guard.
    Section 3.2 and the captions of Figures 2 and 3 use this notation without validating that it captures complexity. The notation is the author's prior work, so a future model built on it is partially self-referential.
  • ad hoc to paper SPIRA and Ocean Guard are typical cases from which a generalizable metrics model can be induced.
    Section 3.2 labels them as 'typical cases'. Section 5 raises the bias question, indicating this assumption is unresolved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems." pith.science (2026). https://pith.science/paper/OWTQNFSS

@misc{pith2026250611295,
  author       = {Pith},
  title        = {Pith review of: A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWTQNFSS}},
  note         = {Machine review of arXiv:2506.11295}
}
read the original abstract

How can the complexity of ML-enabled systems be managed effectively? The goal of this research is to investigate how complexity affects ML-Enabled Systems (MLES). To address this question, this research aims to introduce a metrics-based architectural model to characterize the complexity of MLES. The goal is to support architectural decisions, providing a guideline for the inception and growth of these systems. This paper brings, side-by-side, the architecture representation of two systems that can be used as case studies for creating the metrics-based architectural model: the SPIRA and the Ocean Guard MLES.

Figures

Figures reproduced from arXiv: 2506.11295 by the authors.

Figure 1
Figure 1. Research Methodology. The methodology is divided into two phases, each ad￾dressing a research question from section 2. This methodology categorizes as empirical software engineering with a mixed-method approach. Survey Research with Industry. Survey research proposes selecting a repre￾sentative sample from a well-defined population, identifying characteristics from it (via questionnaires or structured interviews), a… view at source ↗
Figure 2
Figure 2. System Architecture of SPIRA. This architecture follows the nota￾tion presented at the PhD Symposium at CAIN 2025 [6], while the SPIRA ar￾chitecture was submitted to SADIS 2025 [13]. Rectangles represent applications or services, which execute continuously. Stacked rectangles represent pipelines, which execute a task on demand. Lastly, cylinders represent data storage, which may be databases of any type. Components … view at source ↗
Figure 3
Figure 3. System Architecture of Ocean Guard. This architecture follows the no￾tation presented at the PhD Symposium at CAIN 2025 [6], while the Ocean Guard architecture was submitted to SummerSOC 2025 [8]. Rectangles represent applica￾tions or services, which execute continuously. Stacked rectangles represent pipelines, which execute a task on demand. Lastly, cylinders represent data storage, which may be databases of any ty… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 14 canonical work pages

  1. [9]

    Journal of Systems and Software180, 111031 (10 2021)

    Giray, G.: A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges. Journal of Systems and Software180, 111031 (10 2021). https://doi.org/10.1016/J.JSS.2021.111031

    arXiv 2021

  2. [17]

    Shankar, S., Garcia, R., Hellerstein, J.M., Parameswaran, A.G.: Operationaliz- ing Machine Learning: An Interview Study (9 2022), https://arxiv.org/abs/2209. 09125v1

    work page 2022

  3. [1]

    Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2019 pp

    Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software Engineering for Machine Learning: A Case Study. Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2019 pp. 291–300 (5 2019). https://doi.org/10.1109/ICSE-SEIP.2019...

    arXiv 2019

  4. [2]

    IEEE Soft- ware 37(4), 43–49 (7 2020)

    Benton, W.C.: Machine Learning Systems and Intelligent Applications. IEEE Soft- ware 37(4), 43–49 (7 2020). https://doi.org/10.1109/MS.2020.2985224

    work page arXiv 2020

  5. [3]

    ACM SIGPLAN Notices10(6) (1975)

    Brooks, F.P.: The Mythical Man-Month. ACM SIGPLAN Notices10(6) (1975). https://doi.org/10.1145/390016.808439

    work page arXiv 1975

  6. [4]

    Guide to Advanced Empirical Software Engi- neering pp

    Easterbrook, S., Singer, J., Storey, M.A., Damian, D.: Selecting Empirical Methods for Software Engineering Research. Guide to Advanced Empirical Software Engi- neering pp. 285–311 (2008). https://doi.org/10.1007/978-1-84800-044-5{_}11

    doi:10.1007/978-1-84800-044-5 2008

  7. [5]

    https://doi.org/10.1201/b17461

    Fenton, N., Bieman, J.: Software Metrics: A Rigorous and Practical Approach, Third Edition (2014). https://doi.org/10.1201/b17461

    work page doi:10.1201/b17461 2014

  8. [6]

    A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems

    Ferreira, R.C.: A Metrics-Oriented Architectural Model to Characterize Complex- ity on Machine Learning-Enabled Systems. Proceedings - 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI, CAIN 2025 (6 2025), http://arxiv.org/abs/2506.08153

    work page Pith review arXiv 2025

Show all 18 references
  1. [7]

    Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (10 2022)

    Ferreira, R.C., Gomes, D., Tamae, V., Wernke, F., Goldman, A.: SPIRA: Building an Intelligent System for Respiratory Insufficiency Detection. Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (10 2022). https://doi.org/ 10.5753/ISE.2022.227048

    2022

  2. [8]

    Ferreira, R.C., Trapmann, R., Heuvel, W.J.v.d.: MLOps with Microservices: A Case Study on the Maritime Domain (6 2025), http://arxiv.org/abs/2506.06202

    2025

  3. [10]

    In: Proceedings of the 19th European Conference on Software Architecture

    Kumara, I., Arts, R., Ferreira, R.C., Nucci, D.D., Kazman, R., Tamburri, D.A., Heuvel, W.J.v.d.: MLOps in Practice: Requirements and a Reference Architec- ture from Industry. In: Proceedings of the 19th European Conference on Software Architecture. Limassol, Cyprus (4 2025)

    2025

  4. [11]

    Authorea Preprints (10 2023)

    Kumara, I., Arts, R., Nucci, D.D., Kazman, R., Van Den Heuvel, W.J., Tamburri, D.A.: Requirements and Reference Architecture for MLOps: Insights from Indus- try. Authorea Preprints (10 2023). https://doi.org/10.36227/techrxiv.21397413.v1, https://www.techrxiv.org/doi/full/10.3...

    2023 doi

  5. [12]

    Complexity (2001)

    Latva-Koivisto, A.M.: Finding a complexity measure for business process models. Complexity (2001)

    2001

  6. [13]

    Lawand, D., Quaresma, L., Bolgheroni, R., Goldman, A., Ferreira, R.C.: Is Your Training Pipeline Production-Ready? A Case Study in the Healthcare Domain (6 2025), http://arxiv.org/abs/2506.06946

    2025 arXiv

  7. [14]

    In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)

    Nahar, N., Zhang, H., Lewis, G., Zhou, S., Kästner, C.: The Product Beyond the Model – An Empirical Study of Repositories of Open-Source ML Products. In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). pp. 63–75. IEEE Computer Society (2025). https:...

    2025

  8. [15]

    Noy, N.F., Mcguinness, D.L.: Ontology Development 101: A Guide to Cre- ating Your First Ontology https://protege.stanford.edu/publications/ontology_ development/ontology101.pdf

  9. [16]

    Martin Fowler (2019)

    Sato, D., Wider, A., Windheuser, C.: Continuous Delivery for Machine Learning. Martin Fowler (2019)

    2019

  10. [18]

    Verschuren, P., Doorewaard, H.: Designing a Research Project: Project Design (2010)

    2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.