Making a Pipeline Production-Ready: Challenges and Lessons Learned in the Healthcare Domain

2; (2) Jheronimus Academy of Data Science; 3; (3) Technical University of Eindhoven; 4); (4) Tilburg University); Alfredo Goldman (1); Daniel Angelo Esteves Lawand (1); Lucas Quaresma Medina Lam (1); Marcelo Finger (1) ((1) University of S\~ao Paulo

REVIEW 3 major objections 5 minor 1 cited by

Making a Pipeline Production-Ready: Challenges and Lessons Learned in the Healthcare Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that incrementally applying design patterns, Ports and Adapters, and test-driven development to a speech-analysis ML training pipeline raised its extensibility, maintainability, robustness, and resiliency across three…

desk verdict A well-structured experience report of a real ML pipeline refactoring, but its central monotone-improvement claim rests on unmeasured self-assessment and needs revision before it can convince. read the letter →

arxiv 2506.06946 v3 pith:DN3KC6OU submitted 2025-06-07 cs.SE cs.AIcs.LG

Daniel Angelo Esteves Lawand (1) , Lucas Quaresma Medina Lam (1) , Roberto Oliveira Bolgheroni (1) , Renato Cordeiro Ferreira (1 , 2 , 3 , 4) , Alfredo Goldman (1)

show 4 more authors

Marcelo Finger (1) ((1) University of São Paulo (2) Jheronimus Academy of Data Science (3) Technical University of Eindhoven (4) Tilburg University)

This is my paper

classification cs.SEcs.AIcs.LG

keywords CodeQualityMLOpsSoftwareArchitectureMachineLearningEnabledSystemsHealthcareDomainExperienceReportContinuousTrainingPipelineMicroservices

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This experience report argues that a machine-learning training pipeline for detecting respiratory insufficiency from speech improved in software quality through three deliberate rewrites: from a proof-of-concept script tangle, to a modular monolith using design patterns, to a set of test-driven microservices. The authors claim each version improved extensibility, maintainability, robustness, and resiliency. If true, the paper shows that standard software engineering practices can be applied incrementally to productionize ML pipelines, even in a demanding healthcare setting. The value for a general reader is a concrete demonstration that architecture and testing discipline, not just model accuracy, determine whether a research pipeline can survive in production.

What carries the argument

The central mechanism is the three-stage architectural migration of the Continuous Training subsystem: first identify bad smells in the experimental code, then modularize with design patterns and Ports and Adapters, then decompose into microservices with test-driven development. Named structures include the Big Ball of Mud (an unplanned, tangled codebase), the Modular Monolith (a single deployable unit with clear internal module boundaries), and Microservices (multiple independent deployment units). The migration carries the argument by showing with code excerpts how the same functionality was progressively reorganized, making behaviors explicit and testable.

What would settle it

Run the metrics-oriented architectural model from reference [5] against the three versions' code and compare scores: if version 3 does not dominate version 2 on maintainability, extensibility, robustness, and resiliency, the monotone improvement claim is refuted. Alternatively, a fault-injection experiment could test whether a failure in one version 3 stage really avoids a full pipeline re-execution.

Watch

Extended reading notes

Core claim

The central claim is that SPIRA's Continuous Training subsystem evolved in three versions, each raising its quality attributes. Version 1, a Big Ball of Mud inherited from the proof-of-concept code, had high coupling and low cohesion, shown through a Misplaced Responsibility bad smell. Version 2 reimplemented the code as a Modular Monolith by applying design patterns such as Chain of Responsibility, Strategy, and Template Method, plus the Ports and Adapters pattern, which improved maintainability and extensibility. Version 3 split the monolith into multiple deployment units following the Workflow Pipeline ML design pattern and added automated tests via Test-Driven Development, which improved robustness and resiliency. The paper's conclusion is that incremental application of these practices produced a production-ready pipeline where each step built on the previous one.

Load-bearing premise

The improvement across versions rests on the authors' own qualitative judgment of code they wrote, with no independent metrics or failure counts reported.

Editorial extensions

If this is right

If the claim holds, other ML projects can follow the same staged path from proof-of-concept to production without a single risky rewrite.
Applying design patterns and Ports and Adapters before microservice decomposition can improve maintainability and extensibility for ML training pipelines.
Test-driven development and integration tests around external dependencies can make ML pipelines more robust and resilient in production.
Healthcare ML-enabled systems can benefit from standard software engineering practices rather than relying on specialized MLOps tooling alone.
The lessons learned suggest that involving software engineering practices from the start of a data science project could reduce the cost of productionization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own metrics-oriented architectural model could be applied to the three versions to quantify the claimed improvements; since no such measurements are reported, that quantitative check remains an open test.
Splitting a monolith into microservices may trade operational complexity and deployment overhead for fault isolation, a cost the paper does not deeply quantify.
The same phased refactoring sequence could be tried on other research pipelines, but its generalizability beyond SPIRA would need independent replication.
Test counts, failure injection results, or cyclomatic complexity measurements would convert the qualitative story into a falsifiable benchmark.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the SPIRA paper is a cleanly written experience report comparing three versions of an ML training pipeline. The architecture story is coherent, the code is open source, and the authors are honest that this is an experience report, not a controlled study. But the central improvement claim is supported only by self-assessment, with no metrics, test counts, or production evidence, despite the authors having their own metrics framework [5]. The healthcare framing is mostly window dressing — the challenges and lessons are generic software engineering.

What is actually new: a documented three-stage refactoring (Big Ball of Mud to Modular Monolith to test-driven microservices) of a real speech-analysis ML pipeline, with concrete code snippets showing the bad smell and the modularized version. That is a useful teaching example. The individual practices — design patterns, Ports and Adapters, TDD — are not new in ML engineering, but the case description is a genuine contribution.

What it does well: the structure is clear, the figures help, the v1-to-v2-to-v3 progression is easy to follow, and the open-source repositories are referenced. The authors cite their own prior work and theses explicitly, which is appropriate for a synthesis.

Soft spots: the abstract and conclusion claim that each version improved extensibility, maintainability, robustness, and resiliency. There is no measurement of any of those attributes. Sections 5 and 6 cite a textbook [19] as evidence. The authors' own [5] is a metrics-oriented model and they never apply it. No test counts, failure rates, recovery times, or deployment logs. The v3 is labeled "Productionized" but no evidence shows it running in production. The comparison is by the same team that built all versions. These are real weaknesses, but proportionate: the paper is an experience report, so qualitative assessment is expected. Still, the claims are stronger than the evidence.

A minor issue: the title says "Healthcare Domain," but the lessons are generic SE guidance; nothing health-specific (regulations, privacy, clinical validation) is engaged. That could mislead a reader.

Bottom line: the paper deserves a serious referee, but needs revision — either tone down the monotone-improvement claim to "qualitative assessment" or add metrics from [5] and test counts, and clarify whether v3 is actually in production. As it stands, a conditional accept is reasonable.

Recommendation: send to peer review with a request for major revision.

Referee Report

3 major / 5 minor

Summary. The paper is an experience report on the SPIRA project, an ML-Enabled System for detecting respiratory insufficiency through speech analysis. It describes the system architecture following a reference model from the authors' earlier work [5], focuses on the Continuous Training subsystem, and recounts its evolution through three versions: v1 as an experimental Big Ball of Mud, v2 as a modular monolith using design patterns and Ports and Adapters, and v3 as a microservices-based pipeline developed with test-driven development. The paper claims each version improved extensibility, maintainability, robustness, and resiliency, and presents two challenges and two lessons learned from the migration experience.

Significance. If the central improvement claim could be verified, the paper would be a valuable case study for applying software engineering practices to ML pipelines in a healthcare domain, and it would provide practical guidance for similar projects. The paper has notable strengths: the system is open source, the architecture is documented with diagrams and code fragments, and the authors explicitly map their work to an existing reference architecture [5]. However, the evidence for the main claim is exclusively the authors' qualitative self-assessment; no metrics, test results, or operational data are reported, and the team that performed the refactorings is the same team that evaluates them. This limits the current contribution to an anecdotal account.

major comments (3)

[Abstract; §5; §6; §9] The central claim that each version improved extensibility, maintainability, robustness, and resiliency is asserted without supporting measurement. Section §5 states that the reimplementation 'made the code more maintainable and extensible [19]' and §6 states that it made the code 'more robust and resilient [19]', with a textbook [19] as the only citation; no operationalization of these quality attributes, no test counts, failure counts, recovery times, or code metrics are provided. Because the authors cite their own metrics-oriented model [5] but do not apply it, the improvement claim is not supported. The authors should either measure the four attributes consistently across v1, v2, and v3 (e.g., using the model in [5] or surrogate metrics such as test coverage, coupling, and failure recovery), or explicitly reframe the claim as subjective experience rather than demonstrated improvement.
[§6; §2.6; §9] The label 'Productionized Architecture' for v3 and the abstract's 'production-ready' imply that the system is running in production, but the paper provides no evidence of production deployment or operation. §6 says only that the goal was 'to enable executing the pipeline with a Scheduler Service,' and §2.6 states 'Once the SPIRA MLES reaches production,' which suggests the system has not yet reached that state. The authors should either supply deployment logs, operational data, or a statement of current deployment status, or soften the terminology to 'production-intent architecture.'
[§5; §6] The paper uses a software architecture textbook [19] as evidence for the empirical effects of the refactorings, e.g., §5 'This reimplementation made the code more maintainable and extensible [19]' and §6 'This reimplementation made the code more robust and resilient [19].' A textbook can motivate the expected benefits of practices, but it does not demonstrate that the particular refactorings in SPIRA had those effects. The manuscript would be strengthened by reporting concrete observable indicators, such as the number of failing tests before and after the refactoring, time to add a new feature, or number of defects found in integration.

minor comments (5)

[Abstract] There are typos: 'insufficiency respiratory' should be 'respiratory insufficiency,' and 'Microser vices' should be 'Microservices.'
[§8] The phrase 'whose goal if to help' should be 'whose goal is to help.'
[Fig. 1] The caption in Figure 1 is extremely long and duplicates much of the main text; consider shortening it or moving the detailed notation to a dedicated legend.
[§2.2] The phrase 'make the two latter roles' is awkward; consider 'fulfill the latter two roles.'
[References] The v2 and v3 repositories are said to be open source, but the relevant URLs are not listed; providing links would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the quality-improvement claim is narrative self-assessment, not a derived prediction that reduces to its inputs.

full rationale

The paper is an experience report, not a derivation: it contains no fitted parameters, equations, or predictions that could be shown to equal their inputs by construction. The central claim—that each version improved extensibility, maintainability, robustness, and resiliency—is asserted from the authors' own experience and supported by textbook references such as [19], not derived from a model or metric. The self-citations to Ferreira et al. [5], SPIRA project papers [6,7], and the bachelor theses [13,14] supply context, code provenance, and architectural vocabulary, but none is used as a load-bearing inference that forces the improvement conclusion. Even if the lack of independent metrics makes the claim weakly supported, that is an evidence-quality and falsifiability concern, not a circularity concern. No specific circular step can be quoted or exhibited, so the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an experience report, so the ledger contains no fitted numeric parameters and no invented entities. The load-bearing premises are: the validity of the Big Ball of Mud classification of v1, the causal efficacy of the applied software engineering practices, and the validity of the authors' qualitative self-assessment as a measure of quality improvement. These premises come from prior literature or from the authors' own judgment, not from measurements in this paper.

assumptions (3)

domain assumption The v1 code is an instance of "Big Ball of Mud", and this classification implies the claimed quality deficits.
§4 classifies the proof-of-concept as Big Ball of Mud [8] based on a single code snippet showing a Misplaced Responsibility bad smell; no structural or historical analysis is provided.
domain assumption Applying design patterns, Ports and Adapters, microservices, and TDD causes the claimed improvements in extensibility, maintainability, robustness, and resiliency.
§5 and §6 attribute improvements to these practices with citations to textbooks [1], [19], and [20]; the causal link is assumed, not measured.
ad hoc to paper The authors' qualitative self-assessment across versions is a valid measure of quality improvement.
No metrics, test counts, or external evaluation are presented; the improvement judgment is made by the same team that performed the refactorings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Making a Pipeline Production-Ready: Challenges and Lessons Learned in the Healthcare Domain." pith.science (2026). https://pith.science/paper/DN3KC6OU

@misc{pith2026250606946,
  author       = {Pith},
  title        = {Pith review of: Making a Pipeline Production-Ready: Challenges and Lessons Learned in the Healthcare Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DN3KC6OU}},
  note         = {Machine review of arXiv:2506.06946}
}

read the original abstract

Deploying a Machine Learning (ML) training pipeline into production requires good software engineering practices. Unfortunately, the typical data science workflow often leads to code that lacks critical software quality attributes. This experience report investigates this problem in SPIRA, a project whose goal is to create an ML-Enabled System (MLES) to pre-diagnose insufficiency respiratory via speech analysis. This paper presents an overview of the architecture of the MLES, then compares three versions of its Continuous Training subsystem: from a proof of concept Big Ball of Mud (v1), to a design pattern-based Modular Monolith (v2), to a test-driven set of Microservices (v3) Each version improved its overall extensibility, maintainability, robustness, and resiliency. The paper shares challenges and lessons learned in this process, offering insights for researchers and practitioners seeking to productionize their pipelines.

Figures

Figures reproduced from arXiv: 2506.06946 by the authors.

**Figure 1.** Architecture of the SPIRA ML-Enabled System. The architecture is described with the same notation of the reference architecture described in Ferreira et al. [5]. Rectangles represent services, which execute continuously. Stacked rectangles represent pipelines, which execute a task on demand. Lastly, cylinders represent data storage, which may be databases of any type. Components are connected by arrows. Black arrows… view at source ↗

**Figure 2.** Incremental Development of the SPIRA Continuous Training Subsystem. Dashed boxes represent the versions of the subsystem. Rounded boxes represent deployment units. Squared boxes represent components. Black arrows with a filled tip illustrate the data flow, while the labels in italic represent the data types. ( v1 architecture ) 1 random.seed(c.train_config["seed"]) 2 torch.manual_seed(c.train_config["seed"]) 3 torch… view at source ↗

**Figure 3.** Example of improving modularization between v1 and v2 architectures. The v1 snippet (left) shows a Misplaced Responsibility bad smell [15] at the class Dataset. Lines 1-7 handle random number generation. Lines 8-16 assign values to attributes. Lines 18-19 and 22-24 handle assertions. Line 30 handles data loading. The v2 snippet (right) shows the application of design patterns at the module pipeline. Line 15-23 handl… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems
cs.SE 2025-06 unverdicted novelty 4.0 of 10

A research proposal describing a planned metrics-based model for ML-enabled system complexity, illustrated with two architecture diagrams, but containing no computed metrics or validated results.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · cited by 1 Pith paper

[5]

Proceedings - 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI, CAIN 2025 (2025)

Ferreira, R.C.: A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems. Proceedings - 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI, CAIN 2025 (2025)

work page 2025
[19]

O’Reilly Media, 1st edn

Richards, M., Ford, N.: Fundamentals of Software Architecture. O’Reilly Media, 1st edn. (2020)

work page 2020
[1]

Addison-Wesley Professional, 1st edn

Beck, K.: Test Driven Development: By Example. Addison-Wesley Professional, 1st edn. (2002)

work page 2002
[2]

Addison-Wesley Professional, 2nd edn

Beck, K., Andres, C.: Extreme Programming Explained. Addison-Wesley Professional, 2nd edn. (2004)

work page 2004
[3]

True Positive Inc., 1st edn

Burkov, A.: Machine Learning Engineering. True Positive Inc., 1st edn. (2020)

work page 2020
[4]

Findings of the Association for Computational Linguistics pp

Casanova, E., Gris, L., Camargo, A., Da Silva, D., Gazzola, M., Sabino, E., Levin, A.S., Candido, A., Aluisio, S., Finger, M.: Deep Learning against COVID-19: Respiratory Insufficiency Detection in Brazilian Portuguese Speech. Findings of the Association for Computational Linguistics pp. 625–633 (2021)

work page 2021
[6]

Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (2022)

Ferreira, R.C., Gomes, D., Tamae, V., Wernke, F., Goldman, A.: SPIRA: Building an Intelligent System for Respiratory Insufficiency Detection. Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (2022)

work page 2022
[7]

Finger, M., Aluísio, S.M., Spazzapan, E.A., Berti, L.C., Camargo Neto, A.C.d., Candido Junior, A., Casanova, E., Svartman, F.R.F., Ferreira, R.C., Fernandes Jr, R., Goldman, A., Gris, L.R., Leyton, P., Levin, A.S.S., Martins, M.V.M., Queiroz, M.G.d., Quirino, J.H., Medeiros, B.R.d., Sabino, E.C., Silva, D.d.: DetectingRespiratoryInsufficiencybyVoiceAnalys...

work page 2021

Show all 21 references

[8]

In: PLoP ’97 / EutoPLoP ’97

Foote, B., Yoder, J.: Big Ball of Mud. In: PLoP ’97 / EutoPLoP ’97. Monticello (1999)

1999
[9]

Addison-Wesley Professional, 1st edn

Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Professional, 1st edn. (1994)

1994
[10]

The MIT Press, 1st edn

Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press, 1st edn. (2016)

2016
[11]

IEEE Software19(2) (2002)

Hunt, A., Thomas, D.: Software archaeology. IEEE Software19(2) (2002)

2002
[12]

O’Reilly Media, 1st edn

Lakshmanan, V., Robinson, S., Munn, M.: Machine Learning Design Patterns. O’Reilly Media, 1st edn. (2020)

2020
[13]

Lam, L.Q.M., Bolgheroni, R.O., Ferreira, R.C., Goldman, A.: Productionizing SPIRA’s Model Trainer System. Ph.D. thesis, University of São Paulo, São Paulo (2024)

2024
[14]

Lawand, D.A.E., Ferreira, R.C., Goldman, A.: Enabling MLOps in the SPIRA Training Pipeline. Ph.D. thesis, University of São Paulo, São Paulo (2023)

2023
[15]

Pearson, 1 edn

Martin, R.C.: Clean Code. Pearson, 1 edn. (2008)

2008
[16]

Pearson, 1st edn

Martin, R.C.: Clean Architecture. Pearson, 1st edn. (2017)

2017
[17]

O’Reilly Media, 2nd edn

Morris, K.: Infrastructure as Code. O’Reilly Media, 2nd edn. (2025)

2025
[18]

O’Reilly Media, 2nd edn

Newman, S.: Building Microservices. O’Reilly Media, 2nd edn. (2021)

2021
[20]

Manning Publications, 1st edn

Richardson, C.: Microservices Patterns. Manning Publications, 1st edn. (2018)

2018
[21]

Manning Publications, 1st edn

Wilson, B.: Machine Learning Engineering in Action. Manning Publications, 1st edn. (2022)

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [5]

Proceedings - 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI, CAIN 2025 (2025)

Ferreira, R.C.: A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems. Proceedings - 2025 IEEE/ACM 4th International Conference on AI Engineering - Software Engineering for AI, CAIN 2025 (2025)

work page 2025

[2] [19]

O’Reilly Media, 1st edn

Richards, M., Ford, N.: Fundamentals of Software Architecture. O’Reilly Media, 1st edn. (2020)

work page 2020

[3] [1]

Addison-Wesley Professional, 1st edn

Beck, K.: Test Driven Development: By Example. Addison-Wesley Professional, 1st edn. (2002)

work page 2002

[4] [2]

Addison-Wesley Professional, 2nd edn

Beck, K., Andres, C.: Extreme Programming Explained. Addison-Wesley Professional, 2nd edn. (2004)

work page 2004

[5] [3]

True Positive Inc., 1st edn

Burkov, A.: Machine Learning Engineering. True Positive Inc., 1st edn. (2020)

work page 2020

[6] [4]

Findings of the Association for Computational Linguistics pp

Casanova, E., Gris, L., Camargo, A., Da Silva, D., Gazzola, M., Sabino, E., Levin, A.S., Candido, A., Aluisio, S., Finger, M.: Deep Learning against COVID-19: Respiratory Insufficiency Detection in Brazilian Portuguese Speech. Findings of the Association for Computational Linguistics pp. 625–633 (2021)

work page 2021

[7] [6]

Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (2022)

Ferreira, R.C., Gomes, D., Tamae, V., Wernke, F., Goldman, A.: SPIRA: Building an Intelligent System for Respiratory Insufficiency Detection. Workshop Brasileiro de Engenharia de Software Inteligente (ISE)1, 19–22 (2022)

work page 2022

[8] [7]

Finger, M., Aluísio, S.M., Spazzapan, E.A., Berti, L.C., Camargo Neto, A.C.d., Candido Junior, A., Casanova, E., Svartman, F.R.F., Ferreira, R.C., Fernandes Jr, R., Goldman, A., Gris, L.R., Leyton, P., Levin, A.S.S., Martins, M.V.M., Queiroz, M.G.d., Quirino, J.H., Medeiros, B.R.d., Sabino, E.C., Silva, D.d.: DetectingRespiratoryInsufficiencybyVoiceAnalys...

work page 2021