The Systems Engineering Approach in Times of Large Language Models

Christian Cabrera; Jennifer Schooling; Neil D. Lawrence; Viviana Bastidas

REVIEW 3 major objections 7 minor 5 references

The Systems Engineering Approach in Times of Large Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that adopting Large Language Models into critical socio-technical systems should be treated as a systems engineering problem — prioritise the problem and its context before the technology — and supports this by showing…

desk verdict Useful mapping of SE principles to LLM-era challenges, but the 'better equipped' claim in the abstract outruns what the survey can actually support. read the letter →

arxiv 2411.09050 v1 pith:25IIDOKP submitted 2024-11-13 cs.AI cs.CYcs.SE

Christian Cabrera , Viviana Bastidas , Jennifer Schooling , Neil D. Lawrence This is my paper · ORCID

classification cs.AIcs.CYcs.SE

keywords SystemsEngineeringSocio-technicalLargeLanguageModelsAIadoptionthinkingAlignmentandreliabilityLLMchallengesSurvey

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that adopting Large Language Models (LLMs) into critical socio-technical systems should be treated primarily as a systems engineering problem, not a model-centric AI problem. The authors claim the systems engineering approach is better equipped than the AI community alone to facilitate LLM adoption because it prioritises the problem and its context before any technological solution. To support this, they survey 24 studies from 2017 onward that apply systems engineering principles to AI-based systems and map those principles onto four LLM-generated challenges: alignment and reliability, interpretability and accountability, maintainability and sustainability, and security and privacy. The survey finds most existing work addresses alignment and reliability using the Systems Views, Top-Down, and Problem-Solving Cycle principles, while the more dynamic principles remain underused. If the argument is right, the path to dependable LLM adoption runs through requirements definition, stakeholder analysis, lifecycle governance, and public engagement.

What carries the argument

The organising mechanism is Haberfellner et al.'s classification of six systems engineering principles into two categories: Systems Thinking, which groups Systems Views, Agility Systems, and System Dynamics, and the Systems Engineering Process Model, which groups Top-Down, Variant Creation, and Problem-Solving Cycle. The survey uses this classification as a coding scheme to map each of the 24 selected papers onto the principles it applies, and then relates those principles to the four LLM challenge areas. The classification carries the argument by showing that the most-used principles are static and knowledge-intensive, while the principles needed for LLMs' emergent and rapidly changing behaviour are the least used.

What would settle it

A documented real-world case where an LLM-based system, engineered with full top-down requirements decomposition, verification checkpoints, and lifecycle governance, still fails because of an emergent LLM-specific behaviour such as a jailbreak or hallucination that no system-level requirement could have captured would refute the transferability claim.

Watch

Extended reading notes

Core claim

The central discovery is that the systems engineering principles developed for earlier AI/ML systems already speak to the challenges LLMs pose, so LLM adoption can be guided by an existing toolkit rather than starting from scratch. The authors classify principles into Systems Thinking (Systems Views, Agility Systems, System Dynamics) and the Systems Engineering Process Model (Top-Down, Variant Creation, Problem-Solving Cycle), then show that 21 of the 24 surveyed papers address alignment and reliability, with Systems Views, Top-Down, and Problem-Solving Cycle dominating. The underuse of Agility Systems, System Dynamics, and Variant Creation is identified as a gap, because those dynamic and flexible principles are the ones best suited to the fast-changing LLM landscape. The paper's conclusion is that the systems engineering approach offers a good starting point for addressing LLM challenges, provided the community shifts toward a problem-first, context-first culture.

Load-bearing premise

The argument assumes that the difficulties LLMs introduce, such as hallucination, black-box opacity, intellectual debt, and carbon cost, are close enough to earlier AI/ML system difficulties that systems engineering principles shown on ML-based systems will transfer to LLM-based socio-technical systems.

Editorial extensions

If this is right

Adopting LLMs in critical domains should start with problem and context analysis, not model selection, and stakeholders need to be mapped before components are chosen.
The systems engineering principles already used for AI systems — Systems Views, Top-Down, and Problem-Solving Cycle — can be applied directly to LLM-based systems to mitigate alignment and reliability failures.
The underused dynamic principles (Agility Systems, System Dynamics, Variant Creation) are the ones most needed to keep pace with LLMs' rapid change and emergent behaviour.
Public engagement and inclusive requirements definition become core engineering tasks, not optional extras, because LLM systems affect populations that do not speak the developers' language.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims: if the problem-first thesis is right, LLM procurement and deployment in government and healthcare should be led by systems engineers rather than data scientists — a staffing implication the authors do not state.
A testable extension: compare projects that follow problem-first systems engineering against model-first AI projects on outcomes such as time-to-deployment, incident rate, and stakeholder satisfaction; the paper's argument predicts the SE-led projects will show fewer alignment failures.
The paper implies that 'prompt engineering' and model fine-tuning are solution-level activities that should come after an explicit systems context; this reorders the current LLM adoption playbook.
A neighbouring problem the paper opens: how to make the dynamic principles operational — for instance, integrating System Dynamics with the regulatory approval and safety assurance frameworks that currently assume static requirements.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is the systematic mapping: six systems engineering principles against four LLM-era challenge areas, with a clear visual showing which principles dominate (Systems Views, Top-Down, Problem-Solving Cycle) and which are neglected (Agility Systems, System Dynamics, Variant Creation). That gap analysis is new and practically useful, and the authors deserve credit for shipping the survey artifacts and describing the selection chain from 3,504 to 24 papers. The writing is clear and the related work is honestly engaged.

The soft spots are concentrated in the normative framing. The abstract says SE is "better equipped" to facilitate LLM adoption, but the survey design has no comparison arm: every selected paper uses systems engineering by construction, so the finding is that SE has been used, not that it works better than alternatives. The transferability assumption is also thin. Most surveyed work targets pre-LLM ML systems, and the paper does not argue seriously that hallucination, prompt-level interaction, or black-box LLM components are similar enough to prior AI failure modes for those principles to carry over. Their own §6.4 concedes the most-used principles are static and rely on prior knowledge, which cuts against the central claim.

That said, the core descriptive result holds up: the mapping is plausible, the underused principles are worth attention, and the paper explicitly flags limitations. It is a programmatic survey rather than a proof, and the conclusions would be more accurate framed as a research direction than as a comparative verdict. I would not change the evidence, but I would rewrite the abstract and conclusions to match the evidence.

This deserves serious peer review. A good referee can push the authors to soften the comparative claim and to engagement with LLM-specific failure modes. For readers interested in how systems engineering can inform LLM deployment, the survey is a useful entry point and I would cite it.

Referee Report

3 major / 7 minor

Summary. The paper argues that adopting Large Language Models (LLMs) into socio-technical systems is better served by a Systems Engineering (SE) approach than by model-centric AI research alone. It identifies four categories of LLM-induced challenges in socio-technical systems (alignment and reliability, interpretability and accountability, maintainability and sustainability, security and privacy) and reports a semi-automated survey that narrows 3,504 candidate papers down to 24 papers describing SE-style work on AI-based systems. The survey maps the SE principles used in those papers (Systems Views, Agility Systems, System Dynamics, Top-Down, Variant Creation, Problem-Solving Cycle) onto the four challenge categories, concluding that SE principles are a good starting point for addressing LLM challenges and identifying open research directions.

Significance. If the central claim were fully supported, the paper would be valuable for reframing LLM adoption as a socio-technical systems engineering problem, complementing the dominant model-centric view. The paper has notable strengths: a transparent survey pipeline with publicly archived code and data, a clear four-part taxonomy of LLM challenges, and a systematic mapping of SE principles to those challenges. These artifacts make the descriptive survey reproducible and useful for researchers planning SE-informed LLM work. The provisional findings, such as the heavy reliance on static principles and the relative neglect of Agility Systems and Variant Creation, are suggestive and merit further investigation. However, the comparative and transferability claims are not established by the reported evidence, so the significance is conditional on a substantive revision of the paper's conclusions.

major comments (3)

[§4 and Abstract/§7] The comparative claim that the Systems Engineering approach is "better equipped" to facilitate LLM adoption is not supported by the survey design. The search query in §4 requires at least one of "systems engineering", "systems thinking", "dependable systems", or "engineering AI", so every one of the 24 selected papers is an SE-style contribution by construction. There is no comparison arm of non-SE approaches and no outcome measure against which SE and non-SE methods are evaluated. The evidence can support the weaker concluding statement in §7 that SE "offers a good starting point," but not the abstract's comparative "better equipped." This wording should either be removed/weakened or the study should be redesigned with a baseline and an evaluation criterion.
[§5 and §7] The transferability of the surveyed principles to LLM-based systems is assumed rather than argued. Many of the surveyed works address pre-LLM machine learning systems, including RL alignment (Meyer and Gruhn 2019), ML technology readiness (Lavin et al. 2022), and healthcare ML (Salwei and Carayon 2022). The abstract frames these as "similar issues to the ones LLMs pose," but the paper does not analyze whether LLM-specific behaviors such as hallucination, prompt-level interaction, black-box API components, and emergent capabilities change the engineering problem qualitatively. Without such an analysis, the recommendation to apply these SE principles to LLM systems is an hypothesis rather than a finding. The authors should either provide a per-challenge transferability argument grounded in §3 or explicitly frame their conclusion as a research direction.
[§6.4] Section 6.4 concedes that the most applied principles (Systems Views, Top-Down, Problem-Solving Cycle) are "static and rely on prior knowledge," which is a poor match for the rapidly changing LLM landscape. This concession is in tension with the concluding recommendation in §7 that these very principles offer a good starting point for addressing LLM challenges. The paper does not resolve this tension or explain how static, prior-knowledge-reliant principles can be a starting point for a technology whose capabilities and failure modes change quickly. A concrete proposal, such as pairing static principles with the dynamic ones (Agility Systems, System Dynamics, Variant Creation) in a specific workflow, would strengthen the argument.

minor comments (7)

[§3.4] The phrase "sensible data" appears twice; it should read "sensitive data."
[§5.4] The text refers to the "MLTR framework" when describing Lavin et al. (2022); elsewhere the paper and reference list use "MLTRL." Please make the acronym consistent.
[§5.2] The phrase "above 30% of works" is vague; please report the exact count of papers addressing interpretability and accountability (e.g., "8 of 24 papers") and the corresponding counts for each principle in the radar chart.
[§4] The manual filtering and snowballing steps are described numerically, but the inclusion and exclusion criteria are not stated. Since the authors provide code and data, adding a short list of criteria would substantially improve reproducibility.
[Figure 1] The y-axis of Figure 1 is unlabeled; please label it (e.g., "Number of papers") and clarify whether the counts refer to the number of papers applying a given principle or the number of challenge-principle pairs.
[§5.1] The sentence "Yu et al., 2024 proposes five viewpoints" is grammatically inconsistent; "proposes" should be "propose" because the reference is plural.
[References] Several author names contain rendering artifacts, such as "M ¨okander" and "V oirin"; please check the encoding before final submission.

Circularity Check

1 steps flagged · score 2.0 of 10

Survey framing and selection criteria make the SE conclusion partly definitional, but the paper contains no formal circular derivation.

self definitional [Section 4 (Survey Methodology) and Section 3.1]
"We use the following query as an input to search for the relevant papers: (”systems engineering” OR ”systems thinking” OR ”dependable systems” OR ”engineering AI”) AND (”generative ai” OR ”large language model” OR ”llm” OR ”artificial intelligence” OR ”ai” OR ”ml” OR ”deep learning”). ... Alignment issues emerge when systems do not accomplish their requirements (Bastidas et al., 2022)."

The survey query guarantees that every retrieved paper uses systems-engineering vocabulary, so the observation that systems engineering principles have been applied to AI/LLM issues is largely true by construction of the corpus. At the same time, the LLM challenges are pre-framed in Section 3 as systems-engineering categories (Alignment, Interpretability, Maintainability, Security). The conclusion that 'the systems engineering approach offers a good starting point for addressing the LLMs’ challenges' therefore partly restates the inclusion criterion and the challenge taxonomy rather than emerging from a comparative evaluation. The stronger abstract claim that SE is 'better equipped' is not derivable from the survey because no non-SE baseline is included.

full rationale

This is a survey and position argument, not a quantitative derivation: no parameters are fitted, no predictions are made, and no mathematical result is claimed. The surveyed papers are external to the authors, and the self-citations present (e.g., Cabrera et al. 2023/2024, Bastidas et al. 2022) are not load-bearing for the central claim. The main circularity concern is structural rather than formal: the challenge taxonomy uses systems-engineering terms, and the literature search requires systems-engineering terms, so the finding that systems-engineering principles address these challenges is partly guaranteed by the paper's own framing and selection. That is a mild tautology/selection-bias issue that weakens the comparative force of the abstract claim ('better equipped'), but it does not make the survey's descriptive content circular. The paper's own Section 6.4 also concedes that the most-used principles are static and rely on prior knowledge, further limiting the strength of the conclusion. Overall, the central claim is an arguable interpretation of external evidence rather than a derivation equivalent to its inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the framing assumption that LLM challenges are continuous with earlier AI/ML challenges, on the representativeness of the 24-paper sample, and on the validity of a single textbook taxonomy for coding the papers. The one hidden methodological choice is the semantic similarity threshold used in the selection funnel.

free parameters (1)

Semantic similarity threshold for paper selection = not reported
The semi-automated survey tool filters 3,504 papers down to 253 based on a semantic embedding similarity threshold that is not stated in the paper or in the parameters file description. This threshold affects which papers the authors read and therefore which findings the survey reports.

assumptions (3)

domain assumption Haberfellner et al.'s six principles are a complete and valid taxonomy for classifying systems engineering work on AI-based systems.
Section 5 introduces the classification using Haberfellner et al. (2019) as the sole framework, with no comparison to other systems engineering taxonomies and no justification that these six principles exhaust the relevant space.
domain assumption The 24 retained papers are representative of research applying systems engineering to AI-based systems.
Section 4 describes a subjective funnel from 3,504 papers through a semantic filter, manual reading, and snowballing, but it provides no evidence that the final set is representative beyond the authors' screening judgment.
domain assumption LLM challenges are analogous to the challenges of earlier AI/ML systems.
The abstract frames the survey as addressing 'similar issues to the ones LLMs pose,' and Section 5 maps principles from papers about ML-based systems to LLM challenges. The paper does not test whether LLM-specific behaviours such as hallucination and emergent prompting change the engineering problem qualitatively.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Systems Engineering Approach in Times of Large Language Models." pith.science (2026). https://pith.science/paper/25IIDOKP

@misc{pith2026241109050,
  author       = {Pith},
  title        = {Pith review of: The Systems Engineering Approach in Times of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25IIDOKP}},
  note         = {Machine review of arXiv:2411.09050}
}

read the original abstract

Using Large Language Models (LLMs) to address critical societal problems requires adopting this novel technology into socio-technical systems. However, the complexity of such systems and the nature of LLMs challenge such a vision. It is unlikely that the solution to such challenges will come from the Artificial Intelligence (AI) community itself. Instead, the Systems Engineering approach is better equipped to facilitate the adoption of LLMs by prioritising the problems and their context before any other aspects. This paper introduces the challenges LLMs generate and surveys systems research efforts for engineering AI-based systems. We reveal how the systems engineering principles have supported addressing similar issues to the ones LLMs pose and discuss our findings to provide future directions for adopting LLMs.

Figures

Figures reproduced from arXiv: 2411.09050 by the authors.

**Figure 2.** Radar chart comparing applied principles [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Radar chart comparing applied principles [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Radar chart comparing applied principles [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Radar chart comparing applied principles [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages

[1]

Adedjouma, M., Alix, C., Cantat, L., Jenn, E., Mattioli, J., Robert, B., Tschirhart, F., & V oirin, J.-L. (2022). Engineering dependable ai systems. 2022 17th Annual System of Systems Engineering Conference (SOSE) , 458–463. https : / / doi . org / 10 . 1109 / SOSE55472 . 2022 . 9812672 Bastidas, V ., Reychav, I., Ofir, A., Bezbradica, M., & Helfert, M. (...

work page 2022
[5]

https://doi.org/10.1145/3643657.3643910 Cabrera, C., Paleyes, A., Thodoroff, P., & Lawrence, N. D. (2023). Real-world machine learning systems: A survey from a data-oriented architecture perspective. arXiv preprint arXiv:2302.04810. Cai, Y . (2020). Safety analytics for ai systems [Cited by: 0]. Lecture Notes in Computer Science (including subseries Lectu...

arXiv 2023
[29]

Srivastava, A., Bhardwaj, S., & Saraswat, S. (2017). Scrum model for agile methodology. 2017 International Conference on Computing, Communication and Automation (ICCCA) , 864–869. Wade, J., Buenfil, J., & Collopy, P. (2020). A systems engineering approach for artificial intelligence: Inspired by the vlsi revolution of mead & conway. INSIGHT, 23(1), 41–47....

arXiv 2017
[80]

S., Ajagekar, A., & You, F

Decardi-Nelson, B., Alshehri, A. S., Ajagekar, A., & You, F. (2024). Generative ai and process systems engineering: The next frontier. arXiv preprint arXiv:2402.10977. Dzambic, M., Dobaj, J., Seidl, M., & Macher, G. (2022). Architectural patterns for integrating ai technology into safety-critical systems. Proceedings of the 26th European Conference on Pat...

arXiv 2024
[6039]

Llinas, J., Fouad, H., & Mittu, R. (2021). Systems engineering for artificial intelligence-based systems: A review in time. In W. F. Lawless, R. Mittu, D. A. Sofge, T. Shortell, & T. A. McDermott (Eds.), Systems engineering and artificial intelligence (pp. 93–113). Springer International Publishing. https : / / doi . org / 10 . 1007/978-3-030-77283-3 6 Ma...

work page arXiv 2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Adedjouma, M., Alix, C., Cantat, L., Jenn, E., Mattioli, J., Robert, B., Tschirhart, F., & V oirin, J.-L. (2022). Engineering dependable ai systems. 2022 17th Annual System of Systems Engineering Conference (SOSE) , 458–463. https : / / doi . org / 10 . 1109 / SOSE55472 . 2022 . 9812672 Bastidas, V ., Reychav, I., Ofir, A., Bezbradica, M., & Helfert, M. (...

work page 2022

[2] [5]

https://doi.org/10.1145/3643657.3643910 Cabrera, C., Paleyes, A., Thodoroff, P., & Lawrence, N. D. (2023). Real-world machine learning systems: A survey from a data-oriented architecture perspective. arXiv preprint arXiv:2302.04810. Cai, Y . (2020). Safety analytics for ai systems [Cited by: 0]. Lecture Notes in Computer Science (including subseries Lectu...

arXiv 2023

[3] [29]

Srivastava, A., Bhardwaj, S., & Saraswat, S. (2017). Scrum model for agile methodology. 2017 International Conference on Computing, Communication and Automation (ICCCA) , 864–869. Wade, J., Buenfil, J., & Collopy, P. (2020). A systems engineering approach for artificial intelligence: Inspired by the vlsi revolution of mead & conway. INSIGHT, 23(1), 41–47....

arXiv 2017

[4] [80]

S., Ajagekar, A., & You, F

Decardi-Nelson, B., Alshehri, A. S., Ajagekar, A., & You, F. (2024). Generative ai and process systems engineering: The next frontier. arXiv preprint arXiv:2402.10977. Dzambic, M., Dobaj, J., Seidl, M., & Macher, G. (2022). Architectural patterns for integrating ai technology into safety-critical systems. Proceedings of the 26th European Conference on Pat...

arXiv 2024

[5] [6039]

Llinas, J., Fouad, H., & Mittu, R. (2021). Systems engineering for artificial intelligence-based systems: A review in time. In W. F. Lawless, R. Mittu, D. A. Sofge, T. Shortell, & T. A. McDermott (Eds.), Systems engineering and artificial intelligence (pp. 93–113). Springer International Publishing. https : / / doi . org / 10 . 1007/978-3-030-77283-3 6 Ma...

work page arXiv 2021