{"id":"13972758-e9fe-43b1-b17a-2bc2fe090786","arxiv_id":"2411.09050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A survey of 24 papers finds that Systems Views, Top-Down, and Problem-Solving Cycle are the most used systems engineering principles for AI-based systems, and the authors argue these should guide LLM adoption.","lead":"This paper surveys 24 research papers to map how systems engineering principles are used to build AI-based systems, and applies this map to the challenges of adopting large language models. It argues that a systems engineering approach, which starts from problems and context rather than technology, is the right frame for integrating LLMs into critical socio-technical systems.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey's selection criteria guarantee all 24 papers use systems engineering, so the comparative claim 'better equipped' has no baseline; LLM-specific transferability is asserted, not demonstrated.","rationale":"The reader's conditional verdict is appropriate, and I agree that transferability is unproven. However, I see the more basic problem as selection bias: the survey's inclusion criteria ensure every selected paper uses systems engineering, so the evidence cannot support the comparative 'better equipped' claim even if transferability were granted. The paper is transparent about its method and artifacts, and the descriptive finding about which SE principles dominate is plausible, so this is not a reason to reject. But the abstract's normative claim is stronger than the survey design can bear. The conditional verdict already captures this: the claim would need either a softened framing or additional comparative and LLM-specific evidence. My concrete test would directly test whether a non-SE control corpus exists and whether the selected papers are actually LLM-related, which would settle whether the concern lands.","tokens_in":12752,"tokens_out":4460,"duration_ms":49109,"concrete_test":"Run a control search replicating the §4 pipeline but with the SE-terms OR-clause removed and the AI/LLM terms kept; sample 100 retrieved papers and code them for whether they invoke SE principles and whether they address the four challenge categories of §3. Separately, count how many of the 24 selected papers mention LLMs or generative AI in title or abstract. If a substantial share of non-SE papers address the same LLM challenges, or if almost no selected paper is LLM-specific, then the central claim is unsupported by the survey.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the inference from 'systems engineering principles have been used to address AI challenges' to 'systems engineering is better equipped to facilitate LLM adoption'. The survey method (§4) makes this inference untestable: the search query requires at least one of 'systems engineering', 'systems thinking', 'dependable systems', or 'engineering AI', so every one of the 24 selected papers is an instance of SE-style work by construction. There is no comparison arm and no outcome measure, so the comparative 'better equipped' has no evidentiary basis. The transferability assumption is equally unsecured: most surveyed work targets pre-LLM ML (e.g., RL alignment in Meyer & Gruhn 2019, MLTRL in Lavin et al. 2022, healthcare ML in Salwei & Carayon 2022), and the paper supplies no argument that hallucination, prompt-level interaction, and black-box LLM components are 'similar issues' rather than qualitatively new failure modes. The paper's own §6.4 concedes that the most used principles (Systems Views, Top-Down, Problem-Solving Cycle) are static and rely on prior knowledge, which is a poor match for a rapidly changing LLM landscape. The conclusion in §7 is therefore a research direction dressed as a comparative finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that adopting Large Language Models (LLMs) into socio-technical systems is better served by a Systems Engineering (SE) approach than by model-centric AI research alone. It identifies four categories of LLM-induced challenges in socio-technical systems (alignment and reliability, interpretability and accountability, maintainability and sustainability, security and privacy) and reports a semi-automated survey that narrows 3,504 candidate papers down to 24 papers describing SE-style work on AI-based systems. The survey maps the SE principles used in those papers (Systems Views, Agility Systems, System Dynamics, Top-Down, Variant Creation, Problem-Solving Cycle) onto the four challenge categories, concluding that SE principles are a good starting point for addressing LLM challenges and identifying open research directions.","tokens_in":12908,"tokens_out":3362,"duration_ms":34091,"significance":"If the central claim were fully supported, the paper would be valuable for reframing LLM adoption as a socio-technical systems engineering problem, complementing the dominant model-centric view. The paper has notable strengths: a transparent survey pipeline with publicly archived code and data, a clear four-part taxonomy of LLM challenges, and a systematic mapping of SE principles to those challenges. These artifacts make the descriptive survey reproducible and useful for researchers planning SE-informed LLM work. The provisional findings, such as the heavy reliance on static principles and the relative neglect of Agility Systems and Variant Creation, are suggestive and merit further investigation. However, the comparative and transferability claims are not established by the reported evidence, so the significance is conditional on a substantive revision of the paper's conclusions.","major_comments":[{"comment":"The comparative claim that the Systems Engineering approach is \"better equipped\" to facilitate LLM adoption is not supported by the survey design. The search query in §4 requires at least one of \"systems engineering\", \"systems thinking\", \"dependable systems\", or \"engineering AI\", so every one of the 24 selected papers is an SE-style contribution by construction. There is no comparison arm of non-SE approaches and no outcome measure against which SE and non-SE methods are evaluated. The evidence can support the weaker concluding statement in §7 that SE \"offers a good starting point,\" but not the abstract's comparative \"better equipped.\" This wording should either be removed/weakened or the study should be redesigned with a baseline and an evaluation criterion.","section":"§4 and Abstract/§7"},{"comment":"The transferability of the surveyed principles to LLM-based systems is assumed rather than argued. Many of the surveyed works address pre-LLM machine learning systems, including RL alignment (Meyer and Gruhn 2019), ML technology readiness (Lavin et al. 2022), and healthcare ML (Salwei and Carayon 2022). The abstract frames these as \"similar issues to the ones LLMs pose,\" but the paper does not analyze whether LLM-specific behaviors such as hallucination, prompt-level interaction, black-box API components, and emergent capabilities change the engineering problem qualitatively. Without such an analysis, the recommendation to apply these SE principles to LLM systems is an hypothesis rather than a finding. The authors should either provide a per-challenge transferability argument grounded in §3 or explicitly frame their conclusion as a research direction.","section":"§5 and §7"},{"comment":"Section 6.4 concedes that the most applied principles (Systems Views, Top-Down, Problem-Solving Cycle) are \"static and rely on prior knowledge,\" which is a poor match for the rapidly changing LLM landscape. This concession is in tension with the concluding recommendation in §7 that these very principles offer a good starting point for addressing LLM challenges. The paper does not resolve this tension or explain how static, prior-knowledge-reliant principles can be a starting point for a technology whose capabilities and failure modes change quickly. A concrete proposal, such as pairing static principles with the dynamic ones (Agility Systems, System Dynamics, Variant Creation) in a specific workflow, would strengthen the argument.","section":"§6.4"}],"minor_comments":[{"comment":"The phrase \"sensible data\" appears twice; it should read \"sensitive data.\"","section":"§3.4"},{"comment":"The text refers to the \"MLTR framework\" when describing Lavin et al. (2022); elsewhere the paper and reference list use \"MLTRL.\" Please make the acronym consistent.","section":"§5.4"},{"comment":"The phrase \"above 30% of works\" is vague; please report the exact count of papers addressing interpretability and accountability (e.g., \"8 of 24 papers\") and the corresponding counts for each principle in the radar chart.","section":"§5.2"},{"comment":"The manual filtering and snowballing steps are described numerically, but the inclusion and exclusion criteria are not stated. Since the authors provide code and data, adding a short list of criteria would substantially improve reproducibility.","section":"§4"},{"comment":"The y-axis of Figure 1 is unlabeled; please label it (e.g., \"Number of papers\") and clarify whether the counts refer to the number of papers applying a given principle or the number of challenge-principle pairs.","section":"Figure 1"},{"comment":"The sentence \"Yu et al., 2024 proposes five viewpoints\" is grammatically inconsistent; \"proposes\" should be \"propose\" because the reference is plural.","section":"§5.1"},{"comment":"Several author names contain rendering artifacts, such as \"M ¨okander\" and \"V oirin\"; please check the encoding before final submission.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The survey artifacts and descriptive mapping are useful, but the comparative claim in the abstract is not supported by the selection procedure and the transferability assumption is unexamined. If the authors revise the claims to match the evidence and address the §6.4 tension, the paper could be acceptable; in its current form, the central claim overreaches. This is a substantive revision rather than a cosmetic one, hence major_revision rather than minor_revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the systematic mapping: six systems engineering principles against four LLM-era challenge areas, with a clear visual showing which principles dominate (Systems Views, Top-Down, Problem-Solving Cycle) and which are neglected (Agility Systems, System Dynamics, Variant Creation). That gap analysis is new and practically useful, and the authors deserve credit for shipping the survey artifacts and describing the selection chain from 3,504 to 24 papers. The writing is clear and the related work is honestly engaged.\n\nThe soft spots are concentrated in the normative framing. The abstract says SE is \"better equipped\" to facilitate LLM adoption, but the survey design has no comparison arm: every selected paper uses systems engineering by construction, so the finding is that SE has been used, not that it works better than alternatives. The transferability assumption is also thin. Most surveyed work targets pre-LLM ML systems, and the paper does not argue seriously that hallucination, prompt-level interaction, or black-box LLM components are similar enough to prior AI failure modes for those principles to carry over. Their own §6.4 concedes the most-used principles are static and rely on prior knowledge, which cuts against the central claim.\n\nThat said, the core descriptive result holds up: the mapping is plausible, the underused principles are worth attention, and the paper explicitly flags limitations. It is a programmatic survey rather than a proof, and the conclusions would be more accurate framed as a research direction than as a comparative verdict. I would not change the evidence, but I would rewrite the abstract and conclusions to match the evidence.\n\nThis deserves serious peer review. A good referee can push the authors to soften the comparative claim and to engagement with LLM-specific failure modes. For readers interested in how systems engineering can inform LLM deployment, the survey is a useful entry point and I would cite it.","headline":"Useful mapping of SE principles to LLM-era challenges, but the 'better equipped' claim in the abstract outruns what the survey can actually support.","tokens_in":13501,"tokens_out":815,"would_cite":true,"duration_ms":10121,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that adopting Large Language Models into critical socio-technical systems should be treated as a systems engineering problem — prioritise the problem and its context before the technology — and supports this by showing…","keywords":["Systems Engineering","Socio-technical Systems","Large Language Models","AI adoption","Systems thinking","Alignment and reliability","LLM challenges","Survey"],"falsifier":"A documented real-world case where an LLM-based system, engineered with full top-down requirements decomposition, verification checkpoints, and lifecycle governance, still fails because of an emergent LLM-specific behaviour such as a jailbreak or hallucination that no system-level requirement could have captured would refute the transferability claim.","tokens_in":12489,"feed_emoji":"⚙️","tokens_out":5624,"duration_ms":47706,"temperature":0.7,"pith_summary":"This paper argues that adopting Large Language Models (LLMs) into critical socio-technical systems should be treated primarily as a systems engineering problem, not a model-centric AI problem. The authors claim the systems engineering approach is better equipped than the AI community alone to facilitate LLM adoption because it prioritises the problem and its context before any technological solution. To support this, they survey 24 studies from 2017 onward that apply systems engineering principles to AI-based systems and map those principles onto four LLM-generated challenges: alignment and reliability, interpretability and accountability, maintainability and sustainability, and security and privacy. The survey finds most existing work addresses alignment and reliability using the Systems Views, Top-Down, and Problem-Solving Cycle principles, while the more dynamic principles remain underused. If the argument is right, the path to dependable LLM adoption runs through requirements definition, stakeholder analysis, lifecycle governance, and public engagement.","feed_headline":"Systems engineering, not AI alone, should lead LLM adoption","feed_subtitle":"A survey of 24 studies shows problem-first principles already tame the hardest issues in AI systems.","key_machinery":"The organising mechanism is Haberfellner et al.'s classification of six systems engineering principles into two categories: Systems Thinking, which groups Systems Views, Agility Systems, and System Dynamics, and the Systems Engineering Process Model, which groups Top-Down, Variant Creation, and Problem-Solving Cycle. The survey uses this classification as a coding scheme to map each of the 24 selected papers onto the principles it applies, and then relates those principles to the four LLM challenge areas. The classification carries the argument by showing that the most-used principles are static and knowledge-intensive, while the principles needed for LLMs' emergent and rapidly changing behaviour are the least used.","core_discovery":"The central discovery is that the systems engineering principles developed for earlier AI/ML systems already speak to the challenges LLMs pose, so LLM adoption can be guided by an existing toolkit rather than starting from scratch. The authors classify principles into Systems Thinking (Systems Views, Agility Systems, System Dynamics) and the Systems Engineering Process Model (Top-Down, Variant Creation, Problem-Solving Cycle), then show that 21 of the 24 surveyed papers address alignment and reliability, with Systems Views, Top-Down, and Problem-Solving Cycle dominating. The underuse of Agility Systems, System Dynamics, and Variant Creation is identified as a gap, because those dynamic and flexible principles are the ones best suited to the fast-changing LLM landscape. The paper's conclusion is that the systems engineering approach offers a good starting point for addressing LLM challenges, provided the community shifts toward a problem-first, context-first culture.","pith_inferences":["Beyond the paper's claims: if the problem-first thesis is right, LLM procurement and deployment in government and healthcare should be led by systems engineers rather than data scientists — a staffing implication the authors do not state.","A testable extension: compare projects that follow problem-first systems engineering against model-first AI projects on outcomes such as time-to-deployment, incident rate, and stakeholder satisfaction; the paper's argument predicts the SE-led projects will show fewer alignment failures.","The paper implies that 'prompt engineering' and model fine-tuning are solution-level activities that should come after an explicit systems context; this reorders the current LLM adoption playbook.","A neighbouring problem the paper opens: how to make the dynamic principles operational — for instance, integrating System Dynamics with the regulatory approval and safety assurance frameworks that currently assume static requirements."],"forward_implications":["Adopting LLMs in critical domains should start with problem and context analysis, not model selection, and stakeholders need to be mapped before components are chosen.","The systems engineering principles already used for AI systems — Systems Views, Top-Down, and Problem-Solving Cycle — can be applied directly to LLM-based systems to mitigate alignment and reliability failures.","The underused dynamic principles (Agility Systems, System Dynamics, Variant Creation) are the ones most needed to keep pace with LLMs' rapid change and emergent behaviour.","Public engagement and inclusive requirements definition become core engineering tasks, not optional extras, because LLM systems affect populations that do not speak the developers' language."],"supporting_citations":[{"why":"Supplies the two-category classification of systems engineering principles used to code the surveyed papers.","marker":"Haberfellner et al. (2019)"},{"why":"Supplies the systematic review methodology and semi-automated selection process the survey follows.","marker":"Kitchenham & Brereton (2013)"},{"why":"Defines socio-technical systems and their social and technical subsystems, the object of analysis.","marker":"Baxter & Sommerville (2010)"},{"why":"Characterises generative AI and LLM properties such as probabilistic outputs, black-box operation, and carbon footprint that create the challenges.","marker":"Feuerriegel et al. (2024)"},{"why":"Documents specific LLM risks, including hallucination, bias, and privacy extraction, that the survey maps to systems challenges.","marker":"D’Antonoli et al. (2024)"},{"why":"Introduces intellectual debt, which the paper uses to frame interpretability and accountability issues in LLM-based systems.","marker":"Zittrain (2022)"},{"why":"Establishes the environmental cost of large models, supporting the sustainability challenge.","marker":"Schwartz et al. (2020)"},{"why":"Offers the MLTRL framework, one of the surveyed systems engineering responses used for alignment, reliability, and maintainability.","marker":"Lavin et al. (2022)"}],"fun_headline_variants":["LLM adoption needs systems engineering, not more AI hype","Problem-first systems engineering can guide LLM adoption","Systems engineering principles already address LLM challenges","Use systems engineering's underused dynamic tools for LLM era","LLM challenges need systems engineering's problem-first approach"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the difficulties LLMs introduce, such as hallucination, black-box opacity, intellectual debt, and carbon cost, are close enough to earlier AI/ML system difficulties that systems engineering principles shown on ML-based systems will transfer to LLM-based socio-technical systems.","fun_headline_variants_meta":{"raw":{"variants":["LLM adoption needs systems engineering, not more AI hype","Problem-first systems engineering can guide LLM adoption","Systems engineering principles already address LLM challenges","Use systems engineering's underused dynamic tools for LLM era","LLM challenges need systems engineering's problem-first approach"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2012,"prompt_tokens":832,"completion_tokens":1180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1104}},"tokens_in":448,"tokens_out":1180,"duration_ms":9556,"temperature":1.0,"reasoning_tokens":1104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:06:48.734502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A documented real-world case where an LLM-based system, engineered with full top-down requirements decomposition, verification checkpoints, and lifecycle governance, still fails because of an emergent LLM-specific behaviour such as a jailbreak or hallucination that no system-level requirement could have captured would refute the transferability claim.","supporting_citations":[],"review_version":1}