{"id":"b6f548a2-9299-4b5c-934f-b056ef77516e","arxiv_id":"2501.07992","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new holonic system-of-systems architecture gives every component an LLM-powered reasoning layer and four management roles, but it is proposed and sketched, never implemented or measured.","lead":"This paper proposes a blueprint for coordinating independent machines, such as self-driving cars and delivery drones, by giving each one an AI language-model 'brain' that plans, talks to the other machines, and understands plain-language orders from people. The design is illustrated through a cross-town trip that mixes driving and flying, but no system was built or tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of real-time adaptability rests on unverified LLM control-loop reliability; paper defers the decisive measurement.","rationale":"The reader's weakest assumption—that an LLM can serve as a reliable real-time reasoning core—is precisely the load-bearing concern. The paper's abstract and Section 5 assert real-time adaptability and consistent reasoning, but every supporting component is deferred: the evaluation section calls the work conceptual, the metrics are proposals, and the suggested GPT-3 stack raises genuine latency and correctness risks in safety-relevant control. I find no internal contradiction that invalidates the architecture more fundamentally; rather, the central claim is empirically unsubstantiated and faces a concrete, plausible failure mode. The reader and I agree on this point, and the REJECT verdict remains appropriate because the paper's strongest claims outpace its evidence. A constructive path exists: the proposed simulated evaluation, if executed with latency and correctness measurements, could upgrade the work to CONDITIONAL or ACCEPT.","tokens_in":95,"tokens_out":1624,"duration_ms":30119,"concrete_test":"Implement a minimal two-holon prototype (task holon and machine resource holon) in ROS2/Gazebo as Section 6.1 suggests, using fine-tuned GPT-3 in the reasoning layer. Inject scripted events (road closure, reroute) and measure end-to-end latency from event detection to actuator command, plus action-plan correctness (e.g., collision-free route) over 100 runs. If p95 latency exceeds the control deadline or any malformed/hallucinated command reaches the actuator, the paper's real-time adaptability claim is not supported without additional safeguards.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The architecture's value claim—Section 5's 'consistent reasoning capabilities... real-time adaptability'—depends on the LLM serving as 'the central reasoning core' of every holon (Section 3.1), generating correct action plans within operational deadlines. This premise is unevaluated: Section 6 explicitly labels the work 'a conceptual framework,' and Section 6.2 lists response time only as a future metric. The proposed implementation (Section 6.1) uses fine-tuned GPT-3, whose generation latency is nondeterministic and can exceed the control-loop interval for UGVs/UAVs; no latency bound, timeout, or degraded-mode fallback is specified. LLM hallucinations or syntax errors would propagate through the communication layer to physical actuators, with no verification mechanism described. A second assumption—that heterogeneous black-box CS (Section 1) can be wrapped into the uniform ROS-based holon facade (Section 3.1) without vendor cooperation—is also unshown, but the real-time control premise is the more load-bearing: if it fails, the 'autonomous adaptation' claim collapses even in a fully wrapped system. The paper acknowledges the risk only indirectly as a future 'response time' metric, so the strongest advertised benefit is currently an article of faith.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a layered holonic architecture for system of systems, in which each holon is decomposed into reasoning, communication, and capabilities layers, and introduces four specialized holon roles (supervisor, planner, task, resource) that use LLMs for decision-making. The architecture is illustrated with a 3D mobility scenario involving UGVs and UAVs for passenger transport, and the paper proposes evaluation metrics for future empirical work. The paper explicitly identifies itself as a conceptual framework and states that implementation and evaluation are future work.","tokens_in":8834,"tokens_out":4817,"duration_ms":46571,"significance":"If the architecture worked as claimed, it would advance the state of the art in holonic SoS architectures by adding LLM-based reasoning to holons and by importing manufacturing-inspired specialized holon roles into the SoS domain. The layered decomposition is clearly described, the figures and sequence diagram are helpful, and the paper is transparent about the lack of implementation. The main advertised benefits, especially \"real-time adaptability,\" however, rest on unverified assumptions about LLM reliability and latency, and the case study provides no empirical evidence.","major_comments":[{"comment":"The claim that the reasoning layer \"ensures real-time adaptability\" (abstract and Section 5) is not supported by any measurement, simulation, or feasibility argument. Section 6.1 proposes a fine-tuned GPT-3 for the reasoning layer, whose generation latency is nondeterministic and can exceed the control-loop intervals of UGVs/UAVs, yet no latency bound, timeout, or degraded-mode fallback is specified. Section 6.2 lists response time only as a future metric. Because the LLM is the \"central reasoning core\" of every holon, this unverified premise is load-bearing for the paper's central contribution.","section":"Section 3.1 / Section 5"},{"comment":"The case study is a narrative description with figures and a sequence diagram, not a demonstration. No simulation, prototype, or quantitative data are presented, so the claims in Section 5 about dynamic adjustment, resource reallocation, and improved interoperability are not evidenced. Section 6 explicitly states that this is a conceptual framework and that implementation in a simulated environment is future work, which means the paper's stated benefits are currently unsupported.","section":"Section 4"},{"comment":"The architecture assumes that heterogeneous black-box CS, which are \"unknown at design time\" (Section 1), can be wrapped into a uniform three-layer, ROS-based holon facade without vendor cooperation. No mechanism is described for generating the interface, protocol, or capability descriptions needed by the communication layer to invoke black-box CS capabilities. This assumption is load-bearing for the interoperability claim and is not substantiated in the paper.","section":"Section 3.1 / Section 1"},{"comment":"The assertion that this work is \"the first to introduce\" specialized holons in the SoS context is not supported by a systematic comparison. The cited ontological holonic architectures in [7] and [20] already represent CS as holons with roles and dynamic composition, so the novelty should be scoped more carefully, for example by focusing specifically on the LLM-enhanced reasoning layer rather than on the specialized holon roles per se.","section":"Section 5"}],"minor_comments":[{"comment":"The arXiv title contains a typo: \"LLM-Ehnanced\" should be \"LLM-Enhanced.\"","section":"Title / Abstract"},{"comment":"Terminology is inconsistent: Section 3.2 introduces \"Planner Holon\" and \"Supervisor Holon,\" but Section 4.2 later refers to \"Plan Holons\" and \"Supervise Holons.\" Please unify the terminology.","section":"Section 4.2"},{"comment":"Notation such as \"S-SoS\", \"S-CS1\", \"c1\", \"m1\", \"Pa\", \"Ta1\", and sub-plan identifiers is introduced without definitions, making the sequence diagram and text difficult to follow.","section":"Section 4.2"},{"comment":"The proposed evaluation metrics are listed but not operationalized; for example, \"adaptability\" and \"user satisfaction\" need concrete measurement procedures and comparison baselines.","section":"Section 6.2"},{"comment":"The phrase \"Human Resource Holon ... encapsulates humans as holistic entities\" is vague. The paper should clarify how user intent, goals, and status are captured, and how consent and privacy are handled.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The paper is honest in Section 6 about being a conceptual framework, and the architecture description is clear. However, the abstract and Section 5 use assertive language (\"ensures real-time adaptability,\" \"enabling each holon to adapt autonomously\") that goes beyond what the paper demonstrates. The entire evaluation is deferred to future work, and the case study is illustrative only. For a journal that publishes purely conceptual architecture papers, a major revision that tempers the claims and adds a concrete feasibility analysis of the LLM control loop might be acceptable; in its current form, however, the central contribution rests on an untested premise, so I cannot recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a coherent conceptual architecture proposal, not a piece of engineering. Its central claim—real-time adaptability via LLM reasoning—is entirely unmeasured, and the authors themselves say evaluation is future work. As a design document it's readable and honest; as a contribution to evidence it's thin.\n\nThe new bit is the layered holon (reasoning/communication/capabilities) and the transplant of manufacturing-style supervisor/planner/task/resource holons into SoS, plus a human resource holon. That combination is genuinely new in the SoS literature, and the 3D mobility case study does a decent job of illustrating the workflows. The paper is also honest in Section 6: it calls itself a conceptual framework and lists evaluation metrics instead of pretending to have results.\n\nThe soft spots are large but few. Section 5 claims 'consistent reasoning capabilities' and 'real-time adaptability'; Section 3.1 makes the LLM the central reasoning core. Neither is demonstrated, and the LLM failure modes—latency, hallucination, parsing errors—are precisely the ones that matter when plans control physical vehicles. The paper mentions future work on verification in Section 7, but the core premise remains untested. A second issue: Section 1 promises interoperability among black-box CS, but the architecture's uniform ROS-based communication layer presumes a common platform; how a black-box system gets wrapped is not addressed. Also, the 'first to introduce' claim (Section 5) is narrowly scoped to the SoS context, so it's probably fine, but the novelty is incremental—it follows the authors' own ICSOFT 2024 paper. These are not fatal to a design proposal, but they are fatal to the abstract's 'ensure real-time adaptability' phrasing.\n\nWho is this for? Readers working on SoS architectures who want a catalog of LLM-holon design options and a reference for specialized holon roles. It could be useful as a position piece. It is not ready for a journal that demands evidence; it belongs at a workshop or as a short paper with heavy hedging.\n\nRecommendation: I'd send it to peer review only if the venue tolerates conceptual papers; the right outcome might be reject with encouragement to implement the promised simulation and resubmit. As a desk decision, I'd lean toward asking for major revision rather than outright reject, because the design is clear and the problem is real. But the current version overclaims.","headline":"A clear conceptual architecture for LLM-driven holons whose central real-time claim is unverified; readable, honest, but too thin on evidence for acceptance.","tokens_in":9548,"tokens_out":2755,"would_cite":false,"duration_ms":26726,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a three-layer holonic architecture—reasoning, communication, capabilities—with large language models as the reasoning core of every holon, and adds supervisor, planner, task, and resource holons so that systems of…","keywords":["system of systems","holonic architecture","large language models","self-adaptive systems","interoperability","reconfigurability","human-system interaction","3D mobility"],"falsifier":"Take the paper's own 3D mobility scenario and inject a road closure after a task holon has committed to a route; if the LLM's replanning exceeds the response-time budget or yields an action plan the vehicle cannot execute, with no non-LLM fallback catching the error, then the claimed real-time adaptability is refuted.","tokens_in":8362,"feed_emoji":"🤖","tokens_out":8822,"duration_ms":83327,"temperature":0.7,"pith_summary":"The paper argues that the standard holonic way of building systems of systems can be upgraded by giving each holon a three-layer structure—reasoning, communication, and capabilities—and by placing a large language model at the centre of the reasoning layer. It introduces four specialised holons drawn from intelligent manufacturing—supervisor, planner, task, and resource—to coordinate heterogeneous, often black-box constituent systems at runtime. The authors claim this combination improves interoperability, reconfigurability, and human-system interaction in self-adaptive systems of systems, and they illustrate it with a 3D mobility scenario in which ground and aerial vehicles plan and execute multimodal trips for a passenger. A reader should care because modern systems of systems such as urban transport must absorb unknown systems at runtime and accept instructions from non-experts, and the paper offers a concrete architectural design for making that possible.","feed_headline":"Large language models become each holon's reasoning core","feed_subtitle":"The aim: seamless runtime coordination of ground and air vehicles in a smart city.","key_machinery":"The load-bearing object is the holon, defined as a semi-autonomous unit that is simultaneously a whole and a part of a larger whole. The machinery is the three-layer holon: a reasoning layer whose core is a large language model (command processing, context management, decision making), a communication layer that translates the LLM's action plans into executable commands through a robotics middleware and routes messages between holons, and a capabilities layer that wraps the constituent system's resources. Around this core sit four specialised holon roles—supervisor, planner, task, and resource—each given role-specific prompt context, with the human resource holon providing the natural-language interface for people. The same machinery also does dual-level planning: a supervisor-level plan is decomposed into task holons and sub-plans, so local changes can be handled without restarting the global mission.","core_discovery":"The paper's central claim is that specialised holons—supervisor, planner, task, and resource—use large language models within their reasoning layers to support decision-making and ensure real-time adaptability, and that the layered holon design lets each holon adapt autonomously to environmental changes while keeping behaviour coherent across the whole system of systems. The reasoning layer gives every holon the same three-part routine: preprocess an incoming command, build a context-aware prompt, and produce an action plan that the communication layer turns into concrete commands for the capabilities layer. This is presented as the first introduction of manufacturing-style specialised holons in the system-of-systems context, and the 3D mobility case study is the demonstration: a passenger's natural-language request is refined by a supervisor holon, split into driving and flying task holons, and executed by resource holons representing vehicles.","pith_inferences":["The decisive experiment is to stress the LLM reasoning core with injected failures and measure whether task holons detect and replan within the latency a moving vehicle tolerates; that would turn the architecture's central premise into evidence.","Because all four holon roles share the same LLM core, an error in one holon's context management could propagate as contradictory plans across the holarchy; adding a lightweight validation step for each generated action plan would test this.","The architecture's promise extends beyond transportation: any system of systems with black-box components and human operators—emergency response, logistics, manufacturing cells—could adopt the same specialised holon roles and natural-language interface."],"forward_implications":["Heterogeneous, black-box constituent systems can be brought into the system of systems at runtime by wrapping each in the same holon facade, since the LLM interprets commands and the communication layer standardises how capabilities are invoked.","A supervisor holon can produce a high-level plan while task holons locally replan sub-segments, so resource reallocation—for example, switching from vehicle $m_1$ to $m_3$ for a later leg—happens without rebuilding the whole mission.","Human operators and customers can interact through natural language via the human resource holon, lowering the technical expertise needed to monitor or redirect the system of systems.","The architecture avoids a single central controller, preserving the holonic paradigm's scalability while adding structure through specialised holon roles.","The proposed metrics—scalability, adaptability, resource utilisation, response time, and user satisfaction—give an intended comparison against holonic architectures without LLMs and specialised holons."],"supporting_citations":[{"why":"Defines the holon abstraction this paper places at the core of every constituent system.","marker":"[3]"},{"why":"Provides the ontological holonic SoS architecture that this paper extends and treats as the baseline.","marker":"[7]"},{"why":"Supplies the supervisor, planner, task, and resource holon types from intelligent manufacturing.","marker":"[13]"},{"why":"Offers the prior self-adaptive SoS architecture and the simulation approach this work points to for evaluation.","marker":"[23]"},{"why":"Shows how domain-specific ontologies can improve LLM prompt crafting for robotic systems, supporting the context-management design.","marker":"[12]"},{"why":"Establishes the transformer-based capabilities that the reasoning layer relies on.","marker":"[32]"},{"why":"The authors' earlier NLP-enhanced holonic architecture with a centralised module, which this paper decentralises.","marker":"[2]"}],"fun_headline_variants":["LLMs power decision-making in every holon's reasoning layer","Holons get a reasoning core: LLMs enable real-time adaptability","Smart city transport: LLM-driven holons coordinate ground and air","Specialized holons with LLM reasoning adapt systems of systems","From request to action: LLM holons orchestrate multimodal mobility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole design collapses unless a large language model can reliably reason in real time—returning correct action plans quickly enough for moving vehicles—and unless unknown third-party systems can be wrapped into the holon's three layers without their vendors' cooperation; both are stated but not yet shown by measurement.","fun_headline_variants_meta":{"raw":{"variants":["LLMs power decision-making in every holon's reasoning layer","Holons get a reasoning core: LLMs enable real-time adaptability","Smart city transport: LLM-driven holons coordinate ground and air","Specialized holons with LLM reasoning adapt systems of systems","From request to action: LLM holons orchestrate multimodal mobility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2345,"prompt_tokens":903,"completion_tokens":1442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1353}},"tokens_in":519,"tokens_out":1442,"duration_ms":10251,"temperature":1.0,"reasoning_tokens":1353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:30:57.664073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's own 3D mobility scenario and inject a road closure after a task holon has committed to a route; if the LLM's replanning exceeds the response-time budget or yields an action plan the vehicle cannot execute, with no non-LLM fallback catching the error, then the claimed real-time adaptability is refuted.","supporting_citations":[{"cited_title":"In: Proceedings of the 14th International Workshop on Adaptive and Reflective Middleware","cited_arxiv_id":null,"evidence_quote":"Defines the holon abstraction this paper places at the core of every constituent system."},{"cited_title":"Future Generation Computer Systems157, 499–515 (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the ontological holonic SoS architecture that this paper extends and treats as the baseline."},{"cited_title":"Computers in industry57(2), 121–130 (2006)","cited_arxiv_id":null,"evidence_quote":"Supplies the supervisor, planner, task, and resource holon types from intelligent manufacturing."},{"cited_title":"In: Proceedings of the 7th International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence","cited_arxiv_id":null,"evidence_quote":"Offers the prior self-adaptive SoS architecture and the simulation approach this work points to for evaluation."},{"cited_title":"Software: Practice and Experience (2024)","cited_arxiv_id":null,"evidence_quote":"Shows how domain-specific ontologies can improve LLM prompt crafting for robotic systems, supporting the context-management design."},{"cited_title":"In: Proceedings of the 19th International Conference on Software Technologies","cited_arxiv_id":null,"evidence_quote":"The authors' earlier NLP-enhanced holonic architecture with a centralised module, which this paper decentralises."}],"review_version":1}