Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applications

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a declarative JSON schema can specify multi-agent LLM workflows and that a prototype using OpenAI and DeepSeek models completes cybersecurity Q&A, server-security, and network-security tasks correctly.

desk verdict A genuinely concrete multi-agent LLM schema artifact, but the printed network-security evaluation in Listing 1 is internally inconsistent and must be fixed before the feasibility claim can be trusted. read the letter →

arxiv 2506.10467 v4 pith:HEYDLWTP submitted 2025-06-12 cs.CR cs.AI

Felix Härer This is my paper · ORCID
classification cs.CRcs.AI
keywords multi-agentLLMsystemsagentschemalanguageJSONspecificationcybersecuritytasksreasoningchainsevaluationprototypearchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports exploratory research on specifying multi-agent LLM systems declaratively rather than writing them as bespoke code. It introduces an agent schema language — a JSON structure that defines agent types, prompts, executable actions, data, and the criteria that decide whether a result is correct — and an execution architecture that runs those specifications. The central claim is that this combined approach is feasible for real tasks: test cases show question answering, server security, and network security tasks completed correctly by agents using OpenAI and DeepSeek LLMs. If true, this matters because the combination of specification and evaluation would allow systematic, application-level comparison of LLMs, reasoning techniques, and multi-agent coordination in a single framework.

What carries the argument

The central object is the agent schema language, a JSON-based specification format with four top-level parts: agent types (prompts, actions, and data), functions (execution functions and evaluation functions), agent configurations, and LLM configurations. The mechanism that carries the argument is the evaluation-function layer: at run-time the architecture pushes an agent and its prompt onto stacks, sends the prompt to an LLM, runs the prompt's actions on the host execution environment, then calls evaluation functions that match outputs against expected values, patterns, or result classes. Because task success is decided by the schema rather than by ad hoc human inspection, the same specification doubles as an evaluation instrument.

What would settle it

Run the same agent schema with the same models on a larger, independently fixed set of cybersecurity tasks whose expected values are set before the runs and whose hosts and networks were not chosen by the spec authors; if the completion rate collapses or the schema has to be rewritten for each environment, the feasibility claim falls. A cheaper check: change the target host in the network-security task from 10.11.1.24 to a different reachable host and observe whether evaluation still passes.

Watch

Extended reading notes

Core claim

The paper's discovery is that the whole working loop of a multi-agent LLM application can be captured in a schema: which agent types exist, what each agent prompts, which shell or Python actions run on the host, what data is passed between agents, and how each output is judged against expected values, patterns, or result classes. In the demonstrated cybersecurity test cases, all four evaluated models — GPT-4o, o1-preview, DeepSeek R1, and the smaller DeepSeek R1 Distilled — answered ten Q&A questions correctly; GPT-4o, o1-preview, and DeepSeek R1 completed all four server-security and all four network-security tasks, while the 8-billion-parameter distilled model completed all network tasks and two of the four server tasks. The paper reads these results as evidence that the architecture and evaluation approach are feasible.

Load-bearing premise

The load-bearing premise is that the hand-written evaluation criteria in the schema — the expected host address, the result-class patterns, and the choice of tasks themselves — are correct and impartial measures of task success; if those expectations were tailored to the tasks or the environment, the reported completions would not establish general feasibility.

Editorial extensions

If this is right

  • If the schema language works as claimed, multi-agent LLM applications become specifiable as data files, so task definitions and evaluation criteria can be versioned, shared, and reused across runs.
  • Because LLM and agent configurations are separate from agent types, comparing models or reasoning techniques reduces to swapping configurations rather than rewriting the task.
  • Automatic evaluation inside the schema is a step toward application-level benchmarks, where the same specification can test many models on the same tasks.
  • The demonstrated hand-off from a Server-Security-Agent to an Audit-Report-Agent shows that multi-step workflows with specialized agents and data passing are expressible and executable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The schema measures conformance to pre-specified expected values, so it evaluates specification conformance rather than open-ended task success; tasks without a known answer would still need a different judgment mechanism. (Editorial inference.)
  • Nothing in the architecture is cybersecurity-specific, so a natural next test is whether the same schema language handles other software tasks such as data analysis or code-repair workflows.
  • The 8-billion-parameter model's two server-security failures suggest the schema can double as a diagnostic for capability boundaries, revealing which steps in a task a given model cannot yet perform. (Editorial inference.)
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a JSON-based agent schema language for specifying multi-agent LLM systems, an execution architecture with a client application, and an evaluation approach. The prototype is demonstrated on cybersecurity test cases: 10 CyberMetric Q&A questions and 8 server/network security tasks executed by agents using OpenAI GPT-4o, o1-preview, DeepSeek R1, and DeepSeek R1-Distilled (8B). The central claim is feasibility: the architecture and specification can define multi-agent workflows that complete small cybersecurity tasks correctly with commercially and openly available LLMs.

Significance. If the demonstrated feasibility holds, the paper is a useful step toward systematic, application-level evaluation of multi-agent LLM systems. Its strengths are a concrete, executable schema for agent types, actions, and evaluation functions; a modular architecture with separable agent, conversation, and execution components; and a public repository with source code and full results. The Q&A evaluation uses the external CyberMetric ground truth, and the architecture explicitly supports reasoning chains, constraints, and data passing between agents. The contribution is exploratory rather than a controlled benchmark, but it gives the community a concrete specification format and a prototype that can be extended. However, the empirical evidence is thin: each task was run once, the tasks are hand-selected, and one printed evaluation criterion is internally inconsistent, so the feasibility claim is not yet solidly supported by the manuscript as written.

major comments (3)
  1. [Section IV-B, Listing 1, Table I] The network-security evaluation criterion is internally inconsistent with the task parameters. Listing 1 specifies `ipv4-network: "10.1.1.0/24"` for the Network-Security-Agent, but the same listing sets `expected-value: "10.11.1.24"`. The host 10.11.1.24 is not an address in 10.1.1.0/24; it lies in 10.11.1.0/24. Since Table I reports all four network-security rows (3.1–3.4) as correct based on matching this expected value, the correctness classification is not supported by the printed specification. If the intended value is 10.1.1.24, the paper must say so explicitly; if the actual execution scanned a different network, the manuscript does not describe the executed configuration. This is load-bearing because the abstract and conclusion claim correct completion of network security tasks, and the current text does not allow the reader to verify even a single successful execution.
  2. [Section IV, Table I] The evaluation methodology is not sufficient to support the general feasibility claim stated in the abstract and conclusion. Each agent/task combination was executed once, there is no indication of variance across runs, and the eight server/network tasks were hand-picked with no stated selection protocol or inter-rater validation of the expected values. A single successful run can demonstrate existence in principle, but only if the evaluation criterion is unambiguous and the task is defined precisely enough to be reproduced. The paper should either add repeated runs, report task-selection criteria, and provide explicit evaluation functions for every task, or explicitly restrict the claim to "the specific configurations reported here" rather than "feasibility of the architecture and evaluation approach".
  3. [Section IV-B, Listing 2 and Listing 1] The server-security task evaluations are not reproducible from the text. Listing 2 shows the Server-Security-Agent prompt only as "..." and gives no expected-value or result-class definitions for the four server security tasks, yet Table I reports correct/incorrect counts for all four agents. Similarly, the network-security prompt in Listing 1 does not specify which ports are considered "commonly exposed" or which results count as a match beyond the inconsistent expected value. The full evaluation criteria should be included in an appendix or supplement, so that a reader can determine what "correct" means for each of the eight tasks.
minor comments (4)
  1. [Listing 1 caption] The caption states that the Network Security Agent is "expected to find open ports at host 10.11.1.24," while the prompt asks for "reachable hosts with commonly exposed ports." These two phrasings are not equivalent and should be aligned.
  2. [Section IV-B, Figure 2] Figure 2 shows a Python script excerpt but does not state which LLM generated it, which prompt produced it, or which of the four network tasks it corresponds to. Adding this context would strengthen the demonstration.
  3. [Section II] The background section is quite long relative to the paper's contribution and includes general material on transformers and chain-of-thought that is not later used. It could be shortened to focus on benchmarks for cybersecurity and multi-agent specification.
  4. [Various] There are minor typographical issues, such as "open source and commercial LLMs" (should be "open-source") and inconsistent use of "explorative" versus "exploratory." A light copyedit would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the feasibility claim rests on external benchmarks and hand-defined execution checks, not on fitted parameters or self-citation.

full rationale

The paper's central claim is empirical: test cases indicate feasibility of the architecture and specification, demonstrated by Q&A, server-security, and network-security tasks completed correctly by agents using OpenAI and DeepSeek models. Correctness is judged by (a) external ground truth from the CyberMetric dataset for Q&A and (b) author-specified expected values and result-class patterns for execution tasks. No fitted parameter is renamed as a prediction, and no statistical model is calibrated on the outcomes it later 'predicts.' The schema language and architecture are presented as a prototype artifact, not derived from the measured results. The self-citations, [5] and [19], are used for provenance and related-work context (e.g., 'building on an LLM system from prior research' and a citation for constraint-based prompting); they do not supply the load-bearing argument for correctness, which is determined by external task outcomes. The internal inconsistency noted by the skeptic — Listing 1 gives ipv4-network '10.1.1.0/24' while expected-value is '10.11.1.24', a host outside that prefix — is a reproducibility and validity concern for the network-security rows, but it is not a circularity: the evaluation criterion is independent of the agent's output, just possibly mis-specified. Therefore the derivation chain does not reduce to its own inputs, and no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central feasibility claim rests on hand-authored test specifications and pattern-matching evaluation; no fitted parameters or new scientific entities are introduced. The schema language is a software artifact, not a postulated entity.

free parameters (2)
  • expected-value for network scan task = 10.11.1.24
    Hand-specified in Listing 1 as the host expected to have open ports; used as ground truth for evaluating the agent output.
  • result-class patterns in Q&A schema = 'ANSWER: A', 'ANSWER: B', etc.
    Hand-authored patterns in Listing 1 map LLM responses to correctness classes; they define the evaluation function.
assumptions (4)
  • domain assumption LLM responses can be reliably parsed and evaluated by pattern matching
    The evaluation functions in Section III-B and IV assume output syntax can be matched against defined patterns without error.
  • domain assumption The execution environment (Ubuntu 24.04, nmap, Python, shell) is available and safe for generated code
    Section IV-B requires agents to generate and execute arbitrary scripts in the Host Execution Environment.
  • domain assumption CyberMetric Q&A answers provide correct ground truth for the 10 questions
    Section IV-A uses the CyberMetric dataset [20] as the source of correct answers.
  • domain assumption Stack-based execution correctly realizes the specified agent orchestration semantics
    Section III-B defines the execution model; correctness of the prototype is assumed from the successful test runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applications." pith.science (2026). https://pith.science/paper/HEYDLWTP

@misc{pith2026250610467,
  author       = {Pith},
  title        = {Pith review of: Specification and Evaluation of Multi-Agent LLM Systems -- Prototype and Cybersecurity Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEYDLWTP}},
  note         = {Machine review of arXiv:2506.10467}
}
read the original abstract

Recent advancements in LLMs indicate potential for novel applications, as evidenced by the reasoning capabilities in the latest OpenAI and DeepSeek models. To apply these models to domain-specific applications beyond text generation, LLM-based multi-agent systems can be utilized to solve complex tasks, particularly by combining reasoning techniques, code generation, and software execution across multiple, potentially specialized LLMs. However, while many evaluations are performed on LLMs, reasoning techniques, and applications individually, their joint specification and combined application are not well understood. Defined specifications for multi-agent LLM systems are required to explore their potential and suitability for specific applications, allowing for systematic evaluations of LLMs, reasoning techniques, and related aspects. This paper reports the results of exploratory research on (1.) multi-agent specification by introducing an agent schema language and (2.) the execution and evaluation of the specifications through a multi-agent system architecture and prototype. The specification language, system architecture, and prototype are first presented in this work, building on an LLM system from prior research. Test cases involving cybersecurity tasks indicate the feasibility of the architecture and evaluation approach. As a result, evaluations could be demonstrated for question answering, server security, and network security tasks completed correctly by agents with LLMs from OpenAI and DeepSeek.

Figures

Figures reproduced from arXiv: 2506.10467 by the authors.

Figure 1
Figure 1. Architecture describing the components of the client application together with the specification in the agent schema. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Excerpt of a Python script generated for the network scanning task in Listing 1 within the user interface of the client [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Excerpt of a security report generated by an Audit Report Agent after the completion of server-side tasks by a [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [5]

    Conceptual model interpreter for Large Language Models,

    F. H ¨arer, “Conceptual model interpreter for Large Language Models,” in ER Forum 2023, 42nd International Conference on Conceptual Modeling (ER 2023) , vol. 3618. CEUR-WS, 2023

    work page 2023

  2. [1]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, and et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025, arXiv:2501.12948 [cs]

    arXiv 2025

  3. [2]

    Implications of new reasoning capabilities for science and security: Results from a quick initial study,

    J. A. Pruet, K. Duraisamy, V . Agrawal, A. Biswas, R. B. Bujack, M. J. Grosskopf, A. A. Hagberg, B. Hu, E. C. Lawrence, W. Li et al., “Implications of new reasoning capabilities for science and security: Results from a quick initial study,” Los Alamos National Laboratory (LANL), Los Alamos, NM, US, Tech. Rep., 2024

    work page 2024

  4. [3]

    2024 AIME II,

    AoPS Incorporated, “2024 AIME II,” 2024, https://artofproblemsolving.com/wiki/index.php/ 2024 AIME II, Retrieved 2025-02-23

    work page 2024

  5. [4]

    OpenAI Introduces o3,

    Y . Brett, “OpenAI Introduces o3,” 2024, In: Weights & Biases. https://wandb.ai/byyoung3/ml-news/reports/OpenAI-Introduces-o3- Pushing-the-Boundaries-of-AI-Reasoning–VmlldzoxMDY3OTUxMA, Retrieved 2025-02-23

    work page 2024

  6. [6]

    Multi-Agent Systems: A Survey About Its Components, Framework and Workflow,

    D. Maldonado, E. Cruz, J. Abad Torres, P. J. Cruz, and S. d. P. Gam- boa Benitez, “Multi-Agent Systems: A Survey About Its Components, Framework and Workflow,” IEEE Access , vol. 12, pp. 80 950–80 975, 2024

    work page 2024

  7. [7]

    A survey of the consensus for multi-agent systems,

    Y . Li and C. Tan, “A survey of the consensus for multi-agent systems,” Systems Science & Control Engineering , vol. 7, no. 1, pp. 468–482, 2019

    work page 2019

  8. [8]

    A Survey of LLM- based Agents: Theories, Technologies, Applications and Suggestions,

    X. Dong, X. Zhang, W. Bu, D. Zhang, and F. Cao, “A Survey of LLM- based Agents: Theories, Technologies, Applications and Suggestions,” in 2024 3rd International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) , 2024

    work page 2024

Show all 25 references
  1. [9]

    Large Language Model Based Multi-agents: A Survey of Progress and Challenges,

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large Language Model Based Multi-agents: A Survey of Progress and Challenges,” in Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024 , vol. 9, 2024, pp. 8048–8057

    2024

  2. [10]

    Large language models (LLMs): survey, technical frame- works, and future challenges,

    P. Kumar, “Large language models (LLMs): survey, technical frame- works, and future challenges,” Artificial Intelligence Review , vol. 57, no. 10, p. 260, 2024

    2024

  3. [11]

    Chatgpt: A comprehensive review on background, applica- tions, key challenges, bias, ethics, limitations and future scope,

    P. P. Ray, “Chatgpt: A comprehensive review on background, applica- tions, key challenges, bias, ethics, limitations and future scope,” Internet of Things and Cyber-Physical Systems , vol. 3, pp. 121–154, 2023

    2023

  4. [12]

    Rothman, Transformers for Natural Language Processing

    D. Rothman, Transformers for Natural Language Processing. Second Edition. Birmingham, UK: Packt Publishing, O’Reilly Media, 2022

    2022

  5. [13]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17, Red Hook, NY , USA, 2017

    2017

  6. [14]

    Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Lan- guage Models,

    L. Luo, Z. Zhao, C. Gong, G. Haffari, and S. Pan, “Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Lan- guage Models,” 2024, arXiv:2410.13080 [cs]

    2024 arXiv

  7. [15]

    ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness,

    A. Prasad, S. Saha, X. Zhou, and M. Bansal, “ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023, pp. 10...

    2023

  8. [16]

    Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Models,

    Z. Lin, C. Chan, Y . Song, and X. Liu, “Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Models,” in PRICAI 2024: Trends in Artificial Intelligence . Singapore: Springer Nature, 2024, pp. 354–360

    2024

  9. [17]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023

    2023

  10. [18]

    Evaluation of retrieval-augmented generation: A survey,

    H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of retrieval-augmented generation: A survey,” in Big Data. Singapore: Springer Nature Singapore, 2025, pp. 102–120

    2025

  11. [19]

    CMAG: A Framework for Conceptual Model Augmented Generative Artificial Intelligence,

    H.-G. Fill, F. H ¨arer, I. Vasic, D. Borcard, B. Reitemeyer, F. Muff, S. Curty, and M. B ¨uhlmann, “CMAG: A Framework for Conceptual Model Augmented Generative Artificial Intelligence,” in ER Forum 2024, 43rd International Conference on Conceptual Modeling (ER 2024), vol. 3849...

    2024

  12. [20]

    CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge,

    N. Tihanyi, M. A. Ferrag, R. Jain, T. Bisztray, and M. Debbah, “CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge,” in 2024 IEEE International Conference on Cyber Security and Resilience (CSR) , London, UK, ...

    2024

  13. [21]

    SECURE: Benchmarking Large Language Models for Cybersecurity,

    D. Bhusal, M. T. Alam, L. Nguyen, A. Mahara, Z. Lightcap, R. Frazier, R. Fieblinger, G. L. Torales, B. A. Blakely, and N. Rastogi, “SECURE: Benchmarking Large Language Models for Cybersecurity,” in Proceed- ings of the 40th Annual Computer Security Applications Conference , se...

    2024

  14. [22]

    CyberBench: A Multi-Task Benchmark for Evaluating Large Language Models in Cybersecurity,

    Z. Liu, J. Shi, and J. F. Buford, “CyberBench: A Multi-Task Benchmark for Evaluating Large Language Models in Cybersecurity,” in The AAAI- 24 Workshop on Artificial Intelligence for Cyber Security (AICS) , Vancouver, CA, 2024

    2024

  15. [23]

    CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence,

    M. T. Alam, D. Bhusal, L. Nguyen, and N. Rastogi, “CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence,” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, CA, 2024

    2024

  16. [24]

    When LLMs meet cybersecurity: a systematic literature review,

    J. Zhang, H. Bu, H. Wen, Y . Liu, H. Fei, R. Xi, L. Li, Y . Yang, H. Zhu, and D. Meng, “When LLMs meet cybersecurity: a systematic literature review,” Cybersecurity, vol. 8, no. 1, p. 55, 2025

    2025

  17. [25]

    Generative AI for Cyber Security: Analyzing the Potential of ChatGPT, DALL-E, and Other Models for Enhancing the Security Space,

    S. Sai, U. Yashvardhan, V . Chamola, and B. Sikdar, “Generative AI for Cyber Security: Analyzing the Potential of ChatGPT, DALL-E, and Other Models for Enhancing the Security Space,” IEEE Access, vol. 12, pp. 53 497–53 516, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.