Pith. sign in

REVIEW 3 major objections 5 minor 79 references

The paper claims that a self-hosted computer-use agent can safely guide a UAV only if its outputs are compiled into contract-bound skill invocations that the onboard runtime checks for timing, state, and authority before dispatch.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 16:31 UTC pith:HC4WLATA

load-bearing objection Solid architecture with a real evaluation, but the zero-deadline-miss claim doesn't square with the reported latencies, and the security boundary rests on an unverified assumption. the 3 major comments →

arxiv 2607.17951 v1 pith:HC4WLATA submitted 2026-07-20 cs.CR cs.AIcs.ROcs.SYeess.SY

RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control

classification cs.CR cs.AIcs.ROcs.SYeess.SY
keywords SHCUAUAV controlnatural-language controlcontract-bound skill invocationreal-time admission controlruntime enforcementTrustZonedegraded execution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that letting an LLM-based computer-use agent directly issue flight commands is structurally unsafe, because the agent's slow, serial reasoning loop mismatches the fast, state-dependent, physically consequential nature of UAV control. Instead, the authors propose restructuring the agent into a task-level proposer whose outputs become contract-bound UAV skill invocations carrying explicit timing, state, authority, fallback, and evidence fields. An onboard runtime then admits and dispatches only those invocations that are still temporally valid, consistent with the current vehicle state, authorized, and within deadline slack. The paper demonstrates this with a prototype on PX4/Gazebo and a TrustZone-enforced admission path, reporting zero deadline misses across 1,890 normal executions and matching security outcomes across 7,560 degraded and attack checks. If correct, this makes natural-language UAV control feasible without placing the large model or the full flight-control loop in the safety-critical path.

Core claim

The central claim is that an SHCUA should not be a direct controller of a UAV, but a constrained task-level proposer. The authors formalize this with a contract-bound skill invocation object I = ⟨sid, src, op, args, pre, ddl, vw, sc, auth, inv, fb, ev⟩ and an admissibility predicate admissible(I, St, t) = valid(I, St, t) ∧ (L_rem(I) ≤ ddl(I) − t). An invocation is dispatched only if it is temporally valid, state-consistent, and has enough remaining slack before its deadline. Security enforcement is modeled as η(I, St, P) → {allow, deny, defer, fallback}, with a collision-resistant binding B tying operation, parameters, timing, authority, and policy version. Under three stated assumptions — m

What carries the argument

The contract-bound UAV skill invocation is the central object: a structured execution record that attaches timing metadata (generation, receive, deadline, validity window, timing class), state constraints, authority scope, safety/security invariants, fallback behavior, and evidence requirements to each agent-generated action. It works by converting the SHCUA's free-form tool calls into bounded vehicle operations that the onboard runtime can check before dispatch. The corresponding admissibility condition balances slow offboard reasoning against fast onboard execution, and the timing-class distinction (semantic, tactical, reflex) routes decisions to the appropriate layer so reflex-level safet

Load-bearing premise

The entire security argument collapses if an SHCUA-generated action can reach the UAV skill executor or flight controller without first being compiled into an invocation and passing the onboard admission and enforcement checks; the paper asserts this mediated-dispatch confinement but does not demonstrate that the prototype physically or logically prevents the agent from holding a direct command channel.

What would settle it

In the current prototype, list all tools available to the SHCUA and attempt to send a raw MAVLink command or shell instruction directly to the PX4/Gazebo backend, bypassing the mission compiler and admission path; if the vehicle executes it (or if the agent's tool registry contains such a channel), the mediated-dispatch assumption is false. Separately, manipulate the vehicle-state snapshot or clock to deliver an expired invocation while the validity check passes, showing the state snapshot is trusted but not authenticated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Natural-language UAV command becomes deployable without making the LLM part of the real-time control loop, since slow reasoning is decoupled from onboard dispatch.
  • Expired, stale, or state-inconsistent agent decisions are rejected before reaching the flight controller, preventing a class of unsafe commands.
  • Unauthorized, tampered, or replayed invocations are detected at the admission boundary, giving the system a security property rather than just a prompt-level guardrail.
  • Every allowed, denied, deferred, or fallback-mediated invocation produces an evidence-chain record, enabling post-flight audit and traceability of agent-influenced vehicle behavior.
  • The trusted enforcement boundary can be isolated in a TEE or microcontroller without relocating the full agent or high-frequency flight-control loop, keeping the protected computing base small.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The contract-mediation pattern likely generalizes to other LLM-controlled physical systems — ground robots, marine vehicles, or industrial actuators — wherever a slow semantic reasoner proposes actions for a fast, stateful plant; the same admissibility-slack discipline would apply.
  • The security theorems rest on Assumption 1 being true in deployment: the prototype asserts that the agent has no direct channel to the skill executor, but a real deployment would need to prove the agent's tool surface is physically confined (e.g., no shell or MAVLink tool) — a testable extension is to audit the agent's registered tools and show only the RT-SHCUA adapter exists.
  • The evidence chain's hash-linked records provide a natural foundation for incident forensics and regulatory compliance; one could extend it with attestation of the state snapshot St so that the recorded state cannot be a fabricated input to the enforcement decision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RT-SHCUA, an architecture that restructures self-hosted computer-use agent (SHCUA) control of UAVs. Instead of letting an LLM agent issue flight commands directly, the agent's outputs are transformed into contract-bound skill invocations carrying timing, state, authority, fallback, and evidence semantics. An onboard runtime validates admissibility, security, and state constraints before dispatching bounded UAV skills. The paper presents a formal model (validity, admissibility, authorization, evidence) with theorems, a prototype built on OpenClaw, PX4 SITL/Gazebo, and QEMU-based OP-TEE, and an evaluation with 1890 normal executions and 7560 security/degraded checks. The central claim is that SHCUA reasoning can be usefully decoupled from time-critical vehicle execution while maintaining bounded task-level responsiveness, security enforcement, and auditability.

Significance. If the architecture is realized as described, it addresses a real and timely problem: safely mediating open-ended LLM-agent tools in a physical, real-time cyber-physical system. The contract-bound skill invocation model is a sensible separator between semantic reasoning and onboard enforcement, and the workload split between semantic, tactical, and reflex classes is well motivated. The prototype is concrete and the overhead measurements for the OP-TEE path, if interpreted carefully, give useful engineering data. The paper's formal theorems, however, are definitional consequences of the assumptions and do not by themselves provide evidence about the physical or security properties of the implementation; the value of the paper rests on the architecture and prototype evaluation, not on the formal apparatus. The evaluation currently does not substantiate the headline real-time and security claims, which is why major revision is needed.

major comments (3)
  1. [Section VII-B, Fig. 4, Table I] The paper reports OP-TEE P95 end-to-end latencies of 231.6 ms (short), 265.9 ms (standard), and 308.0 ms (extended), while the deadline matrix includes 0.1 s and 0.2 s cells. The claims of 'zero task-level deadline-miss flags and zero fallback events across all 1890 executions' are therefore internally inconsistent with the reported latency distribution unless the measured 'end-to-end' interval is not the interval to which the 'task-level deadline' applies, or the 0.1/0.2 s deadlines were not actually enforced. The paper must specify the exact start and end of the measured latency, the precise definition of 'task-level deadline' relative to that interval, and report per-cell miss counts and deadline-check results. Without this, G1 (real-time responsiveness) is not demonstrated.
  2. [Section IV-E, Assumption 1; Section VI-A] Assumption 1 states that the SHCUA runtime has no direct channel to the UAV skill executor or flight controller, and Theorems 1-3 depend on it. The prototype description in Section VI-A introduces an 'Agent-Side Decision Mediation' adapter but gives no mechanism that prevents OpenClaw from opening a MAVLink socket, writing to a PX4 serial device, or invoking a shell command that reaches the flight controller. No tool allowlist, OS-level confinement (seccomp/AppArmor/container/namespace), or network filter is described. Since OpenClaw is a general computer-use agent, the asserted boundary may not exist in the implementation, and the security theorems would then not apply to the prototype. The paper should either demonstrate the confinement mechanism or explicitly state the assumed trust boundary and its validation.
  3. [Section IV-E, Theorems 1-3] The formal properties are derived essentially by unpacking the definitions of admissible, binding, evidence, and the stated Assumptions 1-4. This makes them true by construction rather than substantive guarantees about the implementation. The paper should state explicitly that the theorems are conditional on the enforcement assumptions and that the evaluation does not validate the assumptions themselves (e.g., Assumption 1). The empirical evaluation stands independently, but the 'formal properties' section as written gives an impression of verified security that is not supported.
minor comments (5)
  1. [Section IV-C, Eq. (1)-(3)] The admissibility condition uses L_rem(I) but the prototype section does not define how L_src, L_comp, L_val, and L_disp are measured. The relationship between L_rem and the end-to-end latency in Fig. 4 should be stated.
  2. [Section VI-E, Fig. 3] The figure labels 'Key (HMAC / Sign Keys)' and 'Policies & Policy Metadata' inside the TrustZone TA; it would be clearer to note whether these are provisioned off-line and how they are updated, since policy versioning appears in the binding.
  3. [Section VII-D, Fig. 7] The OP-TEE overhead (roughly 170-175 ms) is from a QEMU-based emulation environment. This should be reported as emulated-environment cost, not a hardware TrustZone cost, to avoid overgeneralization.
  4. [Section VII-B] Fig. 4 and Fig. 5 show cell-averaged medians/P95s but no variance or per-cell values. Given the inconsistency between P95 and the 0.1/0.2 s deadlines, per-cell latency distributions and miss counts are essential.
  5. [Section VIII, related work] The related-work survey is thorough, but Table IV uses 'Partial' for many rows without a precise scoring criterion. A short explanation of the scoring would help.

Circularity Check

2 steps flagged

Formal Theorems 1–3 reduce to Assumptions 2/4 plus the definitions of admissible, B, and ev; the central security guarantees are true by construction rather than independently derived.

specific steps
  1. self definitional [Section IV-C (admissibility condition) and Section IV-E (Theorem 1 and proof)]
    "An invocation is admissible for execution only if it is temporally valid, state-consistent, and still has enough remaining slack before its deadline: admissible(I, St, t) = valid(I, St, t) ∧ (Lrem(I) ≤ ddl(I)−t). Theorem 1: Real-time admission control. Under Assumptions 1–3, an SHCUA-generated invocation that is expired, state-inconsistent, or lacks sufficient remaining execution slack cannot be dispatched to the UAV skill executor. Proof. ... By Assumption 2, the runtime dispatches I only if the admissibility check succeeds."

    Theorem 1's conclusion is precisely the negation of the admissibility predicate combined with Assumption 2's 'dispatch only if the admissibility check succeeds'. The proof cites the definition and Assumption 2, so the result is true by construction. Assumption 1 also restricts the theorem to invocations that actually enter the runtime, excluding bypasses by assumption rather than ruling them out. The theorem therefore adds no content beyond the model's own definitions.

  2. self definitional [Section IV-D (binding/evidence definitions) and Section IV-E (Theorems 2 and 3)]
    "Theorem 2: Security admission control. Under Assumptions 1, 2, and 4, an unauthorized, tampered, expired, or replayed SHCUA-generated invocation cannot be dispatched as a valid UAV action without detection. Proof. ... By Assumption 2, dispatch requires authorization, freshness, and command-binding checks to succeed. ... By Assumption 4, these checks cannot be bypassed by the ordinary SHCUA runtime."

    The theorem's guarantees are restatements of Assumption 2 (dispatch requires authorization, freshness, and binding checks), the binding definition B=H(sid, op, args, auth, ddl, vw, Pver), and Assumption 4 (non-bypassability). Theorem 3 similarly follows from the ev tuple definition and Assumption 4's non-suppression clause. These are analytic consequences of the enforcement model, not independent derivations: if the checks exist and cannot be bypassed, then unauthorized/tampered/replayed invocations are detected by definition; if the evidence tuple is generated in a protected path, evidence exists by definition.

full rationale

The paper's genuine contribution is architectural and empirical: it builds a prototype with OpenClaw, PX4 SITL/Gazebo, and OP-TEE, and it measures latencies, fallback behavior, trusted-path consistency, and evidence costs. Those measurements are independent of the formal model and are not circular. The circularity lies in presenting Theorems 1–3 as formal properties of the system: each theorem is a restatement of Assumptions 2/4 and the predicates/objects defined in Section IV. Theorem 1's 'expired/state-inconsistent/insufficient-slack cannot be dispatched' is the negation of admissible(I,St,t) plus Assumption 2; Theorem 2's 'unauthorized/tampered/expired/replayed cannot be dispatched without detection' is Assumption 2 plus B=H(·) plus Assumption 4; Theorem 3 follows from ev plus Assumption 4. These are specification-consistency checks, not results that would falsify a design. In addition, Assumption 1—the claim that the SHCUA has no direct channel to the skill executor or flight controller—is asserted but not demonstrated in the prototype; the theorems' scope therefore excludes the most important bypass scenario by assumption. That is partly a correctness risk rather than pure circularity, and it raises the score. Self-citations [47] and [48] are contextual prior work and are not load-bearing for the derivation, so they do not contribute to the circularity score. Overall, the empirical evaluation is substantive and independent, but the headline security theorems reduce by construction to the model's own assumptions and definitions, giving a score of 5.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim rests on enforcement assumptions that are asserted, not verified in the prototype: no direct agent channel, pre-dispatch enforcement, protected enforcement path, and trusted state/policy. The formal properties are immediate consequences of these assumptions. The evaluation adds empirical evidence for the prototype's behavior but does not validate the assumptions on real hardware or against a true no-mediation baseline.

axioms (6)
  • domain assumption Assumption 1 (Mediated dispatch): Every SHCUA-generated UAV action must be compiled into a skill invocation I and submitted to the onboard runtime; the SHCUA runtime has no direct channel to the UAV skill executor or the flight controller.
    Load-bearing for all security properties. If the agent can reach PX4/MAVLink or the shell directly, the admission/enforcement boundary is bypassable. The prototype does not show how OpenClaw's tool access is confined.
  • domain assumption Assumption 2 (Pre-dispatch enforcement): The onboard runtime enforces admissibility, authorization, freshness, and command-binding checks before dispatch, and dispatches only if these succeed.
    Theorems 1-2 are true by this assumption; it is a design contract, not a guarantee verified by the prototype.
  • domain assumption Assumption 4 (Protected enforcement path): Security-critical checks and evidence generation are performed in a trusted/isolated path that cannot be bypassed by the ordinary SHCUA runtime.
    The security theorem requires that the TEE/isolation mechanism is sound. The prototype uses QEMU-emulated OP-TEE, not hardware TrustZone, so this assumption is not validated on real hardware.
  • domain assumption State snapshot St and policy P are authentic and current.
    valid(I, St, t) and eta(I, St, P) depend entirely on St and P. A spoofed state or stale policy invalidates the checks. The paper does not identify a trusted state source.
  • standard math H(.) is collision-resistant or authenticated.
    The binding B=H(sid, op, args, auth, ddl, vw, Pver) security claim requires collision resistance; standard cryptographic assumption, not proved in the paper.
  • domain assumption The UAV platform exposes a closed skill library with complete contract schemas (preconditions, parameter ranges, fallback behavior) for every operation the SHCUA may propose.
    The model assumes any agent proposal can be compiled into a bounded skill. If a request cannot be mapped, the system falls back; safety then depends on the fallback design and the completeness of the skill library.

pith-pipeline@v1.3.0-alltime-deepseek · 28636 in / 15987 out tokens · 138374 ms · 2026-08-01T16:31:35.869915+00:00 · methodology

0 comments
read the original abstract

Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed agent iterations are often acceptable. UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic reasoning from onboard execution and security/safety enforcement. Slow cloud or edge reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated UAV actions.

Figures

Figures reproduced from arXiv: 2607.17951 by Bo Zhang, Di Lu, Jianfeng Ma, Xiyuan Li, Xuewen Dong, Yongzhi Liao, Yulong Shen, Zhiquan Liu.

Figure 1
Figure 1. Figure 1: Overall architecture of the proposed RT-SHCUA framework for UAV control. The architecture separates offboard semantic reasoning from onboard [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Prototype implementation architecture of RT-SHCUA. OpenClaw provides the SHCUA-side agent execution model, PX4 SITL with Gazebo provides [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ARM TrustZone-based implementation of the trusted enforcement [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Normal matrix P95 end-to-end latency under three enforcement modes, three mission lengths, and 0.1–1.5 s task-level deadlines. The matrix reports [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Security enforcement overhead of the OP-TEE-backed path. Bars [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mission-scaling latency results. The left panel shows median end-to [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Degraded and security-triggering decision outcomes. The OP-TEE [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 63 linked inside Pith

  1. [1]

    Taking flight with dialogue: Enabling natural language control for PX4-based drone agent,

    S. K. Lim, M. J. Y . Chong, J. H. Khor, and T. Y . Ling, “Taking flight with dialogue: Enabling natural language control for PX4-based drone agent,”arXiv preprint arXiv:2506.07509, 2025. [Online]. Available: https://arxiv.org/abs/2506.07509

  2. [2]

    Next-generation LLM for UA V: From natural language to autonomous flight,

    L. Yuan, C. Deng, D.-J. Han, I. Hwang, S. Brunswicker, and C. G. Brinton, “Next-generation LLM for UA V: From natural language to autonomous flight,”arXiv preprint arXiv:2510.21739, 2025. [Online]. Available: https://arxiv.org/abs/2510.21739

  3. [3]

    A prompt-driven task planning method for multi-drones based on large language model,

    Y . Liu, “A prompt-driven task planning method for multi-drones based on large language model,”arXiv preprint arXiv:2406.00006, 2024. [Online]. Available: https://arxiv.org/abs/2406.00006

  4. [4]

    A universal large language model–drone command and control interface,

    J. N. Ramos-Silva and P. J. Burke, “A universal large language model–drone command and control interface,” arXiv preprint arXiv:2601.15486, 2026. [Online]. Available: https://arxiv.org/abs/2601.15486

  5. [5]

    Large language model-assisted UA V operations and communications: A multifaceted survey and tutorial,

    Y . Emami, H. Zhou, R. Reddy, A. H. Arani, B. Wang, K. Li, L. Almeida, and Z. Han, “Large language model-assisted UA V operations and communications: A multifaceted survey and tutorial,” arXiv preprint arXiv:2602.19534, 2026. [Online]. Available: https: //arxiv.org/abs/2602.19534 IEEE JOURNAL TEMPLATE, VOL. XX, NO. XX, MONTH 2026 18

  6. [6]

    UA Vs meet LLMs: Overviews and perspectives toward agentic low-altitude mobility,

    Y . Tian, F. Lin, Y . Li, T. Zhang, Q. Zhang, X. Fu, J. Huang, X. Dai, Y . Wang, C. Tian, B. Li, Y . Lv, L. Kov ´acs, and F.-Y . Wang, “UA Vs meet LLMs: Overviews and perspectives toward agentic low-altitude mobility,”Information Fusion, vol. 122, p. 103158, 2025. [Online]. Available: https://arxiv.org/abs/2501.02341

  7. [7]

    When large language models meet UA Vs: How far are we?

    Y . Chen, X. Que, J. Zhang, T. Chen, G. Li, and J. Chen, “When large language models meet UA Vs: How far are we?” arXiv preprint arXiv:2509.12795, 2025. [Online]. Available: https: //arxiv.org/abs/2509.12795

  8. [8]

    TypeFly: Flying drones with large language model,

    G. Chen, X. Yu, N. Ling, and L. Zhong, “TypeFly: Flying drones with large language model,”arXiv preprint arXiv:2312.14950, 2023. [Online]. Available: https://arxiv.org/abs/2312.14950

  9. [9]

    AerialClaw: An open-source framework for LLM-driven autonomous aerial agents,

    K. Li, J. Yang, L. Zhang, G. Yu, C. Yan, Y . Ding, D. Wang, N. Luo, G. Liu, X. Gao, and Q. Wang, “AerialClaw: An open-source framework for LLM-driven autonomous aerial agents,” arXiv preprint arXiv:2606.12142, 2026. [Online]. Available: https: //arxiv.org/abs/2606.12142

  10. [10]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausmanet al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022. [Online]. Available: https: //arxiv.org/abs/2204.01691

  11. [12]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,”arXiv preprint arXiv:2209.07753, 2022. [Online]. Available: https://arxiv.org/abs/2209.07753

  12. [14]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotaret al., “Inner monologue: Embodied reasoning through planning with language models,”arXiv preprint arXiv:2207.05608, 2022. [Online]. Available: https://arxiv.org/abs/2207.05608

  13. [15]

    Available: https://arxiv.org/abs/2209.11302

    [Online]. Available: https://arxiv.org/abs/2209.11302

  14. [16]

    PaLM-E: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, J. Tompson, Q. Vuong, T. Yuet al., “PaLM-E: An embodied multimodal language model,”arXiv preprint arXiv:2303.03378, 2023. [Online]. Available: https://arxiv.org/abs/2303.03378

  15. [17]

    ChatGPT for robotics: Design principles and model abilities,

    S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,”arXiv preprint arXiv:2306.17582, 2023. [Online]. Available: https://arxiv.org/abs/2306. 17582

  16. [19]

    RT-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finnet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,”arXiv preprint arXiv:2307.15818, 2023. [Online]. Available: https://arxiv.org/abs/2307.15818

  17. [20]

    V oxPoser: Composable 3D value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxPoser: Composable 3D value maps for robotic manipulation with language models,”arXiv preprint arXiv:2307.05973, 2023. [Online]. Available: https://arxiv.org/abs/2307.05973

  18. [21]

    Manipulate-anything: Automating real-world robots using vision-language models,

    J. Duan, W. Yuan, W. Pumacay, Y . R. Wang, K. Ehsani, D. Fox, and R. Krishna, “Manipulate-anything: Automating real-world robots using vision-language models,”arXiv preprint arXiv:2406.18915, 2024. [Online]. Available: https://arxiv.org/abs/2406.18915

  19. [22]

    EmbodiedGPT: Vision-language pre-training via embodied chain of thought,

    Y . Mu, Q. Zhang, M. Hu, M. Ding, W. Wang, J. Jin, B. Wang, Y . Qiao, and P. Luo, “EmbodiedGPT: Vision-language pre-training via embodied chain of thought,”arXiv preprint arXiv:2305.15021, 2023. [Online]. Available: https://arxiv.org/abs/2305.15021

  20. [23]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”arXiv preprint arXiv:2308.11432, 2023. [Online]. Available: https://arxiv.org/abs/2308.11432

  21. [24]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”arXiv preprint arXiv:2309.07864, 2023. [Online]. Available: https://arxiv.org/abs/2309.07864

  22. [25]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023. [Online]. Available: https://arxiv.org/abs/2305.16291

  23. [26]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” arXiv preprint arXiv:2302.04761, 2023. [Online]. Available: https://arxiv.org/abs/2302.04761

  24. [27]

    Gorilla: Large language model connected with massive apis,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,”arXiv preprint arXiv:2305.15334, 2023. [Online]. Available: https://arxiv.org/abs/2305. 15334

  25. [28]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022. [Online]. Available: https://arxiv.org/abs/2210.03629

  26. [29]

    TaskMatrix.AI: Completing tasks by connecting foundation models with millions of apis,

    Y . Liang, C. Wu, T. Song, W. Wu, Y . Xia, Y . Liu, Y . Ou, S. Lu, L. Ji, S. Maoet al., “TaskMatrix.AI: Completing tasks by connecting foundation models with millions of apis,”arXiv preprint arXiv:2303.16434, 2023. [Online]. Available: https://arxiv.org/abs/2303. 16434

  27. [30]

    HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,”arXiv preprint arXiv:2303.17580, 2023. [Online]. Available: https://arxiv.org/abs/2303.17580

  28. [32]

    PlanBench-XL: Evaluating long-horizon planning of LLM tool-use agents in large-scale tool ecosystems,

    J. Liu, Q. Lin, C. Qian, R. Wang, E. C. Acikgoz, X. Yang, J. Liu, Z. Wang, X. Chen, H. Ji, and D. Hakkani-Tur, “PlanBench-XL: Evaluating long-horizon planning of LLM tool-use agents in large-scale tool ecosystems,”arXiv preprint arXiv:2606.22388, 2026. [Online]. Available: https://arxiv.org/abs/2606.22388

  29. [33]

    Tool learning with large language models: A survey,

    C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, “Tool learning with large language models: A survey,”arXiv preprint arXiv:2405.17935, 2024. [Online]. Available: https://arxiv.org/abs/2405.17935

  30. [34]

    Mind2Web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2Web: Towards a generalist agent for the web,”arXiv preprint arXiv:2306.06070, 2023. [Online]. Available: https://arxiv.org/abs/2306.06070

  31. [35]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [Online]. Available: https: //arxiv.org/abs/2406.12045

  32. [36]

    VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,

    J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. C. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,”arXiv preprint arXiv:2401.13649, 2024. [Online]. Available: https://arxiv.org/abs/2401.13649

  33. [37]

    WebV oyager: Building an end-to-end web agent with large multimodal models,

    H. He, W. Yao, K. Ma, W. Yu, Y . Dai, H. Zhang, Z. Lan, and D. Yu, “WebV oyager: Building an end-to-end web agent with large multimodal models,”arXiv preprint arXiv:2401.13919, 2024. [Online]. Available: https://arxiv.org/abs/2401.13919

  34. [39]

    WebArena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “WebArena: A realistic web environment for building autonomous agents,”arXiv preprint arXiv:2307.13854, 2023. [Online]. Available: https://arxiv.org/abs/2307.13854

  35. [40]

    AndroidWorld: A dynamic benchmarking environment for autonomous agents,

    C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva, “AndroidWorld: A dynamic benchmarking environment for autonomous agents,” arXiv preprint arXiv:2405.14573, 2024. [Online]. Available: https: //arxiv.org/abs/2405.14573 IEEE JOURNAL ...

  36. [41]

    Windows agent arena: Evaluating multi-modal OS agents at scale,

    R. Bonatti, D. Zhao, F. Bonacci, D. Dupont, S. Abdali, Y . Li, Y . Lu, J. Wagle, K. Koishida, A. Buckeret al., “Windows agent arena: Evaluating multi-modal OS agents at scale,”arXiv preprint arXiv:2409.08264, 2024. [Online]. Available: https://arxiv.org/abs/2409. 08264

  37. [42]

    WorkArena: How capable are web agents at solving common knowledge work tasks?

    A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquezet al., “WorkArena: How capable are web agents at solving common knowledge work tasks?”arXiv preprint arXiv:2403.07718, 2024. [Online]. Available: https://arxiv.org/abs/2403.07718

  38. [43]

    Available: https://arxiv.org/abs/2412.05467

    [Online]. Available: https://arxiv.org/abs/2412.05467

  39. [44]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Leiet al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” arXiv preprint arXiv:2404.07972, 2024. [Online]. Available: https: //arxiv.org/abs/2404.07972

  40. [45]

    OS agents: A survey on MLLM- based agents for general computing devices use,

    X. Hu, T. Xiong, B. Yiet al., “OS agents: A survey on MLLM- based agents for general computing devices use,”arXiv preprint arXiv:2508.04482, 2025. [Online]. Available: https://arxiv.org/abs/2508. 04482

  41. [46]

    A survey on the safety and security threats of computer-using agents: JARVIS or Ultron?

    A. Chen, Y . Wu, J. Zhang, S. Yang, J.-t. Huang, K. Wang, W. Wang, and S. Wang, “A survey on the safety and security threats of computer-using agents: JARVIS or Ultron?”arXiv preprint arXiv:2505.10924, 2025. [Online]. Available: https://arxiv.org/abs/2505.10924

  42. [47]

    When convenience becomes risk: A semantic view of under-specification in host-acting agents,

    D. Lu, Y . Liao, X. Mu, L. Zheng, K. Cheng, X. Dong, Y . Shen, and J. Ma, “When convenience becomes risk: A semantic view of under-specification in host-acting agents,”Journal of Networking and Network Applications, vol. 6, no. 2, pp. 47–54, 2026. [Online]. Available: https://iecscience.org/jpapers/224

  43. [49]

    Available: https://arxiv.org/abs/2407.05291

    [Online]. Available: https://arxiv.org/abs/2407.05291

  44. [50]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,

    J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,”arXiv preprint arXiv:2401.16158, 2024. [Online]. Available: https://arxiv.org/abs/2401.16158

  45. [51]

    Run time assurance for safety-critical systems: An introduction to safety filtering approaches for complex control systems,

    K. Hobbs, M. Mote, M. Abate, S. Coogan, and E. Feron, “Run time assurance for safety-critical systems: An introduction to safety filtering approaches for complex control systems,”arXiv preprint arXiv:2110.03506, 2021. [Online]. Available: https://arxiv.org/abs/2110. 03506

  46. [52]

    SafeDrones: Real-time reliability evaluation of UA Vs using executable digital dependable identities,

    K. Aslansefat, P. Nikolaou, M. Walker, M. N. Akram, I. Sorokos, J. Reich, P. Kolios, M. K. Michael, T. Theocharides, G. Ellinaset al., “SafeDrones: Real-time reliability evaluation of UA Vs using executable digital dependable identities,”arXiv preprint arXiv:2207.05643, 2022. [Online]. Available: https://arxiv.org/abs/2207.05643

  47. [55]

    Not what you’ve signed up for: Compromising real- world LLM-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world LLM-integrated applications with indirect prompt injection,” arXiv preprint arXiv:2302.12173, 2023. [Online]. Available: https: //arxiv.org/abs/2302.12173

  48. [56]

    SwarmGPT: Combining large language models with safe motion planning for drone swarm choreography,

    M. Schuck, D. O. Dahanaggamaarachchi, B. Sprenger, V . Vyas, S. Zhou, and A. P. Schoellig, “SwarmGPT: Combining large language models with safe motion planning for drone swarm choreography,”IEEE Robotics and Automation Letters, vol. 10, no. 11, pp. 12 237–12 244, 2025

  49. [57]

    SwarmGPT-Primitive: A language-driven choreographer for drone swarms using safe motion primitive composition,

    V . Vyas, M. Schuck, D. O. Dahanaggamaarachchi, S. Zhou, and A. P. Schoellig, “SwarmGPT-Primitive: A language-driven choreographer for drone swarms using safe motion primitive composition,”arXiv preprint arXiv:2412.08428, 2024. [Online]. Available: https://arxiv.org/abs/2412. 08428

  50. [58]

    AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,

    E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramer, “AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,”arXiv preprint arXiv:2406.13352, 2024. [Online]. Available: https://arxiv.org/abs/2406. 13352

  51. [59]

    The task shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents,

    F. Jia, T. Wu, X. Qin, and A. Squicciarini, “The task shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents,”arXiv preprint arXiv:2412.16682, 2024. [Online]. Available: https://arxiv.org/abs/2412.16682

  52. [60]

    Identifying the risks of LM agents with an LM-emulated sandbox,

    Y . Ruan, H. Dong, A. Wang, S. Pitis, Y . Zhou, J. Ba, Y . Dubois, C. J. Maddison, and T. Hashimoto, “Identifying the risks of LM agents with an LM-emulated sandbox,”arXiv preprint arXiv:2309.15817, 2023. [Online]. Available: https://arxiv.org/abs/2309.15817

  53. [61]

    Available: https://arxiv.org/abs/2302.02885

    [Online]. Available: https://arxiv.org/abs/2302.02885

  54. [62]

    STPA for learning-enabled systems: A survey and a new practice,

    Y . Qi, Y . Dong, S. Khastgir, P. Jennings, X. Zhao, and X. Huang, “STPA for learning-enabled systems: A survey and a new practice,” arXiv preprint arXiv:2302.10588, 2023. [Online]. Available: https: //arxiv.org/abs/2302.10588

  55. [63]

    CaMeLs can use computers too: System-level security for computer use agents,

    H. Foerster, R. Mullins, T. Blanchard, N. Papernot, K. Nikolic, F. Tramer, I. Shumailov, C. Zhang, and Y . Zhao, “CaMeLs can use computers too: System-level security for computer use agents,”arXiv preprint arXiv:2601.09923, 2026. [Online]. Available: https://arxiv.org/abs/2601.09923

  56. [64]

    Prompt injection attack against LLM-integrated applications,

    Y . Liu, G. Deng, Y . Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng, and Y . Liu, “Prompt injection attack against LLM-integrated applications,”arXiv preprint arXiv:2306.05499, 2023. [Online]. Available: https://arxiv.org/abs/2306.05499

  57. [65]

    InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,

    Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,”arXiv preprint arXiv:2403.02691, 2024. [Online]. Available: https://arxiv.org/abs/2403.02691

  58. [66]

    Zombie agents: Persistent control of self-evolving LLM agents via self-reinforcing injections,

    X. Yang, Y . He, S. Ji, B. Hooi, and J. S. Dong, “Zombie agents: Persistent control of self-evolving LLM agents via self-reinforcing injections,”arXiv preprint arXiv:2602.15654, 2026. [Online]. Available: https://arxiv.org/abs/2602.15654

  59. [67]

    From storage to steering: Memory control flow attacks on LLM agents,

    Z. Xu, X. Zhu, Y . Yao, M. Xue, and Y . Song, “From storage to steering: Memory control flow attacks on LLM agents,”arXiv preprint arXiv:2603.15125, 2026. [Online]. Available: https://arxiv.org/abs/2603. 15125

  60. [68]

    Don’t let the claw grip your hand: A security analysis and defense framework for OpenClaw,

    Z. Shan, J. Xin, Y . Zhang, and M. Xu, “Don’t let the claw grip your hand: A security analysis and defense framework for OpenClaw,”arXiv preprint arXiv:2603.10387, 2026. [Online]. Available: https://arxiv.org/abs/2603.10387

  61. [69]

    AI agents under threat: A survey of key security challenges and future pathways,

    Z. Deng, Y . Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y . Xiang, “AI agents under threat: A survey of key security challenges and future pathways,”arXiv preprint arXiv:2406.02630, 2024. [Online]. Available: https://arxiv.org/abs/2406.02630

  62. [70]

    Measuring harmfulness of computer-using agents,

    A. X. Tian, R. Zhang, J. Tang, and J. Wen, “Measuring harmfulness of computer-using agents,”arXiv preprint arXiv:2508.00935, 2025. [Online]. Available: https://arxiv.org/abs/2508.00935

  63. [72]

    AgentSys: Secure and dynamic LLM agents through explicit hierarchical memory management,

    R. Wen, H. Li, C. Xiao, and N. Zhang, “AgentSys: Secure and dynamic LLM agents through explicit hierarchical memory management,”arXiv preprint arXiv:2602.07398, 2026. [Online]. Available: https://arxiv.org/abs/2602.07398

  64. [73]

    A-MemGuard: A proactive defense framework for LLM-based agent memory,

    Q. Wei, T. Yang, Y . Wang, X. Li, L. Li, Z. Yin, Y . Zhan, T. Holz, Z. Lin, and X. Wang, “A-MemGuard: A proactive defense framework for LLM-based agent memory,”arXiv preprint arXiv:2510.02373, 2025. [Online]. Available: https://arxiv.org/abs/2510.02373

  65. [74]

    Control-aware manipulation of ArduPilot via legitimate MA VLink commands: Simulation and hardware validation,

    F. Benchellal, L. Ben Othmane, Y . Konapalli, C. Tunc, and B. Bhargava, “Control-aware manipulation of ArduPilot via legitimate MA VLink commands: Simulation and hardware validation,”arXiv preprint arXiv:2606.22289, 2026. [Online]. Available: https://arxiv.org/abs/2606. 22289

  66. [75]

    A survey of secure computation using trusted execution environments,

    X. Li, B. Zhao, G. Yang, T. Xiang, J. Weng, and R. H. Deng, “A survey of secure computation using trusted execution environments,”arXiv preprint arXiv:2302.12150, 2023. [Online]. Available: https://arxiv.org/abs/2302.12150

  67. [77]

    Uncovering security threats and architecting defenses in autonomous agents: A case study of OpenClaw,

    Z. Ying, X. Yang, S. Wu, Y . Song, Y . Qu, H. Li, T. Li, J. Wang, A. Liu, and X. Liu, “Uncovering security threats and architecting defenses in autonomous agents: A case study of OpenClaw,”arXiv preprint arXiv:2603.12644, 2026. [Online]. Available: https://arxiv.org/abs/2603.12644

  68. [78]

    OpenClaw PRISM: A zero-fork, defense-in-depth runtime security layer for tool-augmented LLM agents,

    F. Li, “OpenClaw PRISM: A zero-fork, defense-in-depth runtime security layer for tool-augmented LLM agents,”arXiv preprint arXiv:2603.11853, 2026. [Online]. Available: https://arxiv.org/abs/2603. 11853

  69. [79]

    PDRIMA: A policy-driven runtime integrity measurement and attestation approach for ARM TrustZone-based TEE,

    J. Mao and X. Chang, “PDRIMA: A policy-driven runtime integrity measurement and attestation approach for ARM TrustZone-based TEE,”arXiv preprint arXiv:2512.06500, 2025. [Online]. Available: https://arxiv.org/abs/2512.06500

  70. [80]

    Available: https://arxiv.org/abs/2603.07191

    [Online]. Available: https://arxiv.org/abs/2603.07191

  71. [81]

    Security of OpenClaw agents: Fundamentals, attacks, and countermeasures,

    Y . Wang, J. Ba, H. Liu, Y . Pan, J. Wei, Z. Su, T. H. Luan, and L. Du, “Security of OpenClaw agents: Fundamentals, attacks, and countermeasures,”arXiv preprint arXiv:2605.25435, 2026. [Online]. Available: https://arxiv.org/abs/2605.25435

  72. [82]

    Security engineering of OpenClaw: Analyzing attack surface expansion and trust-boundary violations,

    S. Jamshidi, A. M. Dakhel, K. W. Nafi, and F. Khomh, “Security engineering of OpenClaw: Analyzing attack surface expansion and trust-boundary violations,”arXiv preprint arXiv:2606.15008, 2026. [Online]. Available: https://arxiv.org/abs/2606.15008

  73. [86]

    Available: https://arxiv.org/abs/2206.03780

    [Online]. Available: https://arxiv.org/abs/2206.03780

  74. [87]

    Fortress: Securing IoT peripherals with trusted execution environments,

    P. Yuhala, J. Menetrey, P. Felber, M. Pasin, and V . Schiavoni, “Fortress: Securing IoT peripherals with trusted execution environments,” arXiv preprint arXiv:2312.02542, 2023. [Online]. Available: https: //arxiv.org/abs/2312.02542

  75. [88]

    On practicality of using ARM TrustZone trusted execution environment for securing programmable logic controllers,

    Z. Li, D. Mashima, W. S. Ong, E. Esiner, Z. Kalbarczyk, and E.-C. Chang, “On practicality of using ARM TrustZone trusted execution environment for securing programmable logic controllers,” arXiv preprint arXiv:2403.05448, 2024. [Online]. Available: https: //arxiv.org/abs/2403.05448

  76. [2022]

    Available: https://arxiv.org/abs/2212.04088

    [Online]. Available: https://arxiv.org/abs/2212.04088

  77. [2023]

    Available: https://arxiv.org/abs/2307.16789

    [Online]. Available: https://arxiv.org/abs/2307.16789

  78. [2024]

    Available: https://arxiv.org/abs/2406.09246

    [Online]. Available: https://arxiv.org/abs/2406.09246

  79. [2026]

    Available: https://arxiv.org/abs/2605.06393

    [Online]. Available: https://arxiv.org/abs/2605.06393