REVIEW 3 major objections 5 minor 79 references
The paper claims that a self-hosted computer-use agent can safely guide a UAV only if its outputs are compiled into contract-bound skill invocations that the onboard runtime checks for timing, state, and authority before dispatch.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 16:31 UTC pith:HC4WLATA
load-bearing objection Solid architecture with a real evaluation, but the zero-deadline-miss claim doesn't square with the reported latencies, and the security boundary rests on an unverified assumption. the 3 major comments →
RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Control
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that an SHCUA should not be a direct controller of a UAV, but a constrained task-level proposer. The authors formalize this with a contract-bound skill invocation object I = ⟨sid, src, op, args, pre, ddl, vw, sc, auth, inv, fb, ev⟩ and an admissibility predicate admissible(I, St, t) = valid(I, St, t) ∧ (L_rem(I) ≤ ddl(I) − t). An invocation is dispatched only if it is temporally valid, state-consistent, and has enough remaining slack before its deadline. Security enforcement is modeled as η(I, St, P) → {allow, deny, defer, fallback}, with a collision-resistant binding B tying operation, parameters, timing, authority, and policy version. Under three stated assumptions — m
What carries the argument
The contract-bound UAV skill invocation is the central object: a structured execution record that attaches timing metadata (generation, receive, deadline, validity window, timing class), state constraints, authority scope, safety/security invariants, fallback behavior, and evidence requirements to each agent-generated action. It works by converting the SHCUA's free-form tool calls into bounded vehicle operations that the onboard runtime can check before dispatch. The corresponding admissibility condition balances slow offboard reasoning against fast onboard execution, and the timing-class distinction (semantic, tactical, reflex) routes decisions to the appropriate layer so reflex-level safet
Load-bearing premise
The entire security argument collapses if an SHCUA-generated action can reach the UAV skill executor or flight controller without first being compiled into an invocation and passing the onboard admission and enforcement checks; the paper asserts this mediated-dispatch confinement but does not demonstrate that the prototype physically or logically prevents the agent from holding a direct command channel.
What would settle it
In the current prototype, list all tools available to the SHCUA and attempt to send a raw MAVLink command or shell instruction directly to the PX4/Gazebo backend, bypassing the mission compiler and admission path; if the vehicle executes it (or if the agent's tool registry contains such a channel), the mediated-dispatch assumption is false. Separately, manipulate the vehicle-state snapshot or clock to deliver an expired invocation while the validity check passes, showing the state snapshot is trusted but not authenticated.
If this is right
- Natural-language UAV command becomes deployable without making the LLM part of the real-time control loop, since slow reasoning is decoupled from onboard dispatch.
- Expired, stale, or state-inconsistent agent decisions are rejected before reaching the flight controller, preventing a class of unsafe commands.
- Unauthorized, tampered, or replayed invocations are detected at the admission boundary, giving the system a security property rather than just a prompt-level guardrail.
- Every allowed, denied, deferred, or fallback-mediated invocation produces an evidence-chain record, enabling post-flight audit and traceability of agent-influenced vehicle behavior.
- The trusted enforcement boundary can be isolated in a TEE or microcontroller without relocating the full agent or high-frequency flight-control loop, keeping the protected computing base small.
Where Pith is reading between the lines
- The contract-mediation pattern likely generalizes to other LLM-controlled physical systems — ground robots, marine vehicles, or industrial actuators — wherever a slow semantic reasoner proposes actions for a fast, stateful plant; the same admissibility-slack discipline would apply.
- The security theorems rest on Assumption 1 being true in deployment: the prototype asserts that the agent has no direct channel to the skill executor, but a real deployment would need to prove the agent's tool surface is physically confined (e.g., no shell or MAVLink tool) — a testable extension is to audit the agent's registered tools and show only the RT-SHCUA adapter exists.
- The evidence chain's hash-linked records provide a natural foundation for incident forensics and regulatory compliance; one could extend it with attestation of the state snapshot St so that the recorded state cannot be a fabricated input to the enforcement decision.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RT-SHCUA, an architecture that restructures self-hosted computer-use agent (SHCUA) control of UAVs. Instead of letting an LLM agent issue flight commands directly, the agent's outputs are transformed into contract-bound skill invocations carrying timing, state, authority, fallback, and evidence semantics. An onboard runtime validates admissibility, security, and state constraints before dispatching bounded UAV skills. The paper presents a formal model (validity, admissibility, authorization, evidence) with theorems, a prototype built on OpenClaw, PX4 SITL/Gazebo, and QEMU-based OP-TEE, and an evaluation with 1890 normal executions and 7560 security/degraded checks. The central claim is that SHCUA reasoning can be usefully decoupled from time-critical vehicle execution while maintaining bounded task-level responsiveness, security enforcement, and auditability.
Significance. If the architecture is realized as described, it addresses a real and timely problem: safely mediating open-ended LLM-agent tools in a physical, real-time cyber-physical system. The contract-bound skill invocation model is a sensible separator between semantic reasoning and onboard enforcement, and the workload split between semantic, tactical, and reflex classes is well motivated. The prototype is concrete and the overhead measurements for the OP-TEE path, if interpreted carefully, give useful engineering data. The paper's formal theorems, however, are definitional consequences of the assumptions and do not by themselves provide evidence about the physical or security properties of the implementation; the value of the paper rests on the architecture and prototype evaluation, not on the formal apparatus. The evaluation currently does not substantiate the headline real-time and security claims, which is why major revision is needed.
major comments (3)
- [Section VII-B, Fig. 4, Table I] The paper reports OP-TEE P95 end-to-end latencies of 231.6 ms (short), 265.9 ms (standard), and 308.0 ms (extended), while the deadline matrix includes 0.1 s and 0.2 s cells. The claims of 'zero task-level deadline-miss flags and zero fallback events across all 1890 executions' are therefore internally inconsistent with the reported latency distribution unless the measured 'end-to-end' interval is not the interval to which the 'task-level deadline' applies, or the 0.1/0.2 s deadlines were not actually enforced. The paper must specify the exact start and end of the measured latency, the precise definition of 'task-level deadline' relative to that interval, and report per-cell miss counts and deadline-check results. Without this, G1 (real-time responsiveness) is not demonstrated.
- [Section IV-E, Assumption 1; Section VI-A] Assumption 1 states that the SHCUA runtime has no direct channel to the UAV skill executor or flight controller, and Theorems 1-3 depend on it. The prototype description in Section VI-A introduces an 'Agent-Side Decision Mediation' adapter but gives no mechanism that prevents OpenClaw from opening a MAVLink socket, writing to a PX4 serial device, or invoking a shell command that reaches the flight controller. No tool allowlist, OS-level confinement (seccomp/AppArmor/container/namespace), or network filter is described. Since OpenClaw is a general computer-use agent, the asserted boundary may not exist in the implementation, and the security theorems would then not apply to the prototype. The paper should either demonstrate the confinement mechanism or explicitly state the assumed trust boundary and its validation.
- [Section IV-E, Theorems 1-3] The formal properties are derived essentially by unpacking the definitions of admissible, binding, evidence, and the stated Assumptions 1-4. This makes them true by construction rather than substantive guarantees about the implementation. The paper should state explicitly that the theorems are conditional on the enforcement assumptions and that the evaluation does not validate the assumptions themselves (e.g., Assumption 1). The empirical evaluation stands independently, but the 'formal properties' section as written gives an impression of verified security that is not supported.
minor comments (5)
- [Section IV-C, Eq. (1)-(3)] The admissibility condition uses L_rem(I) but the prototype section does not define how L_src, L_comp, L_val, and L_disp are measured. The relationship between L_rem and the end-to-end latency in Fig. 4 should be stated.
- [Section VI-E, Fig. 3] The figure labels 'Key (HMAC / Sign Keys)' and 'Policies & Policy Metadata' inside the TrustZone TA; it would be clearer to note whether these are provisioned off-line and how they are updated, since policy versioning appears in the binding.
- [Section VII-D, Fig. 7] The OP-TEE overhead (roughly 170-175 ms) is from a QEMU-based emulation environment. This should be reported as emulated-environment cost, not a hardware TrustZone cost, to avoid overgeneralization.
- [Section VII-B] Fig. 4 and Fig. 5 show cell-averaged medians/P95s but no variance or per-cell values. Given the inconsistency between P95 and the 0.1/0.2 s deadlines, per-cell latency distributions and miss counts are essential.
- [Section VIII, related work] The related-work survey is thorough, but Table IV uses 'Partial' for many rows without a precise scoring criterion. A short explanation of the scoring would help.
Circularity Check
Formal Theorems 1–3 reduce to Assumptions 2/4 plus the definitions of admissible, B, and ev; the central security guarantees are true by construction rather than independently derived.
specific steps
-
self definitional
[Section IV-C (admissibility condition) and Section IV-E (Theorem 1 and proof)]
"An invocation is admissible for execution only if it is temporally valid, state-consistent, and still has enough remaining slack before its deadline: admissible(I, St, t) = valid(I, St, t) ∧ (Lrem(I) ≤ ddl(I)−t). Theorem 1: Real-time admission control. Under Assumptions 1–3, an SHCUA-generated invocation that is expired, state-inconsistent, or lacks sufficient remaining execution slack cannot be dispatched to the UAV skill executor. Proof. ... By Assumption 2, the runtime dispatches I only if the admissibility check succeeds."
Theorem 1's conclusion is precisely the negation of the admissibility predicate combined with Assumption 2's 'dispatch only if the admissibility check succeeds'. The proof cites the definition and Assumption 2, so the result is true by construction. Assumption 1 also restricts the theorem to invocations that actually enter the runtime, excluding bypasses by assumption rather than ruling them out. The theorem therefore adds no content beyond the model's own definitions.
-
self definitional
[Section IV-D (binding/evidence definitions) and Section IV-E (Theorems 2 and 3)]
"Theorem 2: Security admission control. Under Assumptions 1, 2, and 4, an unauthorized, tampered, expired, or replayed SHCUA-generated invocation cannot be dispatched as a valid UAV action without detection. Proof. ... By Assumption 2, dispatch requires authorization, freshness, and command-binding checks to succeed. ... By Assumption 4, these checks cannot be bypassed by the ordinary SHCUA runtime."
The theorem's guarantees are restatements of Assumption 2 (dispatch requires authorization, freshness, and binding checks), the binding definition B=H(sid, op, args, auth, ddl, vw, Pver), and Assumption 4 (non-bypassability). Theorem 3 similarly follows from the ev tuple definition and Assumption 4's non-suppression clause. These are analytic consequences of the enforcement model, not independent derivations: if the checks exist and cannot be bypassed, then unauthorized/tampered/replayed invocations are detected by definition; if the evidence tuple is generated in a protected path, evidence exists by definition.
full rationale
The paper's genuine contribution is architectural and empirical: it builds a prototype with OpenClaw, PX4 SITL/Gazebo, and OP-TEE, and it measures latencies, fallback behavior, trusted-path consistency, and evidence costs. Those measurements are independent of the formal model and are not circular. The circularity lies in presenting Theorems 1–3 as formal properties of the system: each theorem is a restatement of Assumptions 2/4 and the predicates/objects defined in Section IV. Theorem 1's 'expired/state-inconsistent/insufficient-slack cannot be dispatched' is the negation of admissible(I,St,t) plus Assumption 2; Theorem 2's 'unauthorized/tampered/expired/replayed cannot be dispatched without detection' is Assumption 2 plus B=H(·) plus Assumption 4; Theorem 3 follows from ev plus Assumption 4. These are specification-consistency checks, not results that would falsify a design. In addition, Assumption 1—the claim that the SHCUA has no direct channel to the skill executor or flight controller—is asserted but not demonstrated in the prototype; the theorems' scope therefore excludes the most important bypass scenario by assumption. That is partly a correctness risk rather than pure circularity, and it raises the score. Self-citations [47] and [48] are contextual prior work and are not load-bearing for the derivation, so they do not contribute to the circularity score. Overall, the empirical evaluation is substantive and independent, but the headline security theorems reduce by construction to the model's own assumptions and definitions, giving a score of 5.
Axiom & Free-Parameter Ledger
axioms (6)
- domain assumption Assumption 1 (Mediated dispatch): Every SHCUA-generated UAV action must be compiled into a skill invocation I and submitted to the onboard runtime; the SHCUA runtime has no direct channel to the UAV skill executor or the flight controller.
- domain assumption Assumption 2 (Pre-dispatch enforcement): The onboard runtime enforces admissibility, authorization, freshness, and command-binding checks before dispatch, and dispatches only if these succeed.
- domain assumption Assumption 4 (Protected enforcement path): Security-critical checks and evidence generation are performed in a trusted/isolated path that cannot be bypassed by the ordinary SHCUA runtime.
- domain assumption State snapshot St and policy P are authentic and current.
- standard math H(.) is collision-resistant or authenticated.
- domain assumption The UAV platform exposes a closed skill library with complete contract schemas (preconditions, parameter ranges, fallback behavior) for every operation the SHCUA may propose.
read the original abstract
Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed agent iterations are often acceptable. UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic reasoning from onboard execution and security/safety enforcement. Slow cloud or edge reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated UAV actions.
Figures
Reference graph
Works this paper leans on
-
[1]
Taking flight with dialogue: Enabling natural language control for PX4-based drone agent,
S. K. Lim, M. J. Y . Chong, J. H. Khor, and T. Y . Ling, “Taking flight with dialogue: Enabling natural language control for PX4-based drone agent,”arXiv preprint arXiv:2506.07509, 2025. [Online]. Available: https://arxiv.org/abs/2506.07509
Pith/arXiv arXiv 2025
-
[2]
Next-generation LLM for UA V: From natural language to autonomous flight,
L. Yuan, C. Deng, D.-J. Han, I. Hwang, S. Brunswicker, and C. G. Brinton, “Next-generation LLM for UA V: From natural language to autonomous flight,”arXiv preprint arXiv:2510.21739, 2025. [Online]. Available: https://arxiv.org/abs/2510.21739
arXiv 2025
-
[3]
A prompt-driven task planning method for multi-drones based on large language model,
Y . Liu, “A prompt-driven task planning method for multi-drones based on large language model,”arXiv preprint arXiv:2406.00006, 2024. [Online]. Available: https://arxiv.org/abs/2406.00006
Pith/arXiv arXiv 2024
-
[4]
A universal large language model–drone command and control interface,
J. N. Ramos-Silva and P. J. Burke, “A universal large language model–drone command and control interface,” arXiv preprint arXiv:2601.15486, 2026. [Online]. Available: https://arxiv.org/abs/2601.15486
Pith/arXiv arXiv 2026
-
[5]
Y . Emami, H. Zhou, R. Reddy, A. H. Arani, B. Wang, K. Li, L. Almeida, and Z. Han, “Large language model-assisted UA V operations and communications: A multifaceted survey and tutorial,” arXiv preprint arXiv:2602.19534, 2026. [Online]. Available: https: //arxiv.org/abs/2602.19534 IEEE JOURNAL TEMPLATE, VOL. XX, NO. XX, MONTH 2026 18
arXiv 2026
-
[6]
UA Vs meet LLMs: Overviews and perspectives toward agentic low-altitude mobility,
Y . Tian, F. Lin, Y . Li, T. Zhang, Q. Zhang, X. Fu, J. Huang, X. Dai, Y . Wang, C. Tian, B. Li, Y . Lv, L. Kov ´acs, and F.-Y . Wang, “UA Vs meet LLMs: Overviews and perspectives toward agentic low-altitude mobility,”Information Fusion, vol. 122, p. 103158, 2025. [Online]. Available: https://arxiv.org/abs/2501.02341
Pith/arXiv arXiv 2025
-
[7]
When large language models meet UA Vs: How far are we?
Y . Chen, X. Que, J. Zhang, T. Chen, G. Li, and J. Chen, “When large language models meet UA Vs: How far are we?” arXiv preprint arXiv:2509.12795, 2025. [Online]. Available: https: //arxiv.org/abs/2509.12795
Pith/arXiv arXiv 2025
-
[8]
TypeFly: Flying drones with large language model,
G. Chen, X. Yu, N. Ling, and L. Zhong, “TypeFly: Flying drones with large language model,”arXiv preprint arXiv:2312.14950, 2023. [Online]. Available: https://arxiv.org/abs/2312.14950
Pith/arXiv arXiv 2023
-
[9]
AerialClaw: An open-source framework for LLM-driven autonomous aerial agents,
K. Li, J. Yang, L. Zhang, G. Yu, C. Yan, Y . Ding, D. Wang, N. Luo, G. Liu, X. Gao, and Q. Wang, “AerialClaw: An open-source framework for LLM-driven autonomous aerial agents,” arXiv preprint arXiv:2606.12142, 2026. [Online]. Available: https: //arxiv.org/abs/2606.12142
Pith/arXiv arXiv 2026
-
[10]
Do as i can, not as i say: Grounding language in robotic affordances,
M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausmanet al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022. [Online]. Available: https: //arxiv.org/abs/2204.01691
Pith/arXiv arXiv 2022
-
[12]
Code as policies: Language model programs for embodied control,
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,”arXiv preprint arXiv:2209.07753, 2022. [Online]. Available: https://arxiv.org/abs/2209.07753
Pith/arXiv arXiv 2022
-
[14]
Inner monologue: Embodied reasoning through planning with language models,
W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotaret al., “Inner monologue: Embodied reasoning through planning with language models,”arXiv preprint arXiv:2207.05608, 2022. [Online]. Available: https://arxiv.org/abs/2207.05608
Pith/arXiv arXiv 2022
-
[15]
Available: https://arxiv.org/abs/2209.11302
[Online]. Available: https://arxiv.org/abs/2209.11302
-
[16]
PaLM-E: An embodied multimodal language model,
D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, J. Tompson, Q. Vuong, T. Yuet al., “PaLM-E: An embodied multimodal language model,”arXiv preprint arXiv:2303.03378, 2023. [Online]. Available: https://arxiv.org/abs/2303.03378
Pith/arXiv arXiv 2023
-
[17]
ChatGPT for robotics: Design principles and model abilities,
S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,”arXiv preprint arXiv:2306.17582, 2023. [Online]. Available: https://arxiv.org/abs/2306. 17582
Pith/arXiv arXiv 2023
-
[19]
RT-2: Vision-language-action models transfer web knowledge to robotic control,
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finnet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,”arXiv preprint arXiv:2307.15818, 2023. [Online]. Available: https://arxiv.org/abs/2307.15818
Pith/arXiv arXiv 2023
-
[20]
V oxPoser: Composable 3D value maps for robotic manipulation with language models,
W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxPoser: Composable 3D value maps for robotic manipulation with language models,”arXiv preprint arXiv:2307.05973, 2023. [Online]. Available: https://arxiv.org/abs/2307.05973
Pith/arXiv arXiv 2023
-
[21]
Manipulate-anything: Automating real-world robots using vision-language models,
J. Duan, W. Yuan, W. Pumacay, Y . R. Wang, K. Ehsani, D. Fox, and R. Krishna, “Manipulate-anything: Automating real-world robots using vision-language models,”arXiv preprint arXiv:2406.18915, 2024. [Online]. Available: https://arxiv.org/abs/2406.18915
Pith/arXiv arXiv 2024
-
[22]
EmbodiedGPT: Vision-language pre-training via embodied chain of thought,
Y . Mu, Q. Zhang, M. Hu, M. Ding, W. Wang, J. Jin, B. Wang, Y . Qiao, and P. Luo, “EmbodiedGPT: Vision-language pre-training via embodied chain of thought,”arXiv preprint arXiv:2305.15021, 2023. [Online]. Available: https://arxiv.org/abs/2305.15021
Pith/arXiv arXiv 2023
-
[23]
A survey on large language model based autonomous agents,
L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”arXiv preprint arXiv:2308.11432, 2023. [Online]. Available: https://arxiv.org/abs/2308.11432
Pith/arXiv arXiv 2023
-
[24]
The rise and potential of large language model based agents: A survey,
Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”arXiv preprint arXiv:2309.07864, 2023. [Online]. Available: https://arxiv.org/abs/2309.07864
Pith/arXiv arXiv 2023
-
[25]
V oyager: An open-ended embodied agent with large language models,
G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023. [Online]. Available: https://arxiv.org/abs/2305.16291
Pith/arXiv arXiv 2023
-
[26]
Toolformer: Language models can teach themselves to use tools,
T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” arXiv preprint arXiv:2302.04761, 2023. [Online]. Available: https://arxiv.org/abs/2302.04761
Pith/arXiv arXiv 2023
-
[27]
Gorilla: Large language model connected with massive apis,
S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,”arXiv preprint arXiv:2305.15334, 2023. [Online]. Available: https://arxiv.org/abs/2305. 15334
Pith/arXiv arXiv 2023
-
[28]
ReAct: Synergizing reasoning and acting in language models,
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022. [Online]. Available: https://arxiv.org/abs/2210.03629
Pith/arXiv arXiv 2022
-
[29]
TaskMatrix.AI: Completing tasks by connecting foundation models with millions of apis,
Y . Liang, C. Wu, T. Song, W. Wu, Y . Xia, Y . Liu, Y . Ou, S. Lu, L. Ji, S. Maoet al., “TaskMatrix.AI: Completing tasks by connecting foundation models with millions of apis,”arXiv preprint arXiv:2303.16434, 2023. [Online]. Available: https://arxiv.org/abs/2303. 16434
Pith/arXiv arXiv 2023
-
[30]
HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,
Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,”arXiv preprint arXiv:2303.17580, 2023. [Online]. Available: https://arxiv.org/abs/2303.17580
Pith/arXiv arXiv 2023
-
[32]
J. Liu, Q. Lin, C. Qian, R. Wang, E. C. Acikgoz, X. Yang, J. Liu, Z. Wang, X. Chen, H. Ji, and D. Hakkani-Tur, “PlanBench-XL: Evaluating long-horizon planning of LLM tool-use agents in large-scale tool ecosystems,”arXiv preprint arXiv:2606.22388, 2026. [Online]. Available: https://arxiv.org/abs/2606.22388
Pith/arXiv arXiv 2026
-
[33]
Tool learning with large language models: A survey,
C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, “Tool learning with large language models: A survey,”arXiv preprint arXiv:2405.17935, 2024. [Online]. Available: https://arxiv.org/abs/2405.17935
Pith/arXiv arXiv 2024
-
[34]
Mind2Web: Towards a generalist agent for the web,
X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2Web: Towards a generalist agent for the web,”arXiv preprint arXiv:2306.06070, 2023. [Online]. Available: https://arxiv.org/abs/2306.06070
Pith/arXiv arXiv 2023
-
[35]
τ-bench: A benchmark for tool-agent-user interaction in real-world domains,
S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [Online]. Available: https: //arxiv.org/abs/2406.12045
Pith/arXiv arXiv 2024
-
[36]
VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,
J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. C. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “VisualWebArena: Evaluating multimodal agents on realistic visual web tasks,”arXiv preprint arXiv:2401.13649, 2024. [Online]. Available: https://arxiv.org/abs/2401.13649
Pith/arXiv arXiv 2024
-
[37]
WebV oyager: Building an end-to-end web agent with large multimodal models,
H. He, W. Yao, K. Ma, W. Yu, Y . Dai, H. Zhang, Z. Lan, and D. Yu, “WebV oyager: Building an end-to-end web agent with large multimodal models,”arXiv preprint arXiv:2401.13919, 2024. [Online]. Available: https://arxiv.org/abs/2401.13919
Pith/arXiv arXiv 2024
-
[39]
WebArena: A realistic web environment for building autonomous agents,
S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “WebArena: A realistic web environment for building autonomous agents,”arXiv preprint arXiv:2307.13854, 2023. [Online]. Available: https://arxiv.org/abs/2307.13854
Pith/arXiv arXiv 2023
-
[40]
AndroidWorld: A dynamic benchmarking environment for autonomous agents,
C. Rawles, S. Clinckemaillie, Y . Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva, “AndroidWorld: A dynamic benchmarking environment for autonomous agents,” arXiv preprint arXiv:2405.14573, 2024. [Online]. Available: https: //arxiv.org/abs/2405.14573 IEEE JOURNAL ...
Pith/arXiv arXiv 2024
-
[41]
Windows agent arena: Evaluating multi-modal OS agents at scale,
R. Bonatti, D. Zhao, F. Bonacci, D. Dupont, S. Abdali, Y . Li, Y . Lu, J. Wagle, K. Koishida, A. Buckeret al., “Windows agent arena: Evaluating multi-modal OS agents at scale,”arXiv preprint arXiv:2409.08264, 2024. [Online]. Available: https://arxiv.org/abs/2409. 08264
Pith/arXiv arXiv 2024
-
[42]
WorkArena: How capable are web agents at solving common knowledge work tasks?
A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquezet al., “WorkArena: How capable are web agents at solving common knowledge work tasks?”arXiv preprint arXiv:2403.07718, 2024. [Online]. Available: https://arxiv.org/abs/2403.07718
Pith/arXiv arXiv 2024
-
[43]
Available: https://arxiv.org/abs/2412.05467
[Online]. Available: https://arxiv.org/abs/2412.05467
-
[44]
OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,
T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Leiet al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” arXiv preprint arXiv:2404.07972, 2024. [Online]. Available: https: //arxiv.org/abs/2404.07972
Pith/arXiv arXiv 2024
-
[45]
OS agents: A survey on MLLM- based agents for general computing devices use,
X. Hu, T. Xiong, B. Yiet al., “OS agents: A survey on MLLM- based agents for general computing devices use,”arXiv preprint arXiv:2508.04482, 2025. [Online]. Available: https://arxiv.org/abs/2508. 04482
Pith/arXiv arXiv 2025
-
[46]
A survey on the safety and security threats of computer-using agents: JARVIS or Ultron?
A. Chen, Y . Wu, J. Zhang, S. Yang, J.-t. Huang, K. Wang, W. Wang, and S. Wang, “A survey on the safety and security threats of computer-using agents: JARVIS or Ultron?”arXiv preprint arXiv:2505.10924, 2025. [Online]. Available: https://arxiv.org/abs/2505.10924
Pith/arXiv arXiv 2025
-
[47]
When convenience becomes risk: A semantic view of under-specification in host-acting agents,
D. Lu, Y . Liao, X. Mu, L. Zheng, K. Cheng, X. Dong, Y . Shen, and J. Ma, “When convenience becomes risk: A semantic view of under-specification in host-acting agents,”Journal of Networking and Network Applications, vol. 6, no. 2, pp. 47–54, 2026. [Online]. Available: https://iecscience.org/jpapers/224
2026
-
[49]
Available: https://arxiv.org/abs/2407.05291
[Online]. Available: https://arxiv.org/abs/2407.05291
-
[50]
Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,
J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,”arXiv preprint arXiv:2401.16158, 2024. [Online]. Available: https://arxiv.org/abs/2401.16158
Pith/arXiv arXiv 2024
-
[51]
K. Hobbs, M. Mote, M. Abate, S. Coogan, and E. Feron, “Run time assurance for safety-critical systems: An introduction to safety filtering approaches for complex control systems,”arXiv preprint arXiv:2110.03506, 2021. [Online]. Available: https://arxiv.org/abs/2110. 03506
Pith/arXiv arXiv 2021
-
[52]
K. Aslansefat, P. Nikolaou, M. Walker, M. N. Akram, I. Sorokos, J. Reich, P. Kolios, M. K. Michael, T. Theocharides, G. Ellinaset al., “SafeDrones: Real-time reliability evaluation of UA Vs using executable digital dependable identities,”arXiv preprint arXiv:2207.05643, 2022. [Online]. Available: https://arxiv.org/abs/2207.05643
Pith/arXiv arXiv 2022
-
[55]
K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world LLM-integrated applications with indirect prompt injection,” arXiv preprint arXiv:2302.12173, 2023. [Online]. Available: https: //arxiv.org/abs/2302.12173
Pith/arXiv arXiv 2023
-
[56]
SwarmGPT: Combining large language models with safe motion planning for drone swarm choreography,
M. Schuck, D. O. Dahanaggamaarachchi, B. Sprenger, V . Vyas, S. Zhou, and A. P. Schoellig, “SwarmGPT: Combining large language models with safe motion planning for drone swarm choreography,”IEEE Robotics and Automation Letters, vol. 10, no. 11, pp. 12 237–12 244, 2025
2025
-
[57]
V . Vyas, M. Schuck, D. O. Dahanaggamaarachchi, S. Zhou, and A. P. Schoellig, “SwarmGPT-Primitive: A language-driven choreographer for drone swarms using safe motion primitive composition,”arXiv preprint arXiv:2412.08428, 2024. [Online]. Available: https://arxiv.org/abs/2412. 08428
arXiv 2024
-
[58]
AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,
E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramer, “AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,”arXiv preprint arXiv:2406.13352, 2024. [Online]. Available: https://arxiv.org/abs/2406. 13352
Pith/arXiv arXiv 2024
-
[59]
The task shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents,
F. Jia, T. Wu, X. Qin, and A. Squicciarini, “The task shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents,”arXiv preprint arXiv:2412.16682, 2024. [Online]. Available: https://arxiv.org/abs/2412.16682
Pith/arXiv arXiv 2024
-
[60]
Identifying the risks of LM agents with an LM-emulated sandbox,
Y . Ruan, H. Dong, A. Wang, S. Pitis, Y . Zhou, J. Ba, Y . Dubois, C. J. Maddison, and T. Hashimoto, “Identifying the risks of LM agents with an LM-emulated sandbox,”arXiv preprint arXiv:2309.15817, 2023. [Online]. Available: https://arxiv.org/abs/2309.15817
Pith/arXiv arXiv 2023
-
[61]
Available: https://arxiv.org/abs/2302.02885
[Online]. Available: https://arxiv.org/abs/2302.02885
-
[62]
STPA for learning-enabled systems: A survey and a new practice,
Y . Qi, Y . Dong, S. Khastgir, P. Jennings, X. Zhao, and X. Huang, “STPA for learning-enabled systems: A survey and a new practice,” arXiv preprint arXiv:2302.10588, 2023. [Online]. Available: https: //arxiv.org/abs/2302.10588
Pith/arXiv arXiv 2023
-
[63]
CaMeLs can use computers too: System-level security for computer use agents,
H. Foerster, R. Mullins, T. Blanchard, N. Papernot, K. Nikolic, F. Tramer, I. Shumailov, C. Zhang, and Y . Zhao, “CaMeLs can use computers too: System-level security for computer use agents,”arXiv preprint arXiv:2601.09923, 2026. [Online]. Available: https://arxiv.org/abs/2601.09923
Pith/arXiv arXiv 2026
-
[64]
Prompt injection attack against LLM-integrated applications,
Y . Liu, G. Deng, Y . Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y . Liu, H. Wang, Y . Zheng, and Y . Liu, “Prompt injection attack against LLM-integrated applications,”arXiv preprint arXiv:2306.05499, 2023. [Online]. Available: https://arxiv.org/abs/2306.05499
Pith/arXiv arXiv 2023
-
[65]
InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,
Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,”arXiv preprint arXiv:2403.02691, 2024. [Online]. Available: https://arxiv.org/abs/2403.02691
Pith/arXiv arXiv 2024
-
[66]
Zombie agents: Persistent control of self-evolving LLM agents via self-reinforcing injections,
X. Yang, Y . He, S. Ji, B. Hooi, and J. S. Dong, “Zombie agents: Persistent control of self-evolving LLM agents via self-reinforcing injections,”arXiv preprint arXiv:2602.15654, 2026. [Online]. Available: https://arxiv.org/abs/2602.15654
arXiv 2026
-
[67]
From storage to steering: Memory control flow attacks on LLM agents,
Z. Xu, X. Zhu, Y . Yao, M. Xue, and Y . Song, “From storage to steering: Memory control flow attacks on LLM agents,”arXiv preprint arXiv:2603.15125, 2026. [Online]. Available: https://arxiv.org/abs/2603. 15125
Pith/arXiv arXiv 2026
-
[68]
Don’t let the claw grip your hand: A security analysis and defense framework for OpenClaw,
Z. Shan, J. Xin, Y . Zhang, and M. Xu, “Don’t let the claw grip your hand: A security analysis and defense framework for OpenClaw,”arXiv preprint arXiv:2603.10387, 2026. [Online]. Available: https://arxiv.org/abs/2603.10387
arXiv 2026
-
[69]
AI agents under threat: A survey of key security challenges and future pathways,
Z. Deng, Y . Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y . Xiang, “AI agents under threat: A survey of key security challenges and future pathways,”arXiv preprint arXiv:2406.02630, 2024. [Online]. Available: https://arxiv.org/abs/2406.02630
arXiv 2024
-
[70]
Measuring harmfulness of computer-using agents,
A. X. Tian, R. Zhang, J. Tang, and J. Wen, “Measuring harmfulness of computer-using agents,”arXiv preprint arXiv:2508.00935, 2025. [Online]. Available: https://arxiv.org/abs/2508.00935
arXiv 2025
-
[72]
AgentSys: Secure and dynamic LLM agents through explicit hierarchical memory management,
R. Wen, H. Li, C. Xiao, and N. Zhang, “AgentSys: Secure and dynamic LLM agents through explicit hierarchical memory management,”arXiv preprint arXiv:2602.07398, 2026. [Online]. Available: https://arxiv.org/abs/2602.07398
arXiv 2026
-
[73]
A-MemGuard: A proactive defense framework for LLM-based agent memory,
Q. Wei, T. Yang, Y . Wang, X. Li, L. Li, Z. Yin, Y . Zhan, T. Holz, Z. Lin, and X. Wang, “A-MemGuard: A proactive defense framework for LLM-based agent memory,”arXiv preprint arXiv:2510.02373, 2025. [Online]. Available: https://arxiv.org/abs/2510.02373
arXiv 2025
-
[74]
F. Benchellal, L. Ben Othmane, Y . Konapalli, C. Tunc, and B. Bhargava, “Control-aware manipulation of ArduPilot via legitimate MA VLink commands: Simulation and hardware validation,”arXiv preprint arXiv:2606.22289, 2026. [Online]. Available: https://arxiv.org/abs/2606. 22289
Pith/arXiv arXiv 2026
-
[75]
A survey of secure computation using trusted execution environments,
X. Li, B. Zhao, G. Yang, T. Xiang, J. Weng, and R. H. Deng, “A survey of secure computation using trusted execution environments,”arXiv preprint arXiv:2302.12150, 2023. [Online]. Available: https://arxiv.org/abs/2302.12150
Pith/arXiv arXiv 2023
-
[77]
Z. Ying, X. Yang, S. Wu, Y . Song, Y . Qu, H. Li, T. Li, J. Wang, A. Liu, and X. Liu, “Uncovering security threats and architecting defenses in autonomous agents: A case study of OpenClaw,”arXiv preprint arXiv:2603.12644, 2026. [Online]. Available: https://arxiv.org/abs/2603.12644
arXiv 2026
-
[78]
OpenClaw PRISM: A zero-fork, defense-in-depth runtime security layer for tool-augmented LLM agents,
F. Li, “OpenClaw PRISM: A zero-fork, defense-in-depth runtime security layer for tool-augmented LLM agents,”arXiv preprint arXiv:2603.11853, 2026. [Online]. Available: https://arxiv.org/abs/2603. 11853
arXiv 2026
-
[79]
J. Mao and X. Chang, “PDRIMA: A policy-driven runtime integrity measurement and attestation approach for ARM TrustZone-based TEE,”arXiv preprint arXiv:2512.06500, 2025. [Online]. Available: https://arxiv.org/abs/2512.06500
arXiv 2025
-
[80]
Available: https://arxiv.org/abs/2603.07191
[Online]. Available: https://arxiv.org/abs/2603.07191
-
[81]
Security of OpenClaw agents: Fundamentals, attacks, and countermeasures,
Y . Wang, J. Ba, H. Liu, Y . Pan, J. Wei, Z. Su, T. H. Luan, and L. Du, “Security of OpenClaw agents: Fundamentals, attacks, and countermeasures,”arXiv preprint arXiv:2605.25435, 2026. [Online]. Available: https://arxiv.org/abs/2605.25435
Pith/arXiv arXiv 2026
-
[82]
Security engineering of OpenClaw: Analyzing attack surface expansion and trust-boundary violations,
S. Jamshidi, A. M. Dakhel, K. W. Nafi, and F. Khomh, “Security engineering of OpenClaw: Analyzing attack surface expansion and trust-boundary violations,”arXiv preprint arXiv:2606.15008, 2026. [Online]. Available: https://arxiv.org/abs/2606.15008
arXiv 2026
-
[86]
Available: https://arxiv.org/abs/2206.03780
[Online]. Available: https://arxiv.org/abs/2206.03780
-
[87]
Fortress: Securing IoT peripherals with trusted execution environments,
P. Yuhala, J. Menetrey, P. Felber, M. Pasin, and V . Schiavoni, “Fortress: Securing IoT peripherals with trusted execution environments,” arXiv preprint arXiv:2312.02542, 2023. [Online]. Available: https: //arxiv.org/abs/2312.02542
Pith/arXiv arXiv 2023
-
[88]
Z. Li, D. Mashima, W. S. Ong, E. Esiner, Z. Kalbarczyk, and E.-C. Chang, “On practicality of using ARM TrustZone trusted execution environment for securing programmable logic controllers,” arXiv preprint arXiv:2403.05448, 2024. [Online]. Available: https: //arxiv.org/abs/2403.05448
Pith/arXiv arXiv 2024
-
[2022]
Available: https://arxiv.org/abs/2212.04088
[Online]. Available: https://arxiv.org/abs/2212.04088
-
[2023]
Available: https://arxiv.org/abs/2307.16789
[Online]. Available: https://arxiv.org/abs/2307.16789
-
[2024]
Available: https://arxiv.org/abs/2406.09246
[Online]. Available: https://arxiv.org/abs/2406.09246
-
[2026]
Available: https://arxiv.org/abs/2605.06393
[Online]. Available: https://arxiv.org/abs/2605.06393
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.