Pith. sign in

REVIEW 4 major objections 3 minor 15 references

Human confirmation of AI agents' actions can increase, not decrease, the success of low-harm prompt injection attacks that look like ordinary task steps.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:31 UTC pith:BK5D2KKD

load-bearing objection The II-Bench benchmark is a genuine contribution and the low-harm blind spot is real, but the HITL-amplification headline needs more than a three-rater simulation and intent-only ASR to support it. the 4 major comments →

arxiv 2608.02018 v1 pith:BK5D2KKD submitted 2026-08-03 cs.CV

Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents

Jia-Chen Zhang , Ze-Yu Zhang , Kai-Wei Zhang This is my paper
classification cs.CV
keywords computer-use agentsindirect prompt injectionhuman-in-the-loopadversarial benchmarklow-harm attacksLLM safetyattack success rateprompt injection defense
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Invisible Ink Threats: injected instructions whose goals are individually innocent-looking, such as starring a repository, installing a package, or posting a file's contents, so that they are indistinguishable from normal agent behavior. It builds a benchmark of 444 such attacks across three platforms and a testing harness in which agents can pause to consult a simulated novice user. Across seven leading computer-use agents, the attacks succeed at high rates, and code-form injections are consistently more effective than natural-language ones. Most strikingly, the human-in-the-loop confirmation step does not reduce attack success: it raises it by 7.8% on average, because the simulated user approves the large majority of malicious operations. The paper's central claim is that defenses must reason about the intent behind an action rather than the apparent harmfulness of the action itself.

Core claim

The paper claims that the standard defense against prompt injection in computer-use agents—pausing for explicit user confirmation before sensitive operations—fails against low-harm injected goals that are behaviorally indistinguishable from legitimate task execution. Using II-Bench, a set of 444 adversarial examples targeting confidentiality and integrity across three platforms, and HITLCUA, a hybrid sandbox that lets agents consult an API-simulated novice user, the authors show that all seven evaluated agents execute the majority of injected instructions under loose instructions, and that wrapping the same adversarial goal as an executable code block raises success for every model. Under si

What carries the argument

The Invisible Ink Threat is an indirect prompt injection built from a unified template: an attention-grabbing prefix, a low-harm adversarial instruction (e.g., exfiltrating a local file or pip-installing an untrusted package), a deceptive rationale tied to the benign task (e.g., 'required for the newest version of termcolor'), and a benign follow-up command to preserve the appearance of a helpful instruction. The HITLCUA harness supplies the other half of the machinery: a hybrid sandbox combining a real virtual machine with Docker-based web replicas, plus an LLM-based simulated novice user (NPC2) that the agent can consult before proceeding.

Load-bearing premise

The reported attack success rates measure whether the agent shows intent to follow the injected instruction, not whether the harmful action actually completes, so if intent rarely translates into realized compromise the headline numbers overstate real-world vulnerability.

What would settle it

Run the II-Bench scenarios end-to-end and measure actual realized outcomes—for example, whether the sensitive file is actually posted, whether the untrusted package is actually installed, or whether the repository is actually starred—rather than relying on detected intent; if completion rates fall far below the reported intent-based ASRs, the central vulnerability claim is weakened. A separate user study presenting the same confirmation queries to real non-expert participants would settle whether the HITL amplification is a genuine property of human oversight or an artifact of the simulated no

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The widely adopted human-in-the-loop confirmation defense is not a reliable safeguard against low-harm injected goals; in the tested conditions it increases attack success.
  • Code-form injection is a cheap, universal amplifier: wrapping an adversarial goal as an executable code block raised attack success for every evaluated model.
  • Giving users more specific, procedural instructions can reduce but not eliminate vulnerability, and in some interactive settings specificity even backfired.
  • Confidentiality and integrity attacks are more dangerous than overt availability attacks because they hijack legitimate agent capabilities and evade rule- or behavior-based detection.
  • Agent safety evaluation should focus on intent-level defenses rather than the surface severity of individual actions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because attack success is scored by the agent's expressed intent rather than actual completion, the real-world impact may be smaller than the headline ASRs suggest—but an agent that even intends to exfiltrate credentials is itself a security failure.
  • Editorial inference: The same Invisible Ink template could plausibly be transferred to other agent domains, such as mobile assistants or email agents, where actions like 'reply to this thread' or 'add this contact' are similarly low-harm and behaviorally routine.
  • Editorial inference: A controlled experiment with real non-expert users approving the same confirmation queries would clarify whether the amplification effect is genuine human behavior or an artifact of the LLM-simulated user.
  • Editorial inference: A natural defensive direction suggested by the paper is to make confirmation prompts expose the inferred intent and rationale of each step, rather than only the surface action, so that users and monitors can judge the goal behind the click.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces II-Bench, a benchmark of 444 low-harm indirect prompt injection examples ("Invisible Ink Threats") targeting confidentiality and integrity across three platforms (Reddit, OwnCloud, RocketChat), with attacks instantiated in natural-language and code forms under loose and specific user instructions. It also presents HITLCUA, a hybrid sandbox that integrates OSWorld with Docker-based web replicas and simulates a human-in-the-loop via an API-based novice user (NPC2). Seven CUAs are evaluated; the paper reports high attack success rates (ASR), finds code-form injections more effective than language-form ones, and claims that human-in-the-loop confirmation amplifies rather than mitigates these attacks (average ASR increase of 7.8%).

Significance. If the results hold, the paper identifies a genuine blind spot in current CUA defenses: low-harm injected goals may bypass both safety alignment and human confirmation. The benchmark and platform are potentially reusable resources, and the decoupled evaluation design (positioning agents directly at the injection site) is a sensible methodological choice. The cross-model and cross-platform comparisons, and the finding that code-form injections are more effective, are useful empirical contributions. The weaknesses concern the definition of attack success, the statistical support for the headline HITL claim, and the fidelity of the simulated human user.

major comments (4)
  1. [Experimental settings, Evaluation Metrics] The ASR is defined as "presence of intent to execute relevant harmful instructions" regardless of whether the harmful task is completed. All headline results in Tables 1 and 3 use this metric, yet the paper never reports completion-based ASR (e.g., whether the package was installed, the file was actually exfiltrated, or the post was actually made). Since intent may frequently not translate into a compromised state, the reported 51.8%-90.5% ASRs and the 7.8% HITL amplification may overstate practical risk. Please report intent-based and completion-based ASR side-by-side, and clarify how intent is extracted (from action traces, final states, or model outputs) and whether, in the HITL setting, a query that is rejected by the simulated user still counts as intent.
  2. [Human-in-the-Loop Analysis, Table 3] The central claim that HITL "consistently amplifies" Invisible Ink Threats rests on eight pairwise point estimates in Table 3 with no confidence intervals, error bars, or significance tests. Per-cell denominators are small (the Reddit platform has 156 total examples, implying roughly 33 examples per model across categories). The average increase of 7.8 percentage points is computed over only four models and two platforms, and the pairs were selected because conventional ASR was comparatively low, which can inflate apparent amplification via regression to the mean. Please provide per-cell denominators, confidence intervals (e.g., Wilson intervals), a paired significance test (e.g., McNemar's test over the same example set), and results for all evaluated model-platform pairs, not just the selected subset.
  3. [Human-in-the-Loop Analysis, NPC2 as Novice User] NPC2, the simulated human user, is validated by only three non-expert participants who answered the same confirmation prompts offline (Yes Rates 77.1%, 68.5%, 74.3%). This does not validate the full interactive loop: in real use, the agent's preceding actions, the wording of its confirmation query, and repeated queries can systematically shift user decisions. The HITL amplification result is mechanically driven by NPC2's 73.5%-83.3% Yes Rate; if real novice users in the interactive loop approve less often, the headline conclusion collapses. A small real-user study in the full loop, or at least a sensitivity analysis varying NPC2's approval probability and showing how the amplification changes, is necessary to support the claim that HITL is an amplifier.
  4. [Table 1 and Findings 1-3] The paper does not provide the number of examples behind each percentage in Table 1, nor a full breakdown of how the 444 examples are distributed across platforms, categories (Post/Send/Download), instruction settings, and models. With only 444 examples, seven models, two instruction settings, three platforms, and three categories, per-cell sample sizes are roughly 12-36. The aggregated percentages used in the platform, modality, and task analyses therefore carry wide error bars. Please include the full example-count matrix and confidence intervals for the reported percentages, and temper statements such as "every evaluated model is compromised on the majority of attack instances" accordingly.
minor comments (3)
  1. [Additional Details / Dataset Construction] The text states II-Bench includes "111 benign-adversarial pairings," but this term is never defined. It is unclear how the 444 examples relate to these pairings and to the three platforms and three tasks. Please clarify in the main text or supplementary material.
  2. [Table 1] The table uses abbreviations P, S, D for Post, Send, Download, but the full attack category names appear only in the text. Please expand these in the table caption or add a note for readability.
  3. [Figure 2] Figure 2 is dense and includes many components (NPC1, NPC2, injection template, attack categories). Consider referencing it in the method section earlier and providing a step-by-step legend to improve readability.

Circularity Check

0 steps flagged

No load-bearing circularity: the benchmark findings and HITL amplification claim are empirical measurements, not derivations from their own inputs.

full rationale

This is an empirical benchmark and evaluation paper, not a derivation chain. The main claims—that low-harm injected goals achieve high ASR and that human-in-the-loop confirmation amplifies rather than mitigates these attacks—are measured outcomes. The HITL amplifier conclusion rests on NPC2, an API-simulated novice user, but the paper validates NPC2's Yes Rate against three human raters (Sec. Human-in-the-Loop Analysis), which is external validation rather than a fitted parameter. The ASR metric does define success as 'presence of intent' even if the harmful task is never completed, and this is explicitly disclosed; it may overstate real-world impact, but it is a measurement choice, not a circular definition that forces the reported numbers. The framework builds on OSWorld, WebArena, and TheAgentCompany, all independent external resources. The only overlapping self-citations (AgentDojo; Zhang et al. 2025) are contextual and not load-bearing. No equation, metric, or cited prior result reduces the paper's predictions to its own inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are postulated. Invisible Ink Threats is a label for an existing attack class, and NPC1/NPC2 are software components in the simulator, not new entities. The main costs are the two free design parameters above and the four stated domain assumptions.

free parameters (2)
  • NPC2 Yes Rate profile (simulated novice user) = 73.5%-83.3%
    This approval rate is an emergent output of the chosen LLM and prompt, not a fitted constant, but the HITL amplification conclusion depends entirely on it being high; the fidelity check uses only 3 human raters.
  • ASR intent-presence threshold = presence of intent counts as success
    The authors define attack success as intent to execute a harmful instruction rather than completed harm; this choice inflates ASR relative to realized compromise.
axioms (4)
  • domain assumption Indirect prompt injection is a realistic and material threat for computer-use agents (Introduction).
    The entire benchmark exists because this threat is assumed to be significant; prior work is cited in support, but the paper does not establish real-world incident frequency.
  • domain assumption Human-in-the-loop confirmation is the dominant deployed defense for CUAs (Introduction, Related Work).
    The paper cites one commercial example (Anthropic 2024) and some governance frameworks, then treats this as the baseline reality that the benchmark targets.
  • domain assumption An LLM prompted as a novice user (NPC2) reproduces real non-expert confirmation behavior (Human-in-the-Loop Analysis).
    Validated with only three non-expert participants; the simulated user may over-approve or under-approve in ways that change the HITL result.
  • ad hoc to paper Presence of intent is a valid proxy for attack success (Evaluation Metrics).
    This is the paper's own definition and is load-bearing for all reported ASR numbers; without it, the reported success rates would be lower.

pith-pipeline@v1.3.0-daily-deepseek · 11603 in / 11998 out tokens · 130320 ms · 2026-08-04T16:31:41.008266+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents." pith.science (2026). https://pith.science/paper/BK5D2KKD

@misc{pith2026260802018,
  author       = {Pith},
  title        = {Pith review of: Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BK5D2KKD}},
  note         = {Machine review of arXiv:2608.02018}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Computer-use agents (CUAs), which empower large language models to autonomously operate operating systems and the web, are increasingly vulnerable to indirect prompt injection attacks. A widely adopted defense is the human-in-the-loop paradigm, in which the agent pauses for explicit user confirmation before executing sensitive operations. While effective against conspicuously high-harm attacks, this defense offers little protection against what we term Invisible Ink Threats: low-harm injected goals, such as starring a repository or installing a package, that are behaviorally indistinguishable from legitimate task execution and thus evade both model safety mechanisms and human oversight. To systematically investigate this blind spot, we present II-Bench, a collection of seemingly harmless adversarial tasks. II-Bench comprises 444 examples targeting confidentiality and integrity attacks across three platforms, spanning three attack categories: page navigation and interaction, sensitive information exfiltration, and code download and execution. Each category is instantiated in both natural language and code forms under two levels of instruction specificity. Furthermore, we construct HITLCUA, a comprehensive adversarial testing framework that integrates a real virtual machine operating system environment with isolated Docker-based web platforms, and simulates human participation by allowing CUAs to consult an API-simulated user before proceeding with suspicious operations. Extensive evaluations of leading CUAs reveal that low-harm injections frequently bypass both agent defenses and simulated user review, exposing severe and previously underexplored security risks in current CUAs.

Figures

Figures reproduced from arXiv: 2608.02018 by Jia-Chen Zhang, Kai-Wei Zhang, Ze-Yu Zhang.

Figure 1
Figure 1. Figure 1: Under the same workflow architecture, strong at [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the HITLCUA Framework and II-Bench. The framework incorporates two NPCs: NPC1 simulates a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Schematic diagrams of the interfaces for different [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of different adversarial task results [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 8 linked inside Pith

  1. [2]

    arXiv:2504.11281

    The Obvious Invisible Threat: LLM-Powered GUI Agents’ Vulnerability to Fine-Print Injections. arXiv:2504.11281. Debenedetti, E.; Shumailov, I.; Fan, T.; and et al

    Pith/arXiv arXiv

  2. [3]

    arXiv:2503.18813

    De- feating Prompt Injections by Design. arXiv:2503.18813. Debenedetti, E.; Zhang, J.; and et al

    Pith/arXiv arXiv

  3. [6]

    arXiv:2606.13392

    MiniMax Sparse Attention. arXiv:2606.13392. Li, A.; Zhou, Y.; Raghuram, V. C.; and et al

    Pith/arXiv arXiv

  4. [7]

    arXiv:2502.08586

    Com- mercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks. arXiv:2502.08586. Li, Y.; He, J.; Zhou, X.; and et al

    Pith/arXiv arXiv

  5. [9]

    Ruan, Y.; Dong, H.; Wang, A.; and et al

    Qwen3.7: The Agent Frontier. Ruan, Y.; Dong, H.; Wang, A.; and et al. 2024a. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In The Twelfth International Conference on Learning Repre- sentations. Ruan, Y.; Dong, H.; Wang, A.; and et al. 2024b. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. InInternational Conference on Lea...

    Pith/arXiv arXiv 2024

  6. [10]

    Singh, A.; Fry, A.; Perelman, A.; and et al

    Progent: Se- curing AI Agents with Privilege Control.arXiv e-prints, arXiv:2504.11703. Singh, A.; Fry, A.; Perelman, A.; and et al

    Pith/arXiv arXiv

  7. [11]

    arXiv:2601.03267

    OpenAI GPT-5 System Card. arXiv:2601.03267. Team,V.;Hong,W.;Gu,X.;andetal.2026.GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents. arXiv:2604.26752. Xie, T.; Yuan, M.; Zhang, D.; and et al

    Pith/arXiv arXiv 2026

  8. [14]

    InFindings of the Association for Computational Linguistics: ACL 2024, 10471–10506

    In- jecAgent:BenchmarkingIndirectPromptInjectionsinTool- Integrated Large Language Model Agents. InFindings of the Association for Computational Linguistics: ACL 2024, 10471–10506. Bangkok, Thailand: Association for Compu- tational Linguistics. Zhang, J.-C.; Xiong, Y.-J.; Xia, C.-M.; and et al

    2024

  9. [15]

    InProceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 8387–8401

    Attacking Vision- Language Computer Agents via Pop-ups. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 8387–8401. Vienna,Austria:AssociationforComputationalLinguistics. ISBN 979-8-89176-251-0. Zheng,B.;Gou,B.;Kil,J.;andetal.2024.GPT-4V(ision)isa generalistwebagent,ifgrounded. InProceedi...

    2024

  10. [38]

    InFindings of the Association for Computa- tionalLinguistics:ACL2026,11986–11998.SanDiego,Cal- ifornia, United States: Association for Computational Lin- guistics

    Ying,Z.;Shao,Y.;Gan,J.;andetal.2026.SecureWebArena: AHolisticSecurityEvaluationBenchmarkforLVLM-based Web Agents. InFindings of the Association for Computa- tionalLinguistics:ACL2026,11986–11998.SanDiego,Cal- ifornia, United States: Association for Computational Lin- guistics. ISBN 979-8-89176-395-1. Zhan, Q.; Liang, Z.; Ying, Z.; and Kang, D

    2026

  11. [2022]

    InProceed- ings of the 30th ACM International Conference on Multime- dia,MM’22,3185–3194.NewYork,NY,USA:Association for Computing Machinery

    Online Deep Learning from Doubly-Streaming Data. InProceed- ings of the 30th ACM International Conference on Multime- dia,MM’22,3185–3194.NewYork,NY,USA:Association for Computing Machinery. ISBN 9781450392037. Liao,Z.;Jones,J.;Jiang,L.;andetal.2026. RedTeamCUA: RealisticAdversarialTestingofComputer-UseAgentsinHy- brid Web-OS Environments. arXiv:2505.21936...

    arXiv 2026

  12. [2023]

    InFindings of the Association for Computational Linguistics: EMNLP 2023, 2803–2821

    Understand- ing HTML with Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2023, 2803–2821. Singapore: Association for Computational Lin- guistics. Lai, X.; Xu, W.; Yang, Y.; and et al

    2023

  13. [2024]

    InAdvances in Neural Information Processing Systems, volume 37, 52040–52094

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. InAdvances in Neural Information Processing Systems, volume 37, 52040–52094. Curran Associates, Inc. Xu,F.F.;Song,Y.;Li,B.;andetal.2026.Theagentcompany: benchmarking llm agents on consequential real world tasks. Advances in Neural Information Processing Systems,

    2026

  14. [2025]

    arXiv:2504.14064

    DoomArena: A framework for Testing AI Agents Against Evolving Security Threats. arXiv:2504.14064. Chen, C.; Zhang, Z.; Guo, B.; and et al

    arXiv

  15. [2026]

    arXiv:2605.28116

    MIRAGE: Context-AwarePromptInjectionagainstMobileGUIAgents via User-Generated Content. arXiv:2605.28116. Gur, I.; Nachum, O.; Miao, Y.; and et al

    Pith/arXiv arXiv