Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

MCP-SandboxScan: WASM-based Secure Execution and Runtime Analysis for MCP Tools

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By running untrusted MCP tools inside a WebAssembly sandbox and matching external inputs against agent-visible output regions, this paper argues that scanners can surface auditable source-to-sink evidence of prompt-injection and exfiltratio

desk verdict The prototype idea is real and the full text is honest, but the abstract claims a 100-repo evaluation the body doesn't contain, so the paper as submitted can't support its own headline. read the letter →

arxiv 2601.01241 v2 pith:VJLUNUAJ submitted 2026-01-03 cs.CR cs.SE

Zhuoran Tan , Run Hao , Jeremy Singer , Yutian Tang , Christos Anagnostopoulos This is my paper · ORCID
classification cs.CRcs.SE
keywords MCPtoolsecurityWebAssemblysandboxWASIpromptinjectiondetectionsource-to-sinkflowanalysisdynamicLLMagentsupplychain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tool-augmented LLM agents install MCP tools like third-party packages, but a tool's output is often fed straight back into the agent's prompt — so a malicious tool can hijack reasoning or reflect secrets into LLM-visible fields. The paper argues that the missing primitive is safe execution: an audit framework should run untrusted tools under least privilege and observe what they actually emit, rather than only scanning binaries for static signatures. Its central claim is that a WebAssembly/WASI sandbox, combined with a string-level source-to-sink matcher over extracted prompt and tool-return sinks, produces auditable runtime evidence of prompt-injection and exfiltration risk that static scanners miss. If that holds, MCP tool registries and hosts get a deployable, evidence-based check that complements existing static screening.

What carries the argument

The carrying mechanism is the source-to-sink flow detector operating on post-run artifacts. After execution, the scanner derives candidate snippets from each external source — the full string when at least four characters, fixed-length prefixes for longer strings, and a mid-string window for strings longer than 48 characters — and tests whether any snippet occurs in the textual view of an extracted sink. A match emits a deduplicated FlowMatch record with a confidence label (high for full or long-window matches, low for short or prefix-only matches). This lightweight substring linking is what converts bounded stdout/stderr into auditable, explainable risk evidence; the WASI preopen boundary s

What would settle it

Run the scanner on a tool that reads a secret from a mounted file, base64-encodes it, and emits it inside a JSON prompt field; the paper's own micro-benchmark predicts no flow will be reported. A single real-world tool that leaks this way while passing the scan would show that the substring-match assumption can yield false safety.

Watch

Extended reading notes

Core claim

The paper's central discovery is a mechanism, not a new vulnerability class: runtime-only behaviors that static scans cannot see — a tool that assembles a path at runtime, or reads a secret from a mounted file and prints it into a tool return — become observable when the tool is executed inside a capability-restricted WASM/WASI sandbox. MCP-SandboxScan extracts sinks from stdout using three rules (PROMPT: marker lines, JSON prompt/messages fields, and string leaves of tool-return JSON), collects sources from environment values, size-capped mounted file contents, and output-surfaced HTTP fetch intents, then links each source to each sink by checking whether a short snippet of the source occur

Load-bearing premise

The load-bearing premise is that a security-relevant leak will appear as an unchanged, substring-matchable fragment of an external source inside a stdout-derived sink; tools that encode, compress, obfuscate, or never print the data — channels the paper itself lists as limitations — are invisible to the detector, so the evidence is only as strong as that observability assumption.

Editorial extensions

If this is right

  • Tool registries and MCP hosts could run batch scans that produce per-flow evidence for every tool, making 'why is this flagged' answerable from the report itself.
  • Runtime-only attacks — obfuscated path access, file-derived secrets, dynamically assembled prompt payloads — become visible as either source-to-sink flows or capability denials, closing a gap that binary string-signature scanning leaves open.
  • The sandbox doubles as containment: because execution is capability-restricted and output-bounded, the act of scanning does not hand the untrusted tool a foothold on the host.
  • The machine-readable flow records are designed to feed downstream alerting or allow/deny decisions, turning the audit output into a practical input for host-side policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's flag: the abstract's corpus-scale claims — 100 repositories, 35 dynamically scanned, 1,127 tools with metadata, 12 with observed source-to-sink witnesses — are not backed by any corresponding section in the full text, whose evaluation is limited to three hand-built WASM fixtures, one static-baseline comparison, and a micro-benchmark. The scale claim should be treated as unverified.
  • If the approach were extended to network-response bodies and tool-to-model API interception — avenues the authors name as future work — the same source-sink model could cover exfiltration through fetched content, which the current intent-only HTTP handling explicitly does not.
  • The snippet-matching strategy suggests a quantitative research program: a transformation-aware benchmark over real MCP servers measuring how often true leaks survive base64, compression, truncation, or prompt construction without printing; the paper's own table suggests these will be missed.
  • A natural product-level consequence the authors leave implicit: because flow records are already structured and confidence-labeled, the sandbox could be paired with a host-side guard that refuses or redacts any tool output containing a high-confidence source snippet, turning the scanner from an audit tool into a runtime control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The full-text paper, titled "MCP-SandboxScan," proposes a WebAssembly/WASI-based sandbox for executing untrusted MCP-style tools and generating auditable reports of external-input-to-LLM-sink flows. It models sources as environment variables, mounted file contents, and HTTP-fetch intent markers; extracts sinks from marker-based prompt lines and JSON prompt/message/tool-return fields; and links sources to sinks via snippet substring matching. The body evaluates this prototype on three hand-crafted WASM case studies, a two-scenario static-vs-dynamic comparison, and a single-instance micro-benchmark. The abstract supplied with the submission, however, describes a different artifact named "SandScope" and claims large-scale results — 35 dynamically scanned repositories, 1,127 tools across 71 repositories, 886 security-sensitive tools, and source-to-sink witnesses in 12 — none of which appear in the full text.

Significance. The core idea — a capability-restricted, observable execution layer for MCP tools with evidence-based source-to-sink reporting — is timely and relevant to the agent-security literature. The threat model (tool outputs re-entering the agent's prompt) is well motivated, the artifact is released, and the limitations are stated honestly in Section VI. If the large-scale evaluation claimed in the abstract were present and reproducible, the paper would make a useful practical contribution. As submitted, however, the body's evidence supports only a modest proof-of-concept claim: a prototype can surface simple, verbatim substring exposures in synthetic tools. The headline scalability results are absent, and the detection semantics are correlation-based rather than true flow tracking, so the stronger claims in the abstract are not established.

major comments (4)
  1. [Abstract (metadata) vs. §V] The submission's abstract reports: "SandScope completes shallow dynamic scans for 35 repositories ... recovers metadata for 1,127 tools across 71 repositories, including 886 tools with security-sensitive declared capabilities ... observes source-to-sink witnesses in 12." The full text contains no such corpus evaluation. Section V describes only three hand-crafted WASM tools (Table I), a two-scenario static-vs-dynamic comparison (Table II), and a micro-benchmark with one instance per mode (Table III). There is no repository list, no per-tool results, and no data for the claimed 35-repository or 100-repository corpus. This unsupported quantitative summary is the only evidence for practical scalability and is load-bearing for the paper's central claim. The title mismatch (SandScope vs. MCP-SandboxScan) further indicates the abstract is from a different version.
  2. [§V-B and §VIII] The evaluation does not exercise any real MCP server or third-party MCP tool. All three case studies are small W ASM fixtures constructed by the authors, the static-vs-dynamic comparison is qualitative rather than a measured baseline, and each micro-benchmark row is a single instance. No error rates, variance, or threshold sensitivity are estimated. Consequently the conclusion that the prototype is "a practical baseline for early-stage tool safety assessment" (§VIII) is not supported by the experiments in the body. At most, the current evaluation demonstrates that the scanner behaves as designed on the authors' own examples.
  3. [§IV-D] The "flow" records are produced by substring matching of candidate snippets derived from all environment variables and (size-capped) file contents against extracted sink strings. The sources are collected after execution and are not observed to be read by the tool; any coincidental substring occurrence is reported as a source-to-sink flow. This is string correlation, not information-flow tracking, as the paper itself acknowledges in §VI. The micro-benchmark's "short-token collision FP risk" row (Table III) demonstrates that the method can flag false positives. The abstract's phrase "source-to-sink witnesses" therefore materially overstates the strength of the evidence produced by the current matching semantics.
  4. [Abstract vs. §IV-A] The abstract states that SandScope "executes portable tools under WASI or drives unmodified MCP servers over standard input/output (stdio)." The full text's design and evaluation cover only tools compiled to WASM/WASI; §IV-A describes each tool invocation as "a single W ASM run," and no stdio driver for unmodified native MCP servers is described or evaluated anywhere in the body. Since deployed MCP servers are typically native processes rather than WASM modules, this claimed capability is material to the threat model and would need a concrete design and evaluation before it can support the stated generality.
minor comments (4)
  1. [§IV-D] The matching thresholds — minimum snippet length 4, mid-string window of 24 characters for sources longer than 48 characters, per-file size cap 64KB — are presented without justification or sensitivity analysis. Given that the central flow results depend on these constants, a brief sensitivity study would strengthen the paper.
  2. [Table II] The static-vs-dynamic baseline is reported as two qualitative rows ("miss" vs. "denied evidence"/"flow detected"). Reporting concrete scan outputs, detection counts, or a small corpus would make the comparison more informative.
  3. [Throughout] The text contains typographical artifacts such as "W ASM"/"W ASI" spacing and "protability" (in §VII). These should be cleaned up in revision.
  4. [Reference [9]] The source-code reference points to an anonymous URL. For a journal submission, a stable, de-anonymized repository link (or an artifact appendix) would support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: flows are defined by substring matching and reported as scanner outputs; the abstract's large-scale corpus claim is absent from the full text but that is an evidentiary inconsistency, not a circular derivation.

full rationale

The paper is an engineering/system paper, not a derivation or prediction exercise. Its central detector is operational: sources are constructed from environment values, mounted file contents, and HTTP intent markers; sinks are extracted from stdout using PROMPT markers and JSON prompt/messages/tool-return fields; a flow is emitted exactly when a generated source snippet appears in a sink's textual view (§IV-D). This is an explicit definition of the detection criterion, so the reported flows are scanner outputs, not a result that has been fitted to or derived from the target outcome. The paper does not claim a first-principles theorem, does not fit parameters to a headline result, and openly characterizes false positives/negatives in Table III and §VI. There is no load-bearing self-citation: the references are external prior work, and the anonymous reproducibility link [9] is not used to justify the framework's validity. The abstract asserts a 100-repository corpus with 1,127 tools and 12 witnesses, but the full text reports only three hand-crafted case studies, a static-vs-dynamic baseline, and a micro-benchmark. That is a serious internal inconsistency / unsupported-evidence problem, but it is not circularity: no argument in the full text reduces to its own inputs. The paper's stated limitations (substring matching breaks under encoding/compression; sinks only from stdout; network fetch intent-only) are acknowledged, and they weaken the security claim without creating a circular derivation. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical or conceptual entities; it defines analytical categories (sources, sinks, flows, fetch intents) that are operationalized as string collections. The main load-bearing assumptions are domain assumptions about where LLM-relevant data appears and which environmental channels count as sources. The hand-tuned thresholds are free parameters affecting FP/FN behavior but are not fitted to produce a target result.

free parameters (4)
  • minimum snippet match length = 4 characters
    Section IV-D: snippets shorter than 4 characters are dropped; this threshold directly controls the false-positive short-token collision case in Table III.
  • mid-string window size and source-length trigger = 24-character window for sources longer than 48 characters
    Section IV-D: long sources get a mid-string window to catch partial leakage; the exact 24/48 split is hand-chosen and affects partial-match detection.
  • per-file source size cap = 64 KB
    Section V-A: mounted file contents are capped at 64KB per file when constructing source candidates; larger files are only partially represented, which can cause false negatives.
  • sink metadata path exclusion set = excludes paths like status/id
    Section IV-C: JSON tool-return leaves are filtered to reduce noise; the specific excluded paths are chosen by hand and change which sinks are considered.
assumptions (5)
  • domain assumption Tool outputs are incorporated into the agent's next prompt (e.g., via 'Tool said: ...' templates).
    Section II-A: this is the central premise that makes stdout sinks high-leverage. If an MCP host never feeds tool output into model reasoning, the detected flows would not matter.
  • domain assumption WASI preopened-directory enforcement is a sufficient containment boundary for executing untrusted tools.
    Sections III-IV: the safety of the scanner itself depends on WASM/WASI capability enforcement; any escape from the sandbox would invalidate the secure-execution claim.
  • domain assumption LLM-visible sinks are observable in stdout via PROMPT: markers and JSON prompt/messages fields.
    Section IV-C: sinks are extracted only from standard output. Tools that construct prompts internally, use alternative schemas, or emit sinks only under specific conditions are invisible.
  • domain assumption External inputs are limited to environment variables, mounted files, and output-surfaced HTTP fetch intents.
    Section II-B: network response bodies, host stdin, and other ambient channels are not modeled, so the flow model is incomplete by construction.
  • domain assumption A substring occurrence of a source snippet in a sink is evidence of external-to-sink exposure.
    Section IV-D and VI: matching is string-level; short snippets can collide in benign output (false positives), and encoded or transformed data will not match (false negatives).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MCP-SandboxScan: WASM-based Secure Execution and Runtime Analysis for MCP Tools." pith.science (2026). https://pith.science/paper/VJLUNUAJ

@misc{pith2026260101241,
  author       = {Pith},
  title        = {Pith review of: MCP-SandboxScan: WASM-based Secure Execution and Runtime Analysis for MCP Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJLUNUAJ}},
  note         = {Machine review of arXiv:2601.01241}
}
read the original abstract

Tool-augmented Large Language Model (LLM) agents create a new supply-chain surface: Model Context Protocol (MCP) tools are installed like third-party packages, yet their outputs can enter the agent's reasoning context. This enables confused-deputy risks in which attacker-controlled inputs cause otherwise benign tools to exercise legitimate authority over files, environment variables, or network-facing operations and reflect sensitive or instruction-like content into LLM-visible fields. We present SandScope, an MCP-aware audit framework that combines runtime witness detection with semantic tool profiling. SandScope executes portable tools under WebAssembly System Interface (WASI) or drives unmodified MCP servers over standard input/output (stdio), extracts LLM-visible sinks from tool results and prompt/message fields, and reports auditable source-to-sink witnesses from environment, file, and tool-input sources while separately recording network-intent and egress evidence. Its semantic layer recovers declared capabilities from tools/list metadata and static registrations to characterize attack surface when execution is incomplete. We evaluate SandScope on controlled cross-language subjects, an evasion benchmark, and a 100-repository MCP corpus. SandScope completes shallow dynamic scans for 35 repositories and, through a broader semantic profiling pass, recovers metadata for 1,127 tools across 71 repositories, including 886 tools with security-sensitive declared capabilities. A schema-guided exploration pass over the 35 dynamically scanned repositories re-executes 33 and observes source-to-sink witnesses in 12. These results show that SandScope provides practical, auditable evidence for MCP tool risk through controlled execution, MCP-aware sink extraction, runtime witness reporting, and semantic capability profiling.

Figures

Figures reproduced from arXiv: 2601.01241 by the authors.

Figure 1
Figure 1. System Overview with Threat Model (T: Threats, C: Controls) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FlowGuard: From Signals to Evidence for MCP Security Detection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    FlowGuard detects MCP tool vulnerabilities by verifying runtime evidence rather than treating suspicious metadata or responses as proof.

  2. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

    cs.CR 2026-06 unverdicted novelty 3.0 of 10

    A synthesis of 247 papers on LLM agent security identifies prompt injection and tool hijacking as dominant threats, notes weakly compositional defenses, and argues for trust boundaries and realistic evaluations.

Reference graph

Works this paper leans on

20 extracted references · 5 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Introducing the model context protocol

    Anthropic. Introducing the model context protocol. Anthropic. [Online]. Available: https://www.anthropic.com/news/model-context-protocol

  2. [2]

    Agent based modelling and simulation tools: A review of the state-of-art software,

    S. Abar, G. K. Theodoropoulos, P. Lemarinier, and G. M. O’Hare, “Agent based modelling and simulation tools: A review of the state-of-art software,”Computer Science Review, vol. 24, pp. 13–33,

  3. [3]

    Mcp safety audit: Llms with the model context protocol allow major security exploits,

    B. Radosevich and J. Halloran, “Mcp safety audit: Llms with the model context protocol allow major security exploits,” 2025. [Online]. Available: https://arxiv.org/abs/2504.03767

    arXiv 2025

  4. [4]

    MCIP: Protecting MCP safety via model contextual integrity protocol,

    H. Jing, H. Li, W. Hu, Q. Hu, X. Heli, T. Chu, P. Hu, and Y . Song, “MCIP: Protecting MCP safety via model contextual integrity protocol,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov...

    2025

  5. [5]

    Beyond the protocol: Unveiling attack vectors in the model context protocol (mcp) ecosystem,

    H. Song, Y . Shen, W. Luo, L. Guo, T. Chen, J. Wang, B. Li, X. Zhang, and J. Chen, “Beyond the protocol: Unveiling attack vectors in the model context protocol (mcp) ecosystem,” 2025. [Online]. Available: https://arxiv.org/abs/2506.02040

    arXiv 2025

  6. [6]

    Mcpscan,

    Z. Sha, H. Li, C. Chen, R. Xiong, S. Cui, C. Meng, and W. Wang, “Mcpscan,” 2025. [Online]. Available: https://github.com/antgroup/Trus tworthy LM/mcp-scan

    2025

  7. [7]

    Scansat: Unlocking static and dynamic scan obfuscation,

    L. Alrahis, M. Yasin, N. Limaye, H. Saleh, B. Mohammad, M. Al- Qutayri, and O. Sinanoglu, “Scansat: Unlocking static and dynamic scan obfuscation,”IEEE Transactions on Emerging Topics in Computing, vol. 9, no. 4, pp. 1867–1882, 2021

    2021

  8. [8]

    Wave: a verifiably secure webassembly sandboxing runtime,

    E. Johnson, E. Laufer, Z. Zhao, D. Gohman, S. Narayan, S. Savage, D. Stefan, and F. Brown, “Wave: a verifiably secure webassembly sandboxing runtime,” in2023 IEEE Symposium on Security and Privacy (SP), 2023, pp. 2940–2955

    2023

Show all 20 references
  1. [9]

    Mcp-sandboxscan (anonymous github repository),

    Anonymous, “Mcp-sandboxscan (anonymous github repository),” https: //anonymous.4open.science/r/MCP-SandboxScan-FFFB/README.md, 2025, accessed: 2025-12-31

    2025

  2. [10]

    Mcptox: A benchmark for tool poisoning attack on real-world mcp servers,

    Z. Wang, Y . Gao, Y . Wang, S. Liu, H. Sun, H. Cheng, G. Shi, H. Du, and X. Li, “Mcptox: A benchmark for tool poisoning attack on real-world mcp servers,” 2025. [Online]. Available: https://arxiv.org/abs/2508.14925

    2025 arXiv

  3. [11]

    (2025, May) Mcp: Untrusted servers and confused clients, plus a sneaky exploit

    Embrace The Red. (2025, May) Mcp: Untrusted servers and confused clients, plus a sneaky exploit. Accessed: 2025-08-31. [Online]. Available: https://embracethered.com/blog/posts/2025/model-context-p rotocol-security-risks-and-exploits/

    2025

  4. [12]

    Model context protocol (mcp): Landscape, security threats, and future research directions,

    X. Hou, Y . Zhao, S. Wang, and H. Wang, “Model context protocol (mcp): Landscape, security threats, and future research directions,”

  5. [13]

    Defeating prompt injections by design,

    E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tram `er, “Defeating prompt injections by design,” March 2025

    2025

  6. [14]

    toxic flows,

    L. Beurer-Kellner, M. Milanta, and M. Fischer. (2025, Jul.) Invariant labs exposes novel prompt injection attack vulnerabilities, “toxic flows,” in agentic systems & mcp servers. Accessed: 2025-10-06. [Online]. Available: https://invariantlabs.ai/blog/toxic-flow-analysis

    2025

  7. [15]

    Cheatsheet – a practical guide for securely using third-party mcp servers 1.0,

    OW ASP GenAI Security Project, “Cheatsheet – a practical guide for securely using third-party mcp servers 1.0,” Nov. 2025, version 1.0; accessed via https://genai.owasp.org/resource/cheatsheet- a-practical-guide-for-securely-using-third-party-mcp-servers-1-0/. [On- line]. Avai...

    2025

  8. [16]

    Wasmbox: A lightweight wasm-based runtime for trustworthy multi-tenant embedded systems,

    L. Coppolino, S. D’Antonio, G. Mazzeo, R. Nardone, L. Romano, and M. Schmitt, “Wasmbox: A lightweight wasm-based runtime for trustworthy multi-tenant embedded systems,”IEEE Transactions on Emerging Topics in Computing, vol. 13, no. 2, pp. 467–480, 2025

    2025

  9. [17]

    Sledge: a serverless-first, light-weight wasm runtime for the edge,

    P. K. Gadepalli, S. McBride, G. Peach, L. Cherkasova, and G. Parmer, “Sledge: a serverless-first, light-weight wasm runtime for the edge,” inProceedings of the 21st International Middleware Conference, ser. Middleware ’20. New York, NY , USA: Association for Computing Machiner...

    2020

  10. [18]

    Mcp guardian: A security-first layer for safeguarding mcp-based ai system,

    S. Kumar, A. Girdhar, R. Patil, and D. Tripathi, “Mcp guardian: A security-first layer for safeguarding mcp-based ai system,” 2025. [Online]. Available: https://arxiv.org/abs/2504.12757

    2025 arXiv

  11. [2017]

    Available: https://www.sciencedirect.com/science/articl e/pii/S1574013716301198

    [Online]. Available: https://www.sciencedirect.com/science/articl e/pii/S1574013716301198

  12. [2025]

    Available: https://arxiv.org/abs/2503.23278

    [Online]. Available: https://arxiv.org/abs/2503.23278

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.