{"id":"27615813-5ab7-46fb-9708-81eac3bbd462","arxiv_id":"2601.01241","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"MCP-SandboxScan safely executes MCP tools in WASI and detects external-to-sink data flows, but the abstract's claimed 100-repository evaluation is not present in the paper body.","lead":"This paper describes a WebAssembly-based sandbox that runs untrusted MCP tools and flags when external data (environment variables, files, HTTP intents) leaks into LLM-visible output fields. The advertised large-scale evaluation in the abstract is absent from the full text, which instead reports three small hand-crafted case studies and a microbenchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's large-scale evaluation claim (1,127 tools, 12 witnesses) is absent from the full text; headline evidence unsupported.","rationale":"The reader's strongest claim correctly identifies the missing corpus evaluation as a critical red flag, and I agree with the REJECT verdict. However, the reader's weakest_assumption focuses on the substring-matching detection semantics (Section IV-C/IV-D) as the central weakness. While that is a valid limitation and is honestly acknowledged in Section VI, the more load-bearing concern is the total absence of the advertised large-scale evaluation: the abstract claims 1,127 tools and 12 witnesses, but the full text's evaluation consists of three toy examples and a microbenchmark. If the corpus data existed but were merely omitted from the paper, the concern could be settled by inspecting the repository; if the data do not exist, the central claim of practical scalability collapses regardless of detection semantics. Thus I identify the manuscript's internal inconsistency as the primary concern, differing from the reader's weakest_assumption, and recommend UNCHANGED (REJECT) because the paper as submitted does not support its headline empirical evidence.","tokens_in":9427,"tokens_out":3571,"duration_ms":39806,"concrete_test":"Download the linked repository (https://anonymous.4open.science/r/MCP-SandboxScan-FFFB/) and inspect its contents for any corpus data, evaluation scripts, or result tables corresponding to the claimed 35/71 repositories, 1,127 tools, or 12 witnesses. Also search the full text (including appendices and supplementary material) for per-repository results. If no such artifacts exist, the abstract's large-scale evaluation claim is unsupported by the submitted manuscript.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of practical scalability rests on the abstract's quantitative results: \"SandScope completes shallow dynamic scans for 35 repositories ... recovers metadata for 1,127 tools across 71 repositories ... observes source-to-sink witnesses in 12.\" However, the full text (MCP-SandboxScan) contains no such corpus evaluation. Section V reports only three hand-crafted WASM case studies, a static-vs-dynamic baseline, and a micro-benchmark. No repository list, no per-tool results, and no data for the claimed 100-repo corpus appear anywhere in the manuscript. The title/abstract mismatch (SandScope vs. MCP-SandboxScan) further indicates the abstract is from a different version. Because the headline empirical evidence is absent, the paper's practical-value argument is unsupported as submitted. This is an internal inconsistency, not a mere disagreement with prior consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The full-text paper, titled \"MCP-SandboxScan,\" proposes a WebAssembly/WASI-based sandbox for executing untrusted MCP-style tools and generating auditable reports of external-input-to-LLM-sink flows. It models sources as environment variables, mounted file contents, and HTTP-fetch intent markers; extracts sinks from marker-based prompt lines and JSON prompt/message/tool-return fields; and links sources to sinks via snippet substring matching. The body evaluates this prototype on three hand-crafted WASM case studies, a two-scenario static-vs-dynamic comparison, and a single-instance micro-benchmark. The abstract supplied with the submission, however, describes a different artifact named \"SandScope\" and claims large-scale results — 35 dynamically scanned repositories, 1,127 tools across 71 repositories, 886 security-sensitive tools, and source-to-sink witnesses in 12 — none of which appear in the full text.","tokens_in":9612,"tokens_out":5822,"duration_ms":63995,"significance":"The core idea — a capability-restricted, observable execution layer for MCP tools with evidence-based source-to-sink reporting — is timely and relevant to the agent-security literature. The threat model (tool outputs re-entering the agent's prompt) is well motivated, the artifact is released, and the limitations are stated honestly in Section VI. If the large-scale evaluation claimed in the abstract were present and reproducible, the paper would make a useful practical contribution. As submitted, however, the body's evidence supports only a modest proof-of-concept claim: a prototype can surface simple, verbatim substring exposures in synthetic tools. The headline scalability results are absent, and the detection semantics are correlation-based rather than true flow tracking, so the stronger claims in the abstract are not established.","major_comments":[{"comment":"The submission's abstract reports: \"SandScope completes shallow dynamic scans for 35 repositories ... recovers metadata for 1,127 tools across 71 repositories, including 886 tools with security-sensitive declared capabilities ... observes source-to-sink witnesses in 12.\" The full text contains no such corpus evaluation. Section V describes only three hand-crafted WASM tools (Table I), a two-scenario static-vs-dynamic comparison (Table II), and a micro-benchmark with one instance per mode (Table III). There is no repository list, no per-tool results, and no data for the claimed 35-repository or 100-repository corpus. This unsupported quantitative summary is the only evidence for practical scalability and is load-bearing for the paper's central claim. The title mismatch (SandScope vs. MCP-SandboxScan) further indicates the abstract is from a different version.","section":"Abstract (metadata) vs. §V"},{"comment":"The evaluation does not exercise any real MCP server or third-party MCP tool. All three case studies are small W ASM fixtures constructed by the authors, the static-vs-dynamic comparison is qualitative rather than a measured baseline, and each micro-benchmark row is a single instance. No error rates, variance, or threshold sensitivity are estimated. Consequently the conclusion that the prototype is \"a practical baseline for early-stage tool safety assessment\" (§VIII) is not supported by the experiments in the body. At most, the current evaluation demonstrates that the scanner behaves as designed on the authors' own examples.","section":"§V-B and §VIII"},{"comment":"The \"flow\" records are produced by substring matching of candidate snippets derived from all environment variables and (size-capped) file contents against extracted sink strings. The sources are collected after execution and are not observed to be read by the tool; any coincidental substring occurrence is reported as a source-to-sink flow. This is string correlation, not information-flow tracking, as the paper itself acknowledges in §VI. The micro-benchmark's \"short-token collision FP risk\" row (Table III) demonstrates that the method can flag false positives. The abstract's phrase \"source-to-sink witnesses\" therefore materially overstates the strength of the evidence produced by the current matching semantics.","section":"§IV-D"},{"comment":"The abstract states that SandScope \"executes portable tools under WASI or drives unmodified MCP servers over standard input/output (stdio).\" The full text's design and evaluation cover only tools compiled to WASM/WASI; §IV-A describes each tool invocation as \"a single W ASM run,\" and no stdio driver for unmodified native MCP servers is described or evaluated anywhere in the body. Since deployed MCP servers are typically native processes rather than WASM modules, this claimed capability is material to the threat model and would need a concrete design and evaluation before it can support the stated generality.","section":"Abstract vs. §IV-A"}],"minor_comments":[{"comment":"The matching thresholds — minimum snippet length 4, mid-string window of 24 characters for sources longer than 48 characters, per-file size cap 64KB — are presented without justification or sensitivity analysis. Given that the central flow results depend on these constants, a brief sensitivity study would strengthen the paper.","section":"§IV-D"},{"comment":"The static-vs-dynamic baseline is reported as two qualitative rows (\"miss\" vs. \"denied evidence\"/\"flow detected\"). Reporting concrete scan outputs, detection counts, or a small corpus would make the comparison more informative.","section":"Table II"},{"comment":"The text contains typographical artifacts such as \"W ASM\"/\"W ASI\" spacing and \"protability\" (in §VII). These should be cleaned up in revision.","section":"Throughout"},{"comment":"The source-code reference points to an anonymous URL. For a journal submission, a stable, de-anonymized repository link (or an artifact appendix) would support reproducibility.","section":"Reference [9]"}],"recommendation":"reject","confidential_remarks":"The mismatch between the supplied abstract and the full text is severe: the abstract claims a large-scale empirical evaluation that the body does not contain, and even the artifact name differs. This is not a matter of framing; the headline quantitative results are absent and cannot be checked. I would encourage the authors to resubmit a version whose claims match the body — either by removing the unsupported corpus claims or, preferably, by actually reporting the 35-repository / 1,127-tool evaluation with repository lists and per-tool data. The core sandboxing and substring-flow idea is reasonable and the limitations are honestly stated, but in its current form the submission does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one has a genuine contribution buried under a stale abstract. The body describes MCP-SandboxScan, a WASM/WASI sandbox that runs untrusted MCP tools and links environmental inputs to LLM-visible sinks via substring evidence. That combination—sandboxed execution plus semantic sink extraction—is new relative to static scanners like MCPScan and prior WASM containment work, and it addresses a real gap: safe runtime analysis of untrusted MCP tools. The full-text evaluation is transparently modest: three hand-crafted tools, one static-vs-dynamic baseline, and a micro-benchmark that honestly characterizes false positives and false negatives. The code is released, which is a real plus.\n\nThe problem is the abstract. It advertises SandScope—not MCP-SandboxScan—and claims a 100-repository corpus with 35 dynamic scans, 1,127 tools, and 12 source-to-sink witnesses. None of that appears in the body. No repository list, no per-tool results, no numbers. The only evaluation is the three case studies. So the headline evidence for practical scalability is unsupported as submitted. This isn't a disagreement about interpretation; the abstract and the body describe different papers. A reader who skims the abstract will take away a claim the paper doesn't back.\n\nThe other soft spots are minor by comparison. Detection relies on substring matches of hand-chosen snippets, and the authors themselves acknowledge that encoding, compression, or output that never surfaces in stdout breaks the matching. That's an honest limitation, but it means the 'source-to-sink witness' finding is a lower bound, not a robust signal. The thresholds (4-char minimum, 24-char mid-string window) are sensible but ad hoc, and there's no sensitivity analysis. These are fixable.\n\nWould a serious editor send this out? I think yes. The underlying idea is sound and relevant to MCP supply-chain risk, the body is honest about scope, and the missing corpus evaluation could be added. But as submitted, the abstract overreaches badly. My recommendation: send it to peer review with a clear instruction that the evaluation must be reconciled with the abstract—either add the corpus study or rewrite the abstract to match the body's actual scope.","headline":"The prototype idea is real and the full text is honest, but the abstract claims a 100-repo evaluation the body doesn't contain, so the paper as submitted can't support its own headline.","tokens_in":10130,"tokens_out":3298,"would_cite":false,"duration_ms":32465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By running untrusted MCP tools inside a WebAssembly sandbox and matching external inputs against agent-visible output regions, this paper argues that scanners can surface auditable source-to-sink evidence of prompt-injection and exfiltratio","keywords":["MCP tool security","WebAssembly sandbox","WASI","prompt injection detection","source-to-sink flow analysis","dynamic analysis","LLM agent security","tool supply chain"],"falsifier":"Run the scanner on a tool that reads a secret from a mounted file, base64-encodes it, and emits it inside a JSON prompt field; the paper's own micro-benchmark predicts no flow will be reported. A single real-world tool that leaks this way while passing the scan would show that the substring-match assumption can yield false safety.","tokens_in":9294,"feed_emoji":"🛡️","tokens_out":7094,"duration_ms":57210,"temperature":0.7,"pith_summary":"Tool-augmented LLM agents install MCP tools like third-party packages, but a tool's output is often fed straight back into the agent's prompt — so a malicious tool can hijack reasoning or reflect secrets into LLM-visible fields. The paper argues that the missing primitive is safe execution: an audit framework should run untrusted tools under least privilege and observe what they actually emit, rather than only scanning binaries for static signatures. Its central claim is that a WebAssembly/WASI sandbox, combined with a string-level source-to-sink matcher over extracted prompt and tool-return sinks, produces auditable runtime evidence of prompt-injection and exfiltration risk that static scanners miss. If that holds, MCP tool registries and hosts get a deployable, evidence-based check that complements existing static screening.","feed_headline":"Sandboxed MCP tools yield source-to-sink leak evidence","feed_subtitle":"WASI sandboxes let scanners catch runtime leaks that static string scans miss, with per-flow proof.","key_machinery":"The carrying mechanism is the source-to-sink flow detector operating on post-run artifacts. After execution, the scanner derives candidate snippets from each external source — the full string when at least four characters, fixed-length prefixes for longer strings, and a mid-string window for strings longer than 48 characters — and tests whether any snippet occurs in the textual view of an extracted sink. A match emits a deduplicated FlowMatch record with a confidence label (high for full or long-window matches, low for short or prefix-only matches). This lightweight substring linking is what converts bounded stdout/stderr into auditable, explainable risk evidence; the WASI preopen boundary s","core_discovery":"The paper's central discovery is a mechanism, not a new vulnerability class: runtime-only behaviors that static scans cannot see — a tool that assembles a path at runtime, or reads a secret from a mounted file and prints it into a tool return — become observable when the tool is executed inside a capability-restricted WASM/WASI sandbox. MCP-SandboxScan extracts sinks from stdout using three rules (PROMPT: marker lines, JSON prompt/messages fields, and string leaves of tool-return JSON), collects sources from environment values, size-capped mounted file contents, and output-surfaced HTTP fetch intents, then links each source to each sink by checking whether a short snippet of the source occur","pith_inferences":["Editor's flag: the abstract's corpus-scale claims — 100 repositories, 35 dynamically scanned, 1,127 tools with metadata, 12 with observed source-to-sink witnesses — are not backed by any corresponding section in the full text, whose evaluation is limited to three hand-built WASM fixtures, one static-baseline comparison, and a micro-benchmark. The scale claim should be treated as unverified.","If the approach were extended to network-response bodies and tool-to-model API interception — avenues the authors name as future work — the same source-sink model could cover exfiltration through fetched content, which the current intent-only HTTP handling explicitly does not.","The snippet-matching strategy suggests a quantitative research program: a transformation-aware benchmark over real MCP servers measuring how often true leaks survive base64, compression, truncation, or prompt construction without printing; the paper's own table suggests these will be missed.","A natural product-level consequence the authors leave implicit: because flow records are already structured and confidence-labeled, the sandbox could be paired with a host-side guard that refuses or redacts any tool output containing a high-confidence source snippet, turning the scanner from an audit tool into a runtime control."],"forward_implications":["Tool registries and MCP hosts could run batch scans that produce per-flow evidence for every tool, making 'why is this flagged' answerable from the report itself.","Runtime-only attacks — obfuscated path access, file-derived secrets, dynamically assembled prompt payloads — become visible as either source-to-sink flows or capability denials, closing a gap that binary string-signature scanning leaves open.","The sandbox doubles as containment: because execution is capability-restricted and output-bounded, the act of scanning does not hand the untrusted tool a foothold on the host.","The machine-readable flow records are designed to feed downstream alerting or allow/deny decisions, turning the audit output into a practical input for host-side policy."],"fun_headline_variants":["WASM sandbox exposes MCP tool leaks static scans miss","Runtime sandboxing catches MCP tools' source-to-sink leaks","SandScope: sandboxed MCP tools reveal hidden data flows","WASI execution uncovers runtime-only MCP tool leaks","Executing MCP tools in sandbox finds leaks invisible to static"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a security-relevant leak will appear as an unchanged, substring-matchable fragment of an external source inside a stdout-derived sink; tools that encode, compress, obfuscate, or never print the data — channels the paper itself lists as limitations — are invisible to the detector, so the evidence is only as strong as that observability assumption.","fun_headline_variants_meta":{"raw":{"variants":["WASM sandbox exposes MCP tool leaks static scans miss","Runtime sandboxing catches MCP tools' source-to-sink leaks","SandScope: sandboxed MCP tools reveal hidden data flows","WASI execution uncovers runtime-only MCP tool leaks","Executing MCP tools in sandbox finds leaks invisible to static"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1224,"prompt_tokens":844,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":290}},"tokens_in":588,"tokens_out":380,"duration_ms":132720,"temperature":1.0,"reasoning_tokens":290,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:50:49.965810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the scanner on a tool that reads a secret from a mounted file, base64-encodes it, and emits it inside a JSON prompt field; the paper's own micro-benchmark predicts no flow will be reported. A single real-world tool that leaks this way while passing the scan would show that the substring-match assumption can yield false safety.","supporting_citations":[],"review_version":1}