{"id":"0c13f96e-8d26-47ed-84d0-46fe7b7a0ec8","arxiv_id":"2605.19192","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Evidence-carrying multimodal agents decompose tool calls into predicates, obtain certificates from DOM/OCR/AX verifiers, and use a deterministic gate to authorize actions only when certificates support them, achieving zero unsafe executions in tested tasks.","lead":"This paper formalizes how hallucinations in multimodal AI agents can enable unauthorized tool actions and introduces evidence-carrying agents that require verified certificates before executing privileged operations. Smart generalists and scientists may read it to see a concrete architecture for turning perceptual errors into auditable security controls rather than relying on model accuracy alone.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Verifier hardening against 17 categories does not establish that certificates cannot be bypassed or that predicates cover all safety preconditions","rationale":"The reader's weakest_assumption already isolates the verifier accuracy + predicate sufficiency point; the abstract-only limitation makes this the clearest place where the central claim could fail without contradicting the reported numbers on the tested set.","tokens_in":1845,"tokens_out":290,"duration_ms":47229,"concrete_test":"Re-run the full 200 end-to-end task suite plus the 120 browser tasks after adding 20 new attack variants (generated independently from the original 17 categories) to the verifier red-teaming; if any new bypass produces an unsafe execution under the same HACR audit protocol, the zero-unsafe claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The zero-unsafe result on 200+120 tasks rests on two linked claims: (1) the four hardening steps make the constrained DOM/OCR/AX verifiers produce certificates that accurately reflect content and resist all bypasses, and (2) the action-critical predicate decomposition is both necessary and sufficient. The reported 0/1700 bypass rate and Wilson bounds only address the tested 17 categories; nothing in the abstract or described experiments rules out novel attacks, dynamic content, or missing preconditions that would let an unsafe action pass the deterministic gate even with correct-looking certificates.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes Evidence-Carrying Multimodal Agents (ECA) to mitigate hallucination-to-action conversion, where unsupported perceptual claims enable unsafe tool calls. ECA decomposes each tool call into action-critical predicates, obtains typed certificates from hardened constrained DOM/OCR/AX verifiers, and authorizes actions only via a deterministic gate based on those certificates. It reports zero unsafe executions across 200 end-to-end tasks (Wilson 95% upper bound 2.67%) and 120 browser tasks (upper bound 4.3%), a 0/1,700 bypass rate for 17 canonical attack categories after four hardening steps, and contrasts this with 100% and 49.6% unsafe rates for naive agents and prompt-only defenses in a HACR audit on 500 tasks. Oracle-certificate replay over 7,488 traces isolates gate correctness from model outputs.","tokens_in":2002,"tokens_out":589,"duration_ms":34049,"significance":"If the results hold, this provides a concrete mechanism for grounding multimodal agent actions in verifiable certificates rather than model beliefs, with clear empirical separation from baselines via statistical bounds and oracle isolation. The approach converts perception errors into auditable residuals and demonstrates practical safety gains on end-to-end tasks. Strengths include the use of Wilson bounds for upper limits on failure rates and the replay experiment for isolating the gate component.","major_comments":[{"comment":"Verifier red-teaming across 17 canonical attack categories: the 0/1,700 bypass rate and Wilson upper bound of 0.22% after the four hardening steps only covers the tested categories. The manuscript provides no analysis or experiments addressing novel attacks, adaptive adversaries, dynamic content, or other bypass vectors that could produce certificates allowing unsafe actions to pass the deterministic gate. This assumption is load-bearing for the zero-unsafe-execution claims on the 200 end-to-end and 120 browser tasks.","section":"Verifier red-teaming section"},{"comment":"Action-critical predicate decomposition: the paper treats the decomposition as both necessary and sufficient for capturing all safety-relevant preconditions, yet offers no formal coverage argument, completeness proof, or exhaustive enumeration showing that every possible unsafe precondition is represented by these predicates. If any precondition is omitted, accurate certificates could still authorize unsafe actions.","section":"Action-critical predicates description"}],"minor_comments":[{"comment":"The abstract references 'GPT-5.4 traces'; clarify whether this denotes a specific model release or is a descriptive placeholder for the replay experiment.","section":"Abstract"},{"comment":"Figure captions for the HACR audit and oracle replay results could more explicitly state the threat model and isolation guarantees to aid reader interpretation.","section":"Results figures"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and the recommendation for major revision. The comments highlight important considerations regarding the scope of our security evaluation and the coverage of the predicate decomposition. We respond to each point below with clarifications and proposed revisions that strengthen the presentation of our results without overstating the claims.","responses":[{"response":"We agree that the red-teaming results apply specifically to the 17 canonical attack categories we defined and tested. The 0/1,700 bypass rate and associated Wilson bound demonstrate effective mitigation for those vectors following the four hardening steps. We do not claim or imply resistance to arbitrary novel attacks, adaptive adversaries, or untested dynamic content, as such guarantees would require a different formal security model. The end-to-end zero-unsafe-execution results remain empirical observations under the evaluated threat model and task distribution, with the reported statistical bounds. We will add a dedicated limitations subsection that explicitly discusses these boundaries, notes the potential for adaptive bypasses, and suggests avenues for ongoing red-teaming. This is a partial revision.","revision_made":"partial","referee_comment":"[Verifier red-teaming section] Verifier red-teaming across 17 canonical attack categories: the 0/1,700 bypass rate and Wilson upper bound of 0.22% after the four hardening steps only covers the tested categories. The manuscript provides no analysis or experiments addressing novel attacks, adaptive adversaries, dynamic content, or other bypass vectors that could produce certificates allowing unsafe actions to pass the deterministic gate. This assumption is load-bearing for the zero-unsafe-execution claims on the 200 end-to-end and 120 browser tasks."},{"response":"The predicates are obtained by inspecting the safety-relevant preconditions of each tool API within the concrete task environments we study. We do not assert a formal completeness proof or exhaustive enumeration, as the space of possible unsafe preconditions is open-ended in general multimodal settings. Instead, the approach relies on making any coverage gaps visible through verifier residuals and schema constraints, which are then audited in the HACR evaluation. The empirical results show that the selected predicates block the unsafe executions that occur under naive and prompt-only baselines. We will revise the manuscript to state clearly that the predicates are task-derived rather than universally complete and to include a concrete example of predicate construction for one representative task. This is a full revision to the relevant description and discussion sections.","revision_made":"yes","referee_comment":"[Action-critical predicates description] Action-critical predicate decomposition: the paper treats the decomposition as both necessary and sufficient for capturing all safety-relevant preconditions, yet offers no formal coverage argument, completeness proof, or exhaustive enumeration showing that every possible unsafe precondition is represented by these predicates. If any precondition is omitted, accurate certificates could still authorize unsafe actions."}],"tokens_in":1580,"tokens_out":591,"duration_ms":28621,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Colleague, the core takeaway is that this paper gives a workable architecture for multimodal agents: instead of letting model text authorize tool calls, it decomposes each call into action-critical predicates, pulls typed certificates from hardened DOM/OCR/AX verifiers, and enforces authorization through a deterministic gate. That converts perceptual errors into something you can inspect at the verifier, schema, or implementation layer rather than an opaque authorization breach. They back it with concrete numbers—zero unsafe executions on 200 end-to-end tasks and 120 browser tasks, plus Wilson bounds, and an oracle replay over 7,488 traces that isolates gate correctness from model outputs. The HACR audit also shows clear separation from naive agents and prompt-only defenses on unsupported claims. The red-teaming with four hardening steps yielding 0/1,700 bypasses on 17 categories is a solid piece of engineering evidence. The soft spot is exactly the one the stress-test flags: those 17 categories and the resulting bounds do not establish resistance to novel attacks, dynamic content changes, or cases where the predicate set misses a safety precondition. The verifiers could still be bypassed in ways not covered by the tested set, and the paper does not demonstrate that the decomposition is sufficient for all real-world preconditions. This is aimed at researchers building tool-using multimodal agents who need practical safety controls. Readers working on agent reliability or verifiable execution would find the architecture and the isolation experiments useful. It has enough formalization plus reproducible-style empirical grounding to deserve a serious referee. I would send it to peer review; the idea is worth checking even if the attack coverage needs expansion.","headline":"The ECA design turns model hallucinations into auditable verifier residuals via predicate decomposition and deterministic gates, but its security rests on hardening tested only against 17 known categories.","tokens_in":2488,"tokens_out":402,"would_cite":false,"duration_ms":31992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"ECA predicate certificates and deterministic gate operate in AI agent security with no structural overlap to RS forcing chain or J-cost machinery","alignment":"orthogonal","rationale":"The paper's core machinery (action-critical predicate decomposition, typed DOM/OCR/AX certificates, deterministic gate Π(a,E), HACR metric, and verifier hardening against 17 attack categories) addresses hallucination-to-action conversion in multimodal agents. This has no connection to the RS framework's single-distinction forcing chain, reciprocal cost J(x), golden-ratio ladder, 8-tick periodicity, or parameter-free derivations of c/ℏ/G. No ratio-symmetric cost, φ-identities, or recognition-ladder structure appears; the domains are disjoint.","tokens_in":59889,"confidence":"high","tokens_out":166,"duration_ms":8866,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Multimodal agents eliminate unsafe tool calls by authorizing actions only with certificates from content verifiers.","keywords":["multimodal agents","hallucination","tool use safety","evidence certificates","verifiers","action authorization","agent security","DOM verification"],"falsifier":"Discovery of even one unsafe tool execution by an ECA agent on a task containing a hallucinated action-critical claim, or a successful bypass of the hardened verifiers by any of the 17 attack categories.","tokens_in":2745,"feed_emoji":"🛡️","tokens_out":722,"duration_ms":45036,"temperature":0.7,"pith_summary":"The paper establishes that hallucinations become safety failures when an agent's model text supplies a false precondition for a privileged tool call. It introduces evidence-carrying agents that reject free-form model output as evidence, break each proposed action into critical predicates, and require typed certificates from constrained DOM, OCR, and AX verifiers before any authorization occurs. A deterministic gate then permits only the privileges the certificates explicitly support. Tests across hundreds of tasks show this design produces no unsafe executions, whereas naive agents and prompt-only approaches allow unsafe actions at rates of 100 percent and 49.6 percent respectively when unsupported claims appear. A reader would care because the method converts an opaque model belief into an auditable residual that can be checked independently of the model's correctness.","feed_headline":"Certified predicates stop hallucination exploits in agents","feed_subtitle":"By requiring verifier certificates instead of model text for every tool call, agents record zero unsafe executions where naive systems fail.","key_machinery":"Evidence-carrying multimodal agents (ECA), which decompose each tool call into action-critical predicates, obtain typed certificates from constrained verifiers, and apply a deterministic gate that authorizes privileges only when those certificates are present.","core_discovery":"The central claim is that hallucination-to-action conversion can be prevented by treating model language as inadmissible for authorization and instead requiring deterministic certificates from hardened verifiers for every action-critical predicate. With content-derived certificates the system records zero unsafe executions on 200 end-to-end tasks and 120 browser tasks, while the same unsupported claims trigger unsafe behavior in every naive agent and in nearly half of prompt-only cases. Verifier red-teaming across 17 attack categories confirms that four hardening steps are each necessary to reach a bypass rate of zero in 1,700 trials.","pith_inferences":["The same certificate requirement could be applied to other perception sources such as video frames or API responses in non-browser agents.","Agent platforms may need to expose standardized verifiable interfaces for every tool to support this authorization model.","The separation of proposal by the model from authorization by certificates reduces dependence on model alignment alone for safety."],"forward_implications":["Naive agents reach 100 percent unsafe execution when given unsupported action-critical claims.","Prompt-only defenses still allow unsafe execution in 49.6 percent of cases under the same claims.","ECA records zero unsafe executions on 200 end-to-end tasks and 120 browser tasks.","After four hardening steps, canonical gate bypass drops to zero across 1,700 verifier red-team trials."],"fun_headline_variants":["Evidence certificates authorize every agent tool call","Model claims inadmissible for authorizing privileged actions","Verifiers issue certificates to block hallucination exploits","Action-critical predicates need typed verifier proof","Hardening steps ensure zero unsafe executions in agents"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The hardened DOM, OCR, and AX verifiers always produce accurate certificates that reflect actual content and cannot be bypassed, while the chosen predicates cover every safety-relevant precondition.","fun_headline_variants_meta":{"raw":{"variants":["Evidence certificates authorize every agent tool call","Model claims inadmissible for authorizing privileged actions","Verifiers issue certificates to block hallucination exploits","Action-critical predicates need typed verifier proof","Hardening steps ensure zero unsafe executions in agents"]},"model":"grok-4.3","cost_usd":0.006498,"raw_usage":{"total_tokens":3018,"prompt_tokens":784,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":64978000,"prompt_tokens_details":{"text_tokens":784,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2170,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":784,"tokens_out":64,"duration_ms":29918,"temperature":1.0,"reasoning_tokens":2170,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T08:53:22.462684+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Discovery of even one unsafe tool execution by an ECA agent on a task containing a hallucinated action-critical claim, or a successful bypass of the hardened verifiers by any of the 17 attack categories.","supporting_citations":[],"review_version":2}