{"id":"dc87d9e7-fa1c-4c67-864c-7fa4db72953b","arxiv_id":"2608.05495","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In 19 smart-home scenarios, traditional detectors and multimodal LLMs fail in opposite, complementary ways, so an untested oracle over both reaches 94.1% accuracy versus 76.5% for the best single layer.","lead":"A new pilot benchmark tests whether smart-home agents can tell a real command from ambient content like TV speech or on-screen text. It finds traditional detectors act on everything while multimodal language models refuse almost everything, and argues the two should be combined by routing rather than replacing one with the other.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"L0 is an idealized always-execute proxy, so Table 4's disjoint-correct-sets oracle is guaranteed by construction rather than measured; a real detector that misses any action-required case drops the 94.1% bound.","rationale":"The reader identified the L0 proxy as the weakest assumption, and I agree: it is the load-bearing point for the central oracle claim. The paper is honest that the oracle is an upper bound, but the oracle's value is not merely an upper bound on an empirical effect; it is an upper bound on a construction where L0 is annotated to execute all 17 scenarios. The disjointness of correct sets is then a logical consequence of the annotation plus the observed SCR=0 for VL, so Table 4's 'complementarity' does not provide independent evidence that real detectors and MLLMs are complementary. A single missed fall or a single detector false alarm on a no-action case changes the L0-only set and can move the oracle number. This is exactly the kind of assumption that should be checked before endorsing §5.2's 'route rather than replace' recommendation. I would keep the reader's CONDITIONAL verdict; the concern reinforces rather than overturns it. A minor additional inconsistency: Finding 1 says every MLLM config has SCR=0/3, but Table 3 lists L2 arbitration with SCR=1/3, and that row's Acc=64.7% is also hard to reconcile with its UER/SCR/FBR/HCR values; this is a reporting error but not the central load-bearing issue.","tokens_in":9034,"tokens_out":7394,"duration_ms":64487,"concrete_test":"Run an actual detector stack (e.g., a wake-word engine, pose/fall detector, and OCR keyword matcher) on the same 17 clips with the same candidate device-action pairs and gold labels, and record its real decisions. Then recompute UER/SCR and the L0-only correct set in Table 4, and recompute the oracle accuracy over {real-L0, VL}. If real-L0 misses any of V11/V16/V17 or correctly blocks/asks on any no-action clip, the disjointness and the 94.1% upper bound will change; report the new oracle value and the detector's precision/recall.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4's central complementarity result rests on the L0-only correct set {V11,V16,V17}, and that set is not produced by any detector. In §3, L0 is annotated by the authors: for each scenario they record what a conventional wake-word/pose/OCR detector 'would do', and §4 reports that this proxy fires on all 17 clips because every scenario contains a surface cue. Given that annotation, L0 is correct exactly on the 3 action-required cases (execute on z=1) and incorrect on all 14 no-action cases. Since Table 3 shows the relevant MLLM comparator (VL) has SCR=0/3, the correct sets are disjoint by construction: L0 can only be right where VL is forced to be wrong, and vice versa. The 94.1% oracle is therefore the union of an idealized always-execute baseline and an over-refusing MLLM, not an empirical discovery about real detector stacks. If a real detector misses even one of V11/V16/V17 (a fall missed by a pose/IMU threshold is a known failure mode), the oracle drops to at most 15/17 = 88.2%, and the paper's premise that 'detectors handle all action-required cases' no longer holds. The limitation section concedes the proxy is idealized 'in both directions,' but the routing recommendation in §5.2 depends on the proxy being approximately faithful; that assumption is currently untested, and the headline number changes if it fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PromptShield-Home, a 19-scenario pilot benchmark (17 video clips plus 2 still images) for ambient multimodal prompt injection in smart-home agents. It compares a traditional-detector baseline (L0, annotated by the authors), a single-MLLM layer (L1: vision, vision+ASR, audio-visual), and multi-agent mediation (L2: voting, specialists, arbitration) on a ternary execute/block/ask_user decision. Because most labels are no-action, the paper argues aggregate accuracy is degenerate and reports UER (unsafe-execution rate) and SCR (safe-completion rate). The headline finding is that L0 and the vision MLLM have disjoint correct sets, so an oracle over {L0, VL} reaches 16/17 (94.1%) accuracy versus 13/17 (76.5%) for the best single layer; the authors explicitly label this an upper bound, since no router is implemented, and recommend routing and sensor fusion rather than MLLM replacement.","tokens_in":9346,"tokens_out":6871,"duration_ms":63150,"significance":"The paper's main methodological contribution is the insistence on reporting safety and utility separately: the always-block baseline reaches 82.4% accuracy, so accuracy alone is uninformative in this distribution. The UER/SCR decomposition is simple and correct. The paper also honestly reports its own limitations (small N, author-annotated L0, single-run inference, no router) and checks robustness with five seeds for selected configurations. If the complementarity result held with a real detector baseline, it would be a useful and falsifiable argument for layered routing in smart-home agents. As a pilot study, it is a reasonable first step, but the central empirical claim currently rests on an idealized proxy rather than on measured detector behavior.","major_comments":[{"comment":"The central complementarity result is an arithmetic consequence of the L0 proxy annotation, not an empirical finding about real detectors. Because Section 3 assigns L0 execute on all 17 clips (every scenario is said to contain a surface cue), L0 is exactly an always-execute layer; its correct set is therefore precisely the three action-required cases. Since Table 3 shows VL has SCR=0/3, the correct sets are disjoint by construction, and the 94.1% oracle follows necessarily. The limitation paragraph concedes the proxy is idealized 'in both directions,' but the routing recommendation in §5.2 depends on L0 having perfect recall. A real detector that misses one of V11/V16/V17 lowers the oracle to at most 15/17 = 88.2%. To make the headline claim load-bearing, the authors should either run an actual wake-word/pose/OCR detector stack or explicitly present the result as conditional on an idealized perfect-recall baseline.","section":"Section 3 (Traditional-detector annotation); Table 4"},{"comment":"The over-refusal finding is based on only three action-required cases (one real fall and two command-floor cases), so every MLLM's SCR=0/3 is three failures and a single decision flip changes SCR by 33.3 points. The paper acknowledges the small N, but the generalization 'every MLLM over-refuses' is asserted more strongly than the data support. In particular, Omni flips 16/17 decisions across seeds (Table 5), so at least one model's behavior is unstable rather than systematically cautious. I recommend either expanding the action-required set or softening the claim to 'in this pilot, all tested MLLM configurations failed to execute the available action-required cases.'","section":"Section 4 (Finding 1); Table 3"},{"comment":"The 'route, don't replace' recommendation is a design hypothesis, not an outcome of the experiment: no router is implemented, and the oracle in Table 4 assumes the router always picks the correct layer. The paper does state this, but the discussion should go further and specify what would be needed to realize the 94.1% in practice (e.g., a calibrated confidence signal, a cost model balancing UER and SCR, and a failure analysis of router errors). Without that, the central recommendation is an interesting conjecture rather than a demonstrated result.","section":"Section 5.2; Section 6"}],"minor_comments":[{"comment":"Please show the count explicitly: the table reports 3 + 13 = 16 correct cases and 94.1%, but it never names the one case (V06?) that neither layer solves; adding that case to the table would make the complementarity easier to verify.","section":"Table 4"},{"comment":"Since the raw media are not released, please provide the per-scenario surface-cue annotation (or an appendix table) that justifies the claim that every scenario contains a cue the L0 proxy latches onto; otherwise the L0 baseline is not independently checkable.","section":"Section 3"},{"comment":"Please report the sampling temperature and decoding parameters used for Table 3 versus Table 5; the paper says inference is 'mostly single-run,' but without this detail the reader cannot tell whether the single-run results are greedy or sampled.","section":"Table 5"},{"comment":"There is a likely typo in an author email ('outllook.com'), and the label 'ask_user_or_block' in Table 2 should be defined or written with spaces so it reads as a ternary gold decision rather than a code string.","section":"Header and Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-scale pilot with honestly stated limitations. The main reason I am not recommending reject is that the authors explicitly report the oracle as an upper bound and concede the L0 proxy's idealization; the gap is that they still use that proxy to support a general design recommendation. With either a real detector baseline or a carefully conditional framing, this could be acceptable for a companion venue. I would also ask the PC to weigh whether the 5-page format gives enough room to report the per-scenario annotation that the L0 baseline needs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely honest pilot, and the UER/SCR split is worth borrowing. But the headline 'disjoint correct sets' is close to an artifact of how L0 is defined. L0 isn't a measured detector stack; it's an annotation of what a conventional detector 'would do,' and that annotation fires on all 17 clips. So L0 is effectively always-execute, correct on the 3 action-required cases and wrong on all 14 no-action cases. The vision MLLM, by contrast, never executes an action-required case. Disjointness follows by construction. The paper says it's an idealised proxy and calls the 94.1% an upper bound, so the authors aren't hiding this. But the 'route, don't replace' recommendation depends on the proxy being roughly faithful; if a real fall detector misses V11, the oracle drops. That's untested.\n\nWhat's actually new: the benchmark itself, with benign/harmful pairs (real fall vs yoga pose, self-talk vs command) is a useful testbed. Reporting UER/SCR separately is the right call, since accuracy is degenerate (the best-accuracy config is always-block). The frame-budget sweep is a nice negative result: more frames don't fix over-refusal and can regress injection cases. The robustness check shows Omni's caution is instability, which is a good catch.\n\nSoft spots beyond the L0 proxy: N=17 with only 3 action-required cases, mostly single-run inference, gold labels by author consensus rather than independent annotators. The paper acknowledges all of these. The absence of a real router means the central claim is a motivating upper bound, not a system. The ASR-harms-safety finding is interesting but rests on a handful of clips.\n\nWho should read it: anyone building smart-home agents or evaluating MLLM safety, as a source of scenario design and metric discipline. It doesn't establish that routing works, but it gives a reasonable argument for why it might.\n\nRecommendation: this deserves a serious referee for a workshop/companion venue. For a main conference I'd want a measured L0 or a real router. As it stands, the claims are appropriately hedged and the limitations are explicit, so it's a solid pilot.","headline":"Honest pilot with a useful metric split, but the disjoint-correct-sets oracle is largely an artifact of the idealized always-execute L0 proxy.","tokens_in":9850,"tokens_out":2726,"would_cite":false,"duration_ms":24929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PromptShield-Home finds that no single layer balances safety and utility against ambient prompt injection: detectors execute everything, MLLMs over-refuse, and only an oracle fusion passes 90%.","keywords":["ambient prompt injection","smart-home agents","multimodal large language models","over-refusal","unsafe execution","safe-completion rate","sensor fusion","learned routing"],"falsifier":"Run the 19 PromptShield-Home scenarios through actual wake-word, pose, and OCR detectors with realistic miss and false-alarm rates, then recompute the union of decisions with the vision MLLM; if the real detector layer misses the fall case V11, fires on scenarios the annotation blocks, or shares correct cases with the MLLM, the 94.1% oracle figure collapses toward the best single-layer figure.","tokens_in":1763,"feed_emoji":"🛡️","tokens_out":4702,"duration_ms":93076,"temperature":0.7,"pith_summary":"PromptShield-Home is a pilot benchmark for a safety question unique to smart homes: can an agent distinguish a genuine command from ambient content that merely looks like one, such as TV speech, on-screen text, or an overheard conversation? The paper compares three abstraction layers—traditional detectors, single multimodal LLMs, and multi-agent mediation—across 19 scenarios, evaluating unsafe-execution rate and safe-completion rate rather than aggregate accuracy, because a constant always-block predictor already scores 82%. The core finding is that the two paradigms fail in opposite directions: detectors execute all 17 decision cases, while every MLLM configuration completes none of the three action-required cases and misses a real fall. Their correct sets are disjoint, so an oracle that always picks the right layer reaches 94.1% accuracy versus 76.5% for the best single layer, reported as an upper bound because no router is implemented. The authors conclude that home-agent safety is best served by learned routing and sensor fusion, not by replacing detectors with an MLLM.","feed_headline":"Home agents either act too much or refuse too much","feed_subtitle":"Detectors and multimodal models are complementary; an oracle fusion reaches 94.1% versus 76.5% for the best single layer.","key_machinery":"The argument is carried by the ternary decision space execute/block/ask_user and the paired metrics UER (unsafe-execution rate, the share of no-action cases executed) and SCR (safe-completion rate, the share of action-required cases executed). The second load-bearing mechanism is the layer decomposition into traditional detectors, a single MLLM under vision, vision plus ASR, and audio-visual settings, and multi-agent mediation, together with the set analysis showing that the traditional-detector layer and the vision MLLM have disjoint correct sets. The oracle over these two layers produces the 94.1% upper bound; the authors explicitly note that the detector baseline is an idealized proxy annotation, so the complementarity overstates what a fielded detector stack would contribute.","core_discovery":"The central claim is that no single abstraction layer can balance safety and utility for a home agent facing ambient multimodal prompt injection. Traditional detectors (wake-word, pose, OCR) execute every scenario, giving an unsafe-execution rate of 14/14 and a safe-completion rate of 3/3; every MLLM configuration collapses to the opposite corner, with unsafe execution near zero but safe completion 0/3, missing the genuine fall in every case. Because most scenarios are no-action cases, aggregate accuracy is degenerate: the highest-accuracy configuration, at 82.4%, is byte-identical to an always-block predictor. The decisive discovery is complementarity: the detector layer's correct set and the vision MLLM's correct set are disjoint—no case is solved by both—so an oracle over the two layers reaches 94.1% versus 76.5% for the best single layer. This is an upper bound, not an implemented system, and it motivates routing each decision to the appropriate layer and fusing physical sensors with MLLMs rather than replacing detectors.","pith_inferences":["Extending the paper's logic, the difference between the 94.1% oracle and the 76.5% best single layer defines a 'routing headroom' of 17.6 points, giving future defense designs a concrete target to close.","Because the detector layer is an idealized annotation, a fielded detector stack would likely break the perfect disjointness; a trained router should be tested with measured sensor miss and false-alarm rates before treating complementarity as deployable.","The gold labels were fixed by author consensus, so an independent annotation pass on the same 19 scenarios would test whether the disjoint failure modes survive label variation, and a larger, more balanced scenario set would make the claimed complementarity more robust.","The over-refusal pattern may partly reflect the benchmark's strong prior toward inaction; a real deployment must set the cost asymmetry between a missed fall and an unwanted action explicitly, rather than inherit it from an always-block baseline."],"forward_implications":["A single MLLM agent is not a safe standalone decision-maker in the home: it blocks nearly every genuine command and misses the real fall in every tested configuration.","Adding ASR transcripts to the vision model increases unsafe execution from 1/14 to 4/14 cases, so more modalities do not automatically improve safety; native audio only shifts decisions to ask_user.","Increasing frame count and per-frame resolution does not recover the missed action cases and can flip a spatial-injection case to unsafe execution at high frame counts, so denser sampling is not a remedy for over-refusal.","An oracle-style router over detectors and a vision MLLM suggests a 94.1% accuracy ceiling, but because no router is implemented, a learned router must be built and evaluated to see how much of that gap is reachable.","For fall detection, the paper recommends a high-sensitivity physical sensor trigger (for example IMU or radar) paired with an MLLM false-alarm filter rather than a camera-only agent."],"supporting_citations":[{"why":"Establishes the threat class of indirect instruction injection through images and sounds in multimodal LLMs, which the benchmark operationalizes as ambient prompt injection.","marker":"[1]"},{"why":"Provides a formalization and benchmark methodology for prompt injection attacks and defenses, framing how PromptShield-Home positions its evaluation.","marker":"[9]"},{"why":"Documents accidental triggers of smart speakers, grounding the ambient-content failure mode that motivates the addressee-ambiguity scenarios.","marker":"[11]"},{"why":"Supplies lessons from system-level defenses against indirect prompt injections, which the routing-based defense direction extends.","marker":"[12]"},{"why":"Surveys elderly fall detection systems, supporting the proposed high-sensitivity sensor plus MLLM false-alarm filter for catching falls.","marker":"[15]"},{"why":"Describes a defense against context-aware prompt injection for LLM agents, referenced as related work for role-based and routing defense policies.","marker":"[16]"}],"fun_headline_variants":["Home agents act on everything or refuse everything","Detectors act, MLLMs refuse: no single layer suffices","Oracle fusion for home-agent safety: 94.1% vs 76.5%","Smart-home agents need routing, not replacement","Ambient prompt injection: overaction or overrefusal"],"cache_read_input_tokens":12032,"weakest_assumption_plain":"The load-bearing premise is that the traditional-detector layer can be represented by an idealized annotation of what a wake-word, pose, or OCR detector would fire on; if real detectors miss different cases or fire on different ones, the disjointness and the 94.1% oracle upper bound change.","fun_headline_variants_meta":{"raw":{"variants":["Home agents act on everything or refuse everything","Detectors act, MLLMs refuse: no single layer suffices","Oracle fusion for home-agent safety: 94.1% vs 76.5%","Smart-home agents need routing, not replacement","Ambient prompt injection: overaction or overrefusal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3633,"prompt_tokens":1057,"completion_tokens":2576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":2491}},"tokens_in":673,"tokens_out":2576,"duration_ms":18260,"temperature":1.0,"reasoning_tokens":2491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:02:10.342039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 19 PromptShield-Home scenarios through actual wake-word, pose, and OCR detectors with realistic miss and false-alarm rates, then recompute the union of decisions with the vision MLLM; if the real detector layer misses the fall case V11, fires on scenarios the annotation blocks, or shares correct cases with the MLLM, the 94.1% oracle figure collapses toward the best single-layer figure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a formalization and benchmark methodology for prompt injection attacks and defenses, framing how PromptShield-Home positions its evaluation."},{"cited_title":"Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, and John \"Four\" Flynn","cited_arxiv_id":null,"evidence_quote":"Supplies lessons from system-level defenses against indirect prompt injections, which the routing-based defense direction extends."}],"review_version":1}