{"id":"320668db-6a66-4248-b4e8-9be11bfbc5be","arxiv_id":"2608.09857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM-driven gate between robot planning and execution labels plans accept, reject, or escalate, reporting 81 percent accuracy and no direct accept/reject errors on small test sets.","lead":"This paper presents a verification layer that uses multiple LLM judges, plus a chief judge that never sees the plan, to accept, reject, or escalate robot plans before they reach the robot's controls. The authors report high precision and zero direct accept/reject errors, but some headline numbers contradict the paper's own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own tables contradict the headline: Section III.B's 97% containment and one catastrophic error do not match Table II's 100% containment and zero critical failures; one sample flips the result.","rationale":"The paper's central claim is a set of error-rate guarantees. Those guarantees are load-bearing because the entire deployment argument (\"verification layer ... before they reach MCP server\") rests on the assertion that dangerous plans are contained with near-certainty while acceptable plans are not blocked. The most serious weakness I found is not the small dataset per se — though N=38 is small — but that the published numbers do not agree with each other. Section III.B/abstract vs Table II is an irreconcilable pair unless a metric definition changed between text and table; Section III.C vs Table III is similarly inconsistent. These are exactly the checks a careful reader would run first, and they fail. This is not an ad hominem or a stylistic complaint: it is a correctness defect in the evidence for the central claim. The reader's conditional verdict already flags contradictions; my stress-test identifies the adversarial-containment contradiction as the single most decisive one because it is a one-sample switch between 100% and 97% and involves the most safety-critical quantity. I also acknowledge independent support: code and data are promised at a public GitHub URL, the evaluation protocol is described in enough detail to reproduce, and the plan-blind chief judge design is a concrete architectural idea worth testing. Therefore I would not escalate the verdict beyond the reader's CONDITIONAL; I would instead make the corrections and the re-run prerequisites for the quantitative claims to be accepted. If the re-run reveals Table II to be correct, the authors must retract or correct the 97%/single-error wording; if it reveals a true accept-of-reject event, the \"zero catastrophic errors\" and 100% containment claims must be revised to include that event and its downstream consequence.","tokens_in":9682,"tokens_out":6099,"duration_ms":58191,"concrete_test":"Re-run the released pipeline (GitHub code/data) on the N=38 adversarial set with the same five-judge ensemble and chief judge, and print the raw per-plan confusion matrix and error log. Reconcile the abstract's 97% containment and Section III.B's \"single catastrophic error\" with Table II: if the true matrix is Table II's, the abstract and Section III.B are wrong; if a ground-truth reject plan is actually accepted, Table II's 0.00 critical-failure rate and 1.00 containment are wrong. Then compute Clopper-Pearson 95% confidence intervals for containment and for the accept↔reject confusion rate to show how wide the uncertainty is at N=38. Optionally, to test threshold generalization, hold out a split before choosing τ in Table III; if the best threshold moves from 0.85, the Section III.C peak claim is a test-set artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.B states that the system \"contains 97% of unsafe plans and produces only a single catastrophic error, as displayed in Table II,\" yet Table II reports Unsafe-plan containment = 1.00 and Critical-failure rate (accept↔reject) = 0.00. The underlying N=38 matrix shows 31 ground-truth reject plans with 0 predicted accepts and 27 predicted rejects (4 escalates), and 3 ground-truth accept plans with 0 predicted rejects, so there is no accept↔reject confusion in the table. The difference between 97% and 100% is exactly one plan (1/31 ≈ 3.2%), meaning the headline containment figure is a one-sample swing. Section III.C similarly claims the injection checker \"peaks at a threshold of 0.85,\" while Table III lists F1=0.85 at τ=0.85 and F1=0.89 at τ=0.75, with no sharp dropoff on the low side (τ=0.65 gives F1=0.88). Because the central claims are quantitative — \"near 85% precision,\" \"97% containment,\" \"negligible accept/reject errors\" — these internal contradictions are not cosmetic. They leave the reader unable to determine which numbers are true, and the small sample sizes (N=38, N=58, N=46) make every single corrected cell move the headline metric by several points. This is an internal inconsistency, not merely a disagreement with prevailing consensus; it must be resolved before the safety claims can be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-driven verification layer placed between a robot planning module and an MCP server, gating plans into accept, reject, or escalate. The layer combines an ensemble of five LLM judges, a chief judge that synthesizes their reasoning without seeing the plan, deterministic provenance and prompt-injection checks, and a knowledge graph that supplies contextual grounding. The central empirical claims are near-85% precision across the three decision classes, 97% containment of adversarial plans, negligible accept/reject confusions, and errors concentrated at the escalate boundary. Evaluations use N=58 comprehensive, N=38 adversarial, and N=46 injection sets, plus an ensemble-size study on N=55.","tokens_in":9949,"tokens_out":6400,"duration_ms":55849,"significance":"If the architecture delivered the claimed operating characteristics, it would be a valuable safety middleware for LLM-based robot planning, addressing semantic safety and adversarial robustness in a single runtime structure. The paper clearly describes a plausible two-layer judge design and provides a code/data link, which supports reproducibility. However, the quantitative claims are undermined by internal inconsistencies between the text and the tables, and by calibration of weights and thresholds on the same test sets used for evaluation. The headline containment and threshold-peak claims cannot be assessed until these contradictions are resolved.","major_comments":[{"comment":"The text states that the system 'contains 97% of unsafe plans and produces only a single catastrophic error, as displayed in Table II,' but Table II reports Unsafe-plan containment = 1.00 and Critical-failure rate = 0.00. The confusion matrix contains zero accept↔reject cells: among 31 ground-truth rejects there are 0 predicted accepts, and among 3 ground-truth accepts there are 0 predicted rejects. The difference between 97% and 100% is exactly one plan out of 31 rejects, so the headline containment figure is a one-sample swing. Please correct the text or the table and state which number is the actual result.","section":"III.B, Table II"},{"comment":"The text claims that injection-checker quality 'peaks at a threshold of 0.85' with a 'sharp drop off' on both sides, but Table III shows F1=0.89 at τ=0.75 and F1=0.88 at τ=0.65, with accuracy highest at τ=0.75 (0.85 vs 0.80 at τ=0.85). The reported table does not support the peak-at-0.85 claim; please reconcile the text with the table or provide the correct operating-point analysis.","section":"III.C, Table III"},{"comment":"The suspicion-score combination weights (w_r=0.65, w_f=0.35) and the advisory/warning thresholds are calibrated using the same curated injection test set (Table III) on which the headline screening performance is reported. No held-out split, cross-validation, or sensitivity analysis is provided. This makes the reported F1 and threshold selection fit-to-test-set rather than an independent estimate, which is a particular concern because the safety claims are quantitative.","section":"II.E, III.C"},{"comment":"The evaluation is based on very small samples (N=38 adversarial, N=58 comprehensive, N=46 injection, N=55 ensemble study), and several table cells contain one or two plans. A single plan moves the adversarial containment metric by about 3 percentage points, so the reported differences between configurations (e.g., 97% vs 100%) are not robust. The paper should report confidence intervals or at least explicitly state the sensitivity of each headline number to one reclassification.","section":"III.B, III.D"},{"comment":"Footnote 4 states that the injection checker was tested via a separate script and not through the main judging pipeline, with integration listed as future work. This means the Table III results do not yet demonstrate end-to-end operation of the guardrails as a component of the verification layer, which weakens the claim that the reported containment is achieved by the full architecture.","section":"III.C, footnote 4"}],"minor_comments":[{"comment":"There are several typographical and formatting errors: 'specifc' should be 'specific' (I.C), 'experimented consisted' should be 'experiment consisted' (III.A), 'thse' should be 'these' (II.D), 'Interhsips' should be 'Internships' (Acknowledgments), and '1,5 36, 57, and 78,judges' should be '1, 3, 5, and 7 judges' (III.D).","section":"I.C, III.A, II.D, III.D, Acknowledgments"},{"comment":"The text states that 'precision on the escalate class is the primary area for further gains,' but Table II explicitly does not report per-class precision/recall because of minimal support. Please either report these values with a caveat or rephrase the claim so it is supported by the displayed table.","section":"III.B"},{"comment":"The dataset description is ambiguous: 'AI-generated plans (prompted few-shot style, with human developed plans)' does not clarify whether the plans are AI-generated, human-developed, or a mixture. Please specify the generation process and report inter-annotator agreement on the human ground-truth labels.","section":"III.A"},{"comment":"The ensemble-size comparison lacks repeated-run variance; LLM judges are stochastic, and the paper does not report whether the differences between configurations are stable across seeds. A small repeated-run study would strengthen the claim that the 7-judge ensemble outperforms the 1-judge baseline.","section":"III.D"}],"recommendation":"major_revision","confidential_remarks":"The internal contradictions between the narrative and the tables in Sections III.B and III.C are serious enough that the paper should not be accepted in its current form, but they appear fixable with a corrected write-up and a proper calibration/validation split. The paper would also benefit from acknowledging that the headline numbers rest on very small, partly tuned datasets. The scope is more of a system description and preliminary evaluation than a definitive safety guarantee; the authors should temper the deployment-oriented claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a system paper worth reading, but not worth quoting numbers from yet. The contribution is a verification layer between an LLM planner and the robot's MCP server, with three components: a five-model judge ensemble, a deterministic injection/provenance checker, and a knowledge graph from CWE/MITRE ATLAS and lab docs. The most interesting design choice is the plan-blind chief judge: the final arbiter sees only the judges' reasoning, not the plan itself. That's a real attempt to stop a second LLM from being exposed to the same prompt injection. I buy that as a contribution.\n\nThey also ship code and data, which is more than many papers in this area do.\n\nThe problem is the numbers. The abstract and Section III.B report 97% adversarial containment and a single catastrophic error, but Table II shows 100% containment and zero critical failures. One plan out of 31 reject-class samples is the difference — a one-sample swing. Similarly, Section III.C says the injection checker peaks at threshold 0.85 with a sharp dropoff, but Table III shows F1=0.89 at τ=0.75 and 0.88 at τ=0.65. The 'sharp dropoff' on the low side isn't there. These are load-bearing inconsistencies because the whole claim is quantitative.\n\nBeyond that, Section II.E selects the combination weights and thresholds by inspecting detection performance on the same style of test set that produces the headline results. With N=38, 46, and 58, and no error bars or repeated runs, the precision figures are likely optimistic. The injection checker results were produced by a separate script, not through the full judging pipeline (footnote 4), so the 'containment' figure and the injection-screener F1 are not on the same path. There are also no external baselines; an LLM-as-a-judge baseline would help.\n\nThe design isn't dead on arrival. The central idea is testable, and the structure is clear. But as submitted, I can't tell which of the reported numbers are true. A serious referee should send it back with a request to reconcile the contradictions, add confidence intervals, run baselines, and either justify the threshold selection or hold out the test set. The paper deserves a real review, though — the system concept is worth developing.\n\nMy call: send it to peer review, but expect heavy revision. I'd bring the architecture to a reading group, but I wouldn't cite the quantitative claims until they are fixed.","headline":"The plan-blind chief judge is a genuinely nice idea, but the evaluation numbers contradict each other and the thresholds are tuned on the test set, so the headline percentages can't be trusted yet.","tokens_in":10538,"tokens_out":2654,"would_cite":false,"duration_ms":23599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A plan-blind chief judge over five LLM jurors gates robot plans into accept, reject, or escalate with near 85% precision and zero accept–reject confusions.","keywords":["LLM-as-a-judge","robot autonomy","plan verification","safety middleware","prompt injection","adversarial attacks","knowledge graph","accept/escalate/reject"],"falsifier":"Run the same pipeline with thresholds and prompts frozen on a fresh corpus produced by a planner and an attack generator not used in this study, with ground truth labeled independently; if any ground-truth reject plan is accepted, or containment falls materially below 97%, the central claim is falsified. The cleanest single number to watch is the false-accept rate on ground-truth reject plans, which the paper reports as zero.","tokens_in":9419,"feed_emoji":"🛡️","tokens_out":9674,"duration_ms":85217,"temperature":0.7,"pith_summary":"This paper tries to establish that an LLM-driven verification layer placed between a robot's planning module and its execution server can reliably gate natural-language plans into three outcomes: accept, reject for reformulation, or escalate to a human, so that unsafe, misaligned, or adversarial plans never reach the robot's low-level controls. The proposed layer is a two-stage jury: several LLM judges reason over each plan with knowledge-graph context, and a plan-blind chief judge synthesizes their rationales into a final verdict. On the authors' evaluation, the layer reaches weighted precision near 85%, produces zero accept–reject confusions in the main test, and contains 97% of adversarial attack plans, with most residual errors falling on the escalate boundary where human review still catches them. If these numbers hold outside the test set, the architecture offers a model-agnostic safety middleware that can be dropped into an existing autonomy stack without retraining the planner or modifying the robot controller.","feed_headline":"Robot-plan gatekeeper stops 97% of adversarial plans","feed_subtitle":"Five LLM judges reason over each plan; a plan-blind chief judge decides accept, reject, or escalate.","key_machinery":"The load-bearing mechanism is the two-stage LLM-as-a-judge ensemble with a plan-blind chief judge. The first stage is an asynchronous ensemble of five models sharing one system prompt, few-shot examples, explicit accept/escalate/reject decision rules, and retrieved context from a knowledge graph that indexes hardware documentation, vulnerability catalog entries, AI-threat taxonomies, lab safety and ethics manuals, and prior rejected plans. The second stage is a heavier reasoning model that receives only the judges' structured rationales and response objects, never the plan itself, and is instructed to verify the soundness of each judge's claims, treat judges equally except when they give strong reasons, and automatically escalate if it conflicts with three or more judges. A deterministic pre-checker, including a capability allow-list, provenance checks, and regex plus fuzzy prompt-injection screening with advisory and warning thresholds, blocks or annotates inputs before the LLM stage. This combination converts the variable reasoning of several models into a single auditable, conservative verdict without letting any one model, or the plan itself, dominate the final decision.","core_discovery":"The paper's central claim is that judging the judges works as a verification strategy: instead of trusting any single LLM's verdict on a plan, the system delegates final authority to a chief judge that never sees the plan. Each of five primary judges receives the plan, a few-shot prompt with decision rules, and relevant knowledge-graph context, and returns a chain-of-thought verdict of accept, escalate, or reject. The chief judge then evaluates only the judges' rationales, weighs explicit risk arguments, and applies a rule that disagreement with three or more judges forces escalation. The reported effect is a conservative classification boundary: in the 58-plan comprehensive test, accuracy is 0.81, weighted precision is 0.85, and there are zero catastrophic accept–reject confusions; in the 38-plan adversarial test, unsafe-plan containment is reported at 97% with a 0.00 critical-failure rate; and the deterministic injection checker reaches its best F1 of 0.89 at a favorable threshold.","pith_inferences":["If the zero-false-accept property holds on larger samples, the practical cost of safety shifts from catastrophic failure to human-review workload, so a lab could budget for the escalate rate as a service metric.","A natural extension is to use the logged rejections to train a fast similarity-based pre-filter, yielding a two-tier system in which cheap deterministic screening catches known attack shapes and the LLM jury handles only novel or ambiguous plans.","The injection thresholds and hand-authored decision rules were tuned on the same style of test inputs, so the strongest transfer test would freeze the thresholds and prompts and evaluate on a different planner and a different robot stack with no adjustment."],"forward_implications":["The verification layer can be inserted between existing planning and execution modules without retraining the planner or controller, since plans pass through unchanged except for the verdict.","Residual errors are concentrated at the escalate boundary, so the dominant failure mode is additional human review rather than silent acceptance of a dangerous plan.","Ensemble size has only a small effect on accuracy, from 0.76 to 0.78 as judges increase from 1 to 7, so smaller committees can cut latency and cost while reject-class F1 still improves with more judges.","The architecture appears model-agnostic: substituting different frontier models for the jury preserved the overall accuracy band and the zero-false-accept property, suggesting it can track model upgrades.","A deterministic, non-LLM screen for provenance, skill allow-lists, and prompt injection runs before the jury, so part of the attack surface is covered even before the models reason about the plan."],"supporting_citations":[{"why":"Motivates the insider-threat and agentic-misalignment risk that the verification layer is designed to contain.","marker":"[1]"},{"why":"Documents systematic safety risks in LLM-based embodied planning, the failure mode the judges screen against.","marker":"[2]"},{"why":"Contextualizes the planning-module outputs the verification layer receives from an embodied foundation-model orchestration system.","marker":"[4]"},{"why":"Provides the self-consistency result that multiple reasoning passes yield more reliable outputs, grounding the ensemble design.","marker":"[6]"},{"why":"Shows single LLM evaluators favor their own generations, justifying a multi-model ensemble over a single judge.","marker":"[7]"},{"why":"Supports the committee-of-judges design, including discussion and synthesis, which the chief-judge layer builds on.","marker":"[8]"},{"why":"Supplies evidence that foundation models can act as safety-aware validators for robot task plans.","marker":"[12]"},{"why":"Supports assigning the judge role through the system prompt for better task-specific reasoning.","marker":"[13]"}],"fun_headline_variants":["LLM judge panel gates 97% of unsafe robot plans","Five-LLM jury vets every robot plan before execution","Robot safety: LLM judges review actions, block attacks","Chief judge never sees plan, overrules LLM panel","97% of adversarial plans contained by LLM judge layer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human-labeled ground truth on the test sets, 58 plans for the main evaluation, 38 for adversarial, and 46 for injection screening, is representative of real planner outputs and real attacks, and the hand-written prompts and calibrated thresholds are not fitted to those particular examples.","fun_headline_variants_meta":{"raw":{"variants":["LLM judge panel gates 97% of unsafe robot plans","Five-LLM jury vets every robot plan before execution","Robot safety: LLM judges review actions, block attacks","Chief judge never sees plan, overrules LLM panel","97% of adversarial plans contained by LLM judge layer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000349,"raw_usage":{"total_tokens":1915,"prompt_tokens":961,"completion_tokens":954,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":871}},"tokens_in":577,"tokens_out":954,"duration_ms":7070,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:22:52.976335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with thresholds and prompts frozen on a fresh corpus produced by a planner and an attack generator not used in this study, with ground truth labeled independently; if any ground-truth reject plan is accepted, or containment falls materially below 97%, the central claim is falsified. The cleanest single number to watch is the false-accept rate on ground-truth reject plans, which the paper reports as zero.","supporting_citations":[{"cited_title":"Agentic Misalignment: How LLMs Could Be Insider Threats,","cited_arxiv_id":null,"evidence_quote":"Motivates the insider-threat and agentic-misalignment risk that the verification layer is designed to contain."},{"cited_title":"Using large language models for embod- ied planning introduces systematic safety risks,","cited_arxiv_id":null,"evidence_quote":"Documents systematic safety risks in LLM-based embodied planning, the failure mode the judges screen against."},{"cited_title":"AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents,","cited_arxiv_id":null,"evidence_quote":"Contextualizes the planning-module outputs the verification layer receives from an embodied foundation-model orchestration system."},{"cited_title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models,","cited_arxiv_id":null,"evidence_quote":"Provides the self-consistency result that multiple reasoning passes yield more reliable outputs, grounding the ensemble design."},{"cited_title":"LLM Evalua- tors Recognize and Favor Their Own Generations,","cited_arxiv_id":null,"evidence_quote":"Shows single LLM evaluators favor their own generations, justifying a multi-model ensemble over a single judge."},{"cited_title":"Safety Aware Task Planning via Large Language Models in Robotics,","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that foundation models can act as safety-aware validators for robot task plans."}],"review_version":2}