Pith.

Cryptography and Security · arXiv:2610.11030 v2 · submitted 8 Oct 2026

NOMOS: Compiling Written Policies into Statically Verified Tool-Call Gates for LLM Agents

Min-Young Yu, Tony Kim, Jang Won Choi

The paper says

“Tool-using LLM agents violate the policies they are deployed to enforce, often silently. Prior defenses hand-write rules, query an LLM verifier per action, or compile policies through heavyweight formal machinery.”

more of the abstract

Naive compilation fails: extracted rules block the tool satisfying their own precondition, or read arguments their tool lacks. NOMOS, a four-pass compiler, turns a natural-language policy into a deterministic tool-call gate; static verification with tool-schema-level checks alone (no prover, solver, or LLM) repairs or rejects 37% (airline) and 13% (retail) of candidates, without which most shipped rules are inoperable. Replaying compiled rules over undefended transcripts flags bindings that refuse legitimate work (a development binding refused 95.9% of task-passing calls); no evaluation binding is flagged. On τ2-bench the gate cuts violations of reference-encoded clauses among state-changing calls from 66.3% to 2.6% (airline) and 30.8% to 6.9% (retail), raising airline task success significantly for 2 ≤ k ≤ 4; a 26B on-premise compilation is not significantly worse than hand-written or frontier-compiled rules. Unlike AgentDojo's shipped defenses, it reaches a zero attack success rate (ASR) on banking, where nine attack families collapse onto three structural rules. On the other three suites its ASR is at most 3.6%, from goals with no tool call to govern and one write admitted by a binding weaker than its clause; a second agent model, Llama-3.3-70B, reproduces the effect on both benchmarks. Decisions take microseconds without an LLM call, at a domain-dependent benign-utility cost; compilation runs on-premise on open-weight gemma-4-26B.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 11 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Solid systems paper with honest empirics; the main claim—that LLM-extracted rules need a lightweight static verifier—is convincingly supported across two benchmarks and two models. This line is wrong

1 major objection · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.1, Algorithm 1

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.66):

“What the checks donotestablish is that a vocabulary item means what the clause means, or that a binding is behav- iorally right: the former is left to extraction under majority voting, the latter to the behavioral preflight of Section 3.4 (Section 5.3 shows that the binding of this worked example, AIR-AUTO-FBD-03, reaches past its clause). 3.2 Four-pass compiler Pass 1 (PARSE, deterministic)segments the policy docu- ment into clauses with section provenance.”

Pith says: Objection 1 of 1 · major

The entire static verification and the safety guarantee depend on the completeness and correctness of the human-engineered predicate vocabulary V and the two tables Enable[π] and Reads[π]. The paper acknowledges this (e.g., Section 3.1: "What the checks do not establish is that a vocabulary item means what the clause means") but does not provide a formal bound on the consequences of errors in these tables.

more of objection 1

For example, if an Enable[π] entry omits a tool that can clear π, the self-blocking check may fail to strip that tool, potentially creating a deadlock not caught by Algorithm 1. The ablation study (Table 8) indirectly validates the tables for the two τ²-bench domains, but the claim that NOMOS's static verification requires "no prover, no solver, no LLM" is conditioned on this manually constructed ontology. The paper should either (a) include a systematic perturbation analysis of the tables (e.g., randomly corrupting entries and measuring how many defects go undetected), or (b) state more explicitly that the approach is not fully automatic and requires ongoing human ontology maintenance. Because the predicate vocabulary is load-bearing for the central claim of lightweight verification, this assumption deserves a more quantitative treatment.

6 minor objections

The paper says, at Figure 2

Pith says: minor objection 1 of 6

The figure labels show "37%" and "13%" for structural defects repaired/rejected, while the text (Section 3.3 and Table 3) reports 36.7% and 13.2% for the structural subtotal. The rounding difference is minor, but the figure should be consistent with the table or explain that it rounds.

The paper says, at §4, paragraph on checker validity

Pith says: minor objection 2 of 6

The checker audit description is dense. It would help to separate the two judge passes (full policy vs. encoded clauses) more clearly, perhaps with a bulleted summary of disagreement categories (judge errors, interpretive, checker misses) and their counts.

show 4 more

The paper says, at §5.9, Table 12

Pith says: minor objection 3 of 6

The term "spotlighting" is used without citation; the paper should reference Hines et al. (spotlighting) explicitly in the caption or the paragraph.

The paper says, at §5.5, Table 8 caption

Pith says: minor objection 4 of 6

The caption says "airline 69, retail 46 candidates" but the text states the pool is a separate extraction run frozen for exact attribution. It would be clearer to note that this pool differs from the production compile of Table 3 (which uses 79 and 53 candidates).

The paper says, at §3.4

Pith says: minor objection 5 of 6

The behavioral preflight is described as "zero-LLM" because the replay applies deterministic gate transitions. However, the transcripts themselves are generated by an LLM agent. The paper should explicitly note that the preflight is LLM-free in its execution but depends on prior LLM-generated data, which may not cover all behavioral corner cases.

The paper says, at Appendix A, A.3 retail rule RET-AUTO-FBD-01

Pith says: minor objection 6 of 6

The clause for this rule is an operational instruction with three exclamation marks. While the compiler processes the document as given, the example may distract from the technical exposition; consider noting that the binding is to tools with item-list arguments only.

What would settle it

Have two independent domain experts independently construct the Enable and Reads tables for the τ2-bench airline and retail domains. Run the RESOLVE pass (Algorithm 1) with each table set over the frozen candidate pool of Table 8, and compare the sets of repaired/rejected candidates. If the defect detection outcome changes between annotators for any candidate, quantify the rate. Additionally, artificially introduce a random error (e.g., remove one enabling tool from Enable or one argument key from Reads) for a predicate known to be involved in a repair, and check whether that defect now passes the verification. The first test measures inter-annotator reliability; the second measures sensitivity to table errors.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 11 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 44 s · 43,392 tokens read, 4,416 written
  3. Editor wrote the verdict above 12 s · 44,291 tokens read, 1,343 written
  4. Referee minor revision: 1 major objection, 6 minor 35 s · 43,826 tokens read, 2,849 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper is worth reading for the defect taxonomy and the demonstration that simple set-operation checks suffice. The central finding—that without static verification most extracted rules are inoperable—is cleanly supported by the ablation in Table 8. The reporting is refreshingly honest: utility costs, negative results (retail flat), a checker audit, and the clear statement that the predicate vocabulary is human-engineered and load-bearing.

What's new: measured defect frequencies from real compilations, the collapse of injection attack diversity onto structural predicates, and a behavioral preflight that uses existing transcripts rather than synthesized tests. The system does not claim formal guarantees but provides a practical, inspectable blueprint.

Soft spots: (1) The predicate vocabulary and Enable/Reads tables are human-crafted and not formally verified; the paper acknowledges this, but a deployment would need to worry about correctness under drift. (2) The implementation is proprietary; while evaluations use public benchmarks and the paper prints every enforced rule, one cannot re-implement predicate evaluators from the paper alone. (3) The high utility cost on slack (76 points) is a real deployment barrier; the paper correctly attributes it to structural similarity between benign and malicious tasks but offers no mitigation. (4) The claim that injection diversity collapses onto structural predicates is empirically demonstrated on AgentDojo's nine attacks—interesting but would benefit from testing on a broader attack set.

Who this is for: researchers and practitioners building policy-enforcement layers for LLM agents. It deserves a serious referee.

The strongest objection

The static verification (Algorithm 1) depends on the completeness and correctness of the predicate vocabulary V and the two tables Enable[π] and Reads[π]. The paper states that a human prepares these once per domain, and the AgentDojo port also uses a one-entry generalization table. If a predicate is missing from V, a candidate using that predicate is rejected (line 1 of Algorithm 1), so that is safe. The more subtle risk is an incorrect Enable or Reads entry: e.g., if Enable[π] omits a tool that can clear π, the self-blocking check (line 12) will not strip that tool from the guard set, potentially deadlocking; if Reads[π] is incomplete, an argument-signature mismatch may go undetected. The paper does not provide a formal bound on the consequences of such errors, nor does it test the tables' correctness under perturbation. However, the paper is transparent about this assumption: it states that 'what the checks do not establish is that a vocabulary item means what the clause means' (Section 3.1) and that the human-prepared ontology is load-bearing. Empirically, the ablation study (Table 8) shows that with all checks enabled, no inoperable rules survive in the evaluation domains, which indirectly validates the tables used. The risk is acknowledged and partially mitigated by the paper's reporting of defect frequencies and the independent checker. Therefore, while this is a genuine load-bearing concern, it does not invalidate the central claim, because the paper does not claim formal verification of the vocabulary, only that structural defects are detected using those tables.

The reader identified the same load-bearing assumption: the completeness and correctness of the predicate vocabulary and its static tables. This is indeed the weakest part of the argument, as the paper acknowledges. However, the central claim—that a lightweight static verifier can repair or reject structural defects, making naively compiled rules safe to enforce deterministically—is supported by strong empirical evidence from controlled ablations and independent safety measurements. The paper does not overstate its guarantees; it openly discusses the human role and the residual risks (e.g., behavioral correctness, global deadlocks). Therefore, the reader's ACCEPT verdict is appropriate, and our stress-test does not identify a new concern that would change that verdict. We concur with the reader's analysis.

The referee's report

If the results hold, the paper makes several important contributions. First, it provides the first measured defect taxonomy of LLM-extracted policy rules, showing that the majority are inoperable without verification. Second, it demonstrates that a lightweight, deterministic verifier (set operations only) can repair or reject these defects without any prover or solver, making automatic compilation practical. Third, the empirical results on two benchmarks (τ²-bench and AgentDojo) are strong: substantial violation reductions, zero ASR on banking under standard and adaptive attacks, and reproduction across two unrelated agent models. The transparent checker audit and honest reporting of utility costs (the slack domain pays 76 points) strengthen confidence. The paper also compares favorably against existing defenses (PolicyGuard, Progent, AgentDojo's shipped defenses) and discusses limitations (predicate ontology, proprietary implementation, domain-dependent cost). For the security and AI safety communities, this work provides a principled, cost-effective approach to policy enforcement that separates structural correctness from model behavior.

Pith's reading of the paper (a machine's summary, not the authors')

When an LLM translates a natural-language policy into rules for a deterministic gate, the rules are frequently defective: they block the tools that would satisfy their own preconditions, demand approval for read-only lookups, or refer to arguments the guarded tool does not accept. Naively enforcing these rules would make most of them inoperable—8 of 14 airline rules and 3 of 5 retail rules when all static checks are removed. NOMOS inserts a four-pass compiler whose static verification pass (requiring no prover, solver, or LLM) repairs or rejects those structural defects, so that only 10 airline and 5 retail rules survive. On τ²-bench, the resulting gate cuts policy violations among state-changing calls by 26-fold (airline) and 4.5-fold (retail), and on AgentDojo banking it reaches zero attack success across nine injection families with three rules. The central message is that the missing component between LLM extraction and deterministic enforcement is a lightweight static verifier, and that this verifier can be built from the tool schemas and two manual tables alone.

What carries the argument. The central mechanism is the static verification pass (RESOLVE) of the NOMOS compiler. It checks each candidate rule for self-blocking reachability (whether the rule guards the only tools that can clear its predicate), write-scope (confirmation preconditions guard writes, not reads), argument consistency (the predicate reads arguments the guarded tool actually accepts), domain satisfiability (the enabling tools exist in the domain), unscoped wildcards, and state contradictions. All checks are set operations over names in the tool schema and two static tables (Enable and Reads). No prover, solver, or LLM is needed; the checks run in milliseconds and can be re-run at load time on any rule file.

Core claim. The paper claims that LLM-extracted policy rules harbor a small set of recurring structural defects—self-blocking deadlock, write-scope overreach, argument-signature mismatch, domain unsatisfiability, and unscoped wildcards—and that these defects can be detected and repaired by a deterministic verifier that performs nothing more than set operations over the candidate rule, the tool schemas, and two static tables (which tools can clear a predicate and which arguments it reads). The verifier repairs or rejects 37% of airline extraction candidates and 13% of retail ones. Without it, a majority of the emitted rule set is inoperable. The paper further claims that an additional behavioral defect (a structurally sound rule that refuses the very calls the policy permits) can be caught by replaying the rules over undefended agent transcripts before deployment. Together, these two layers—static verification and behavioral preflight—make the combination of LLM compilation and deterministic enforcement viable without provers, solvers, or runtime LLM calls.

The premise it rests on. The predicate vocabulary V and its two static tables (Enable and Reads) are prepared manually per domain, and if a predicate is missing or an entry is incorrect, a structural defect may go undetected by static verification.

If the paper is right (Pith's inferences, not the authors')

Any system that couples LLM-generated rules to deterministic enforcement should include a static structural verifier before the rules reach runtime.

The defect taxonomy and measured frequencies provide a baseline for evaluating future extraction methods: a useful metric is the fraction of candidates that survive structural verification.

The behavioral preflight using undefended transcripts offers a zero-LLM screening method for self-defeating rules; it can be applied to any deterministic gate that replays recorded traffic.

The collapse of nine injection attack families onto three structural rules suggests that safety policy for tool-using agents can be kept small and audit-friendly, reducing the attack surface to a handful of provenance questions.

The domain-dependent utility cost of provenance rules (cheap where targets are user-named, expensive in open-ended messaging) gives deployments a mechanism to forecast cost before deployment.

The static verifier's reliance on a human-engineered predicate vocabulary with Enable and Reads tables means that the guarantee is only as strong as the correctness and completeness of that ontology. A missing predicate or erroneous entry would let a structural defect through, so future work should consider automatically deriving these tables from tool documentation or policy structure.

The behavioral preflight's use of undefended transcripts assumes that the undefended agent's call distribution is representative of the defended agent's intended behavior. But if the policy changes what calls the agent tries (as it does on airline, where the gate reduces writes per simulation from 1.41 to 0.46), then the preflight's firing rates on undefended transcripts may overestimate or undere

The paper's central measurement—that without verification a majority of compiled rules are inoperable—suggests that other works reporting successful enforcement of LLM-compiled policies may have inadvertently filtered out defective rules during testing, or used verification steps that were not reported as defect-fixing. The paper provides a methodology (ablation over frozen candidate pool) that co

The results on τ²-bench and AgentDojo are on two agent models (gemma-4-26B and Llama-3.3-70B) and four suites. Extending to more agent models and to adversarial contexts beyond standard injection (e.g., data-poisoning attack on the transaction history) would test the generality of the provenance-gate approach. The paper explicitly flags this limitation for attacks that seed structured data into th

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Tool-using LLM agents violate the policies they are deployed to enforce, often silently. Prior defenses hand-write rules, query an LLM verifier per action, or compile policies through heavyweight formal machinery. Naive compilation fails: extracted rules block the tool satisfying their own precondition, or read arguments their tool lacks. NOMOS, a four-pass compiler, turns a natural-language policy into a deterministic tool-call gate; static verification with tool-schema-level checks alone (no prover, solver, or LLM) repairs or rejects 37% (airline) and 13% (retail) of candidates, without which most shipped rules are inoperable. Replaying compiled rules over undefended transcripts flags bindings that refuse legitimate work (a development binding refused 95.9% of task-passing calls); no evaluation binding is flagged. On τ2-bench the gate cuts violations of reference-encoded clauses among state-changing calls from 66.3% to 2.6% (airline) and 30.8% to 6.9% (retail), raising airline task success significantly for 2 ≤ k ≤ 4; a 26B on-premise compilation is not significantly worse than hand-written or frontier-compiled rules. Unlike AgentDojo's shipped defenses, it reaches a zero attack success rate (ASR) on banking, where nine attack families collapse onto three structural rules. On the other three suites its ASR is at most 3.6%, from goals with no tool call to govern and one write admitted by a binding weaker than its clause; a second agent model, Llama-3.3-70B, reproduces the effect on both benchmarks. Decisions take microseconds without an LLM call, at a domain-dependent benign-utility cost; compilation runs on-premise on open-weight gemma-4-26B.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-09

The authors' main claim, as the classifier picked it out 0.62

“NOMOS, a four-pass compiler, turns a natural-language policy into a deterministic tool-call gate; static verification with tool-schema-level checks alone (no prover, solver, or LLM) repairs or rejects 37% (airline) and 13% (retail) of candidates, without which most shipped rules are inoperable.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.92 · human participants 0.04 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.07

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Tool-using LLM agents violate the policies they are deployed to enforce, often silently. background 1.00 0.15 0.04 asserted 1.00
Prior defenses hand-write rules, query an LLM verifier per action, or compile policies through heavyweight formal machinery. background 0.93 0.04 0.02 asserted 1.00
Naive compilation fails: extracted rules block the tool satisfying their own precondition, or read arguments their tool lacks. gap 0.63 0.58 0.03 asserted 1.00
NOMOS, a four-pass compiler, turns a natural-language policy into a deterministic tool-call gate; static verification with tool-schema-level checks alone (no prover, solver, or LLM) repairs or rejects 37% (airline) and 13% (retail) of candidates, without which most shipped rules are inoperable. contribution 0.60 0.95 0.54 asserted 1.00
Replaying compiled rules over undefended transcripts flags bindings that refuse legitimate work (a development binding refused 95.9% of task-passing calls); no evaluation binding is flagged. result 0.95 0.91 0.07 asserted 1.00
On $\tau^2$-bench the gate cuts violations of reference-encoded clauses among state-changing calls from 66.3% to 2.6% (airline) and 30.8% to 6.9% (retail), raising airline task success significantly for $2 \le k \le 4$; a 26B on-premise compilation is not significantly worse than hand-written or frontier-compiled rules. result 1.00 0.95 0.31 asserted 1.00
Unlike AgentDojo's shipped defenses, it reaches a zero attack success rate (ASR) on banking, where nine attack families collapse onto three structural rules. result 1.00 0.88 0.14 asserted 1.00
On the other three suites its ASR is at most 3.6%, from goals with no tool call to govern and one write admitted by a binding weaker than its clause; a second agent model, Llama-3.3-70B, reproduces the effect on both benchmarks. result 0.99 0.93 0.12 asserted 1.00
Decisions take microseconds without an LLM call, at a domain-dependent benign-utility cost; compilation runs on-premise on open-weight gemma-4-26B. result 0.75 0.91 0.07 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-09: 1 request, 7,303 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. NOMOS overview. Offline, once per policy, a four-pass compiler turns a natural-language policy into verified rules (airline: 79
Figure 2 of the paper
Figure 2. From extracted candidates to enforced rules (Table 3). Static verification repairs or rejects 37% of raw candidates on airline (29 of 79)
Figure 3 of the paper
Figure 3. The authentication deadlock as extracted, and its repair (retail; the compilation report’s verbatim entry is in Appendix B). The repair
Figure 4 of the paper
Figure 4. (a) Task success under the gate, airline (
Figure 5 of the paper
Figure 5. Attack success against task success on AgentDojo banking (gemma-4-26B-A4B agent), (a) with no attack present and (b) in

Figures are the authors', reproduced from the paper.

Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment