Pith. sign in

REVIEW 4 major objections 6 minor 12 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Deterministic gates cut silent policy violations in LLM agents by 12 points

2026-07-09 11:44 UTC pith:5B7RV4TJ

load-bearing objection Solid, well-scoped empirical result on a real failure mode; main limitation is same-task-set design and evaluation, honestly disclosed. the 4 major comments →

arxiv 2607.07405 v1 pith:5B7RV4TJ submitted 2026-07-08 cs.AI cs.CR

Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents

Vikas Reddy , Sumanth Reddy Challaram , Abhishek Basu This is my paper
classification cs.AI cs.CR
keywords LLM agentstool usepolicy compliancedeterministic verificationruntime enforcementagent reliabilitysilent failurespre-execution gates
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper identifies a failure mode in tool-using LLM agents: when the underlying tool accepts any well-formed call regardless of domain policy (a 'policy-permissive' tool), the agent can execute a forbidden state mutation — cancelling a non-refundable booking, changing an immutable passenger count — that succeeds silently, with no tool error and no signal in the transcript. On the τ²-bench airline benchmark, 78% of failures on a budget model are these silent wrong-state violations. The authors then show that a lightweight layer of deterministic, read-only pre-execution gates — pure predicate functions that inspect the proposed tool call and current database state before allowing a write — can block the violating action and give the agent a structured rejection to re-plan from. A four-gate suite raises gpt-4o-mini task success from 29.6% to 42.0%, reproduces on a disjoint 15-seed set (+12.3pp, P=0.0008), and the lift concentrates where gates actually fire (+19.2pp on firing tasks versus +5.0pp on non-firing tasks, whose confidence interval includes zero). Two negative controls — a self-enforcing retail domain where tools already reject forbidden calls, and BFCL where calls return structured errors — show that gates help only when tools are permissive and add little where tools already self-enforce. The paper explicitly bounds its claim: gates do not guarantee task success, do not solve agent safety generally, and apply only when policy rules are state-decidable. The contribution is a bounded reliability result showing that deterministic verification at the action boundary can convert a class of silent, unrecoverable state corruptions into explicit rejections the agent can recover from.

Core claim

The central object is the deterministic pre-execution gate: a pure function g(tool_name, args, db_state) → {allow, reject} that reads the same operational state the agent can read, checks the proposed mutating call against an explicit encoding of a domain policy rule, and either allows the call or short-circuits it with a structured reason. The key empirical finding is that this mechanism recovers a measurable fraction of silent policy-violating writes on the τ²-bench airline domain, with the lift replicating on disjoint seeds and concentrating where gates fire. The dominant gate (cancellation_eligibility) has 100% precision over 161 fires and carries most of the task-success lift. The paper

What carries the argument

The four-gate suite consists of: (1) cancellation_eligibility — blocks cancellations unless the reservation is eligible under fare, timing, insurance, and flown-segment rules; (2) baggage_allowance — blocks baggage updates that misprice free bags or remove bags contrary to policy; (3) passenger_count — blocks attempts to change the immutable passenger count; (4) must_read_before_write — blocks writes to records the agent has not read in the current session. Gates are fail-open (exceptions allow the original call), add no model calls, and are audited against ground-truth trajectories for precision. The firing-share decomposition (Δ_aggregate ≈ p_fire × Δ_fire) is used to verify that the lift

Load-bearing premise

The gates were designed from the same policy document and evaluated on the same 50-task set they were written against; the replication is over random seeds, not held-out tasks. The dominant gate has 100% precision over 161 fires, which is partial evidence against overfitting, but generalization to unseen tasks within the same domain — or to a second positive domain — remains untested. The paper explicitly searched for a second positive domain and did not find one.

What would settle it

Run the four-gate suite on a held-out set of airline tasks (written after the gates were frozen) and check whether the lift persists. If the lift disappears, the gates were fitted to the specific 50-task distribution rather than to the policy itself.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Tool-using LLM agents deployed in policy-permissive environments should carry deterministic pre-execution gates for any state-decidable policy rule, since the gate provides a per-run guarantee that no stochastic retry or stronger model can match.
  • Agent benchmarks should distinguish loud errors (tool returns an error) from silent wrong states (tool succeeds but violates policy), since only the latter creates an undetectable trust failure.
  • Gate precision must be audited per-policy and per-model: the paper shows one gate with 5% precision that should not be deployed, and another with 100% precision carrying most of the lift.
  • The failure mode may persist at frontier model scales: gpt-5.2 at default reasoning still attempts policy-violating writes, suggesting that deterministic structure remains valuable even as model capability improves.
  • Benchmark construction should include tasks that induce policy violations, not merely tasks with permissive tools — the paper found that even near-miss benchmarks rarely trigger the failure mode.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper identifies silent policy violations on policy-permissive tools as a distinct failure mode in tool-using LLM agents: the agent issues a forbidden write, the tool executes it without error, and the final state is wrong with no signal in the trace. The authors study this in the τ²-bench airline domain, where 78% of budget-model failures are silent wrong-state failures. They evaluate a lightweight intervention—deterministic, read-only pre-execution gates that check proposed mutating tool calls against explicit policy predicates before execution. A four-gate suite raises gpt-4o-mini pass1 from 29.6% to 42.0% (paired bootstrap P=0.0012), reproduces on a disjoint 15-seed set (+12.3pp, P=0.0008), and the lift concentrates where gates fire (+19.2pp on firing tasks vs. +5.0pp on non-firing, CI includes zero). Two negative controls (retail domain, BFCL) bound the mechanism. Suggestive frontier-model evidence (gpt-5.2, +10.4pp, n=5, unreplicated) is reported separately from the central claim.

Significance. The paper makes a bounded but well-executed contribution. Its core experimental design is careful: paired task-level bootstrap with 20,000 resamples, a disjoint 15-seed replication reproducing the lift to within 0.1pp, a firing-stratum decomposition showing the effect concentrates where gates fire, two negative controls bounding the mechanism, and a per-gate precision audit. The gates are deterministic predicates written from the domain policy document, not fitted parameters, and the success metric comes from the benchmark's own final-state evaluator, not from gate logic—so the central measurement is not circular. The paper is commendably honest about its limitations, explicitly listing nine of them including the single-domain scope, the unreplicated frontier result, and the same-task-set evaluation. The falsifiable prediction that gates help only where tools are policy-permissive and policies are state-decidable is a genuine strength. The contribution is appropriate for a non-archival workshop venue.

major comments (4)
  1. §3.3, Table 6: The headline four-gate suite was selected from a candidate set of five gates based on budget-tier performance on the same 50-task set, and the dominant gate (cancellation_eligibility) carries the entire net lift (removal Δ = -2, while all other gates have positive removal Δ). This means the paper's central empirical result effectively reduces to a single gate's performance on a single domain's 50 tasks. The 100% precision over 161 fires is strong evidence against false blocking, but it does not address whether the gate's policy encoding is complete rather than tuned to the specific violation patterns in this task distribution. The 15-seed replication addresses sampling noise but not task-level overfitting, since the same 50 tasks appear in every seed. The authors acknowledge this in Limitation 8, but the paper would be substantially strengthened by either (a) releasing the
  2. §3.3, footnote 1: The gate predicate code is not released ('Code and gate predicates will be released upon publication'). This is a significant gap for a paper whose central claim is that deterministic gates encoding domain policy rules recover failures. Without the gate code, the claim that gates encode 'explicit encoding of a domain policy rule' (§3.3) is unverifiable from the paper alone. The distinction between a gate that encodes the full cancellation policy (fare class, timing, insurance, flown segments) and one that encodes a subset of conditions selected because they fire on these tasks is load-bearing for the generalization claim. Releasing the gate predicates, at minimum for the dominant cancellation_eligibility gate, should be a condition of acceptance.
  3. §5.1, Limitation 8: The paper's own framing acknowledges that 'a held-out-task evaluation with frozen gates is the cleaner test.' Given that the four-gate suite was selected on the same 50 tasks, the current evaluation cannot distinguish between 'deterministic gates recover a recurring failure class' and 'deterministic gates selected on this task distribution perform well on this task distribution.' The firing-stratum decomposition (§5.2) provides partial evidence against circularity (the lift concentrates where gates fire), but it does not address whether the gate selection itself is fitted to observed violations. If a held-out-task split is feasible within the existing 50-task set (e.g., select gates on 25 tasks, evaluate on the remaining 25), it would substantially de-risk the central claim. If not feasible due to task count, the authors should more prominently flag this as the single
  4. §5.5, Table 6: The baggage_allowance gate has 5% precision (2 true blocks, 40 false blocks) and its removal improves pass1 by +3. This gate is part of the headline four-gate suite (Table 1). Including a gate with 5% precision in the headline configuration, even if the paper reports the four-gate suite rather than a post-hoc optimum, weakens the claim that the suite represents a principled policy encoding. The paper should either justify why this gate is retained in the headline suite despite 95% false-blocking, or report results for a three-gate suite (excluding baggage_allowance) as an additional configuration.
minor comments (6)
  1. §5.2, Table 3: The non-firing stratum shows +5.0pp movement (P=0.18). The paper correctly notes the CI includes zero, but the point estimate is non-trivial. A brief discussion of potential mechanisms (e.g., rejection messages changing agent behavior on subsequent calls within the same trial) would strengthen the analysis.
  2. §5.3: The frontier-model result uses gpt-5.2 at 'harness default reasoning level' with n=5 per task. The P-value (0.020) is computed from 250 trials, but with only 5 trials per task across 50 tasks, the per-task variance is substantial. The paper should note that the effective sample size for the paired test is 50 tasks, not 250 trials.
  3. §6.1: The retail negative control has an encoding bug that blocked corrected retries, producing a -4.7pp point estimate. A negative control with a known implementation bug is difficult to interpret. The paper should either fix the bug and re-run, or more clearly flag that this control is contaminated and the negative result should be read with caution.
  4. Table 1: The policy intuition column for must_read_before_write says 'Block writes to records the agent has not read in the current session.' This is a procedural rather than policy-based rule. Clarifying whether this is derived from the τ²-bench policy document or is an engineering heuristic would help readers.
  5. §3.4: The firing-share decomposition (Δaggregate ≈ p_fire × Δ_fire) is a useful framing, but the approximation sign is not formally justified. A brief note on when this decomposition is exact vs. approximate would help.
  6. §5.6: The canonical deceptive task #48 is a compelling example, but reporting 16/16 across model/settings combinations without specifying which combinations were tested makes it hard to assess. A small table would help.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major comments concerning (1) the central result reducing to a single gate on 50 tasks and the need for held-out-task evaluation, (2) the gate predicate code not being released, (3) the same-task-set evaluation and the feasibility of a held-out-task split, and (4) the baggage_allowance gate's 5% precision and its inclusion in the headline suite. We agree with the substance of all four comments and will revise accordingly: we will release the gate predicate code (including the dominant cancellation_eligibility gate) as a condition of the camera-ready, we will add a three-gate configuration (excluding baggage_allowance) as an additional reported result, and we will more prominently flag the held-out-task limitation. We also provide a substantive response on the feasibility of a held-out-task split within the existing 50-task set.

read point-by-point responses

  1. Referee: §3.3, Table 6: The headline four-gate suite was selected from a candidate set of five gates based on budget-tier performance on the same 50-task set, and the dominant gate (cancellation_eligibility) carries the entire net lift (removal Δ = -2, while all other gates have positive removal Δ). This means the paper's central empirical result effectively reduces to a single gate's performance on a single domain's 50 tasks. The 100% precision over 161 fires is strong evidence against false blocking, but it does not address whether the gate's policy encoding is complete rather than tuned to the specific violation patterns in this task distribution. The 15-seed replication addresses sampling noise but not task-level overfitting, since the same 50 tasks appear in every seed. The authors acknowledge this in Limitation 8, but the paper would be substantially strengthened by either (a) releasing the

    Authors: The referee is correct that the central empirical result concentrates in a single gate, and we agree this is an important framing issue. We want to clarify one point about the removal Δ values in Table 6: the negative removal Δ for cancellation_eligibility (-2) means that removing this gate from the suite decreases pass1 by 2 successful trials, while the positive removal Δ values for the other gates mean that removing them increases pass1—i.e., those gates are net-negative on this distribution. This is consistent with the referee's observation that the lift effectively reduces to the cancellation_eligibility gate. We state this openly in §3.3 and §5.5, but we agree the paper should frame the central result more precisely as 'a single high-precision gate carries the lift, and the suite extends policy coverage without being load-bearing on this distribution.' Regarding the concern about whether the gate's policy encoding is complete rather than tuned to specific violation patterns: this is exactly why releasing the gate predicate code is important, and we address this in our response to the second comment. The 100% precision over 161 fires is evidence against false blocking but, as the referee correctly notes, does not by itself establish that the encoding is complete rather than fitted. We accept this characterization and will revise the framing accordingly. revision_made = 'partial' revision: partial

  2. Referee: §3.3, footnote 1: The gate predicate code is not released ('Code and gate predicates will be released upon publication'). This is a significant gap for a paper whose central claim is that deterministic gates encoding domain policy rules recover failures. Without the gate code, the claim that gates encode 'explicit encoding of a domain policy rule' (§3.3) is unverifiable from the paper alone. The distinction between a gate that encodes the full cancellation policy (fare class, timing, insurance, flown segments) and one that encodes a subset of conditions selected because they fire on these tasks is load-bearing for the generalization claim. Releasing the gate predicates, at minimum for the dominant cancellation_eligibility gate, should be a condition of acceptance.

    Authors: We fully agree. The gate predicate code is essential for verifying the claim that gates encode domain policy rules rather than fitting to observed violation patterns. We will release the complete gate predicate code, including the cancellation_eligibility gate, as a condition of acceptance. To be specific about what the cancellation_eligibility gate encodes: it checks fare class (basic economy is non-cancellable), timing (cancellation must occur before the first segment is flown), insurance status (travel insurance enables cancellation in certain fare classes), and flown-segment status (any segment already flown blocks cancellation). These conditions are drawn directly from the τ²-bench airline policy document, not from observed violation patterns. Releasing the code will allow the referee and readers to verify this directly. We will also release the predicates for all other gates in the suite. revision_made = 'yes' revision: yes

  3. Referee: §5.1, Limitation 8: The paper's own framing acknowledges that 'a held-out-task evaluation with frozen gates is the cleaner test.' Given that the four-gate suite was selected on the same 50 tasks, the current evaluation cannot distinguish between 'deterministic gates recover a recurring failure class' and 'deterministic gates selected on this task distribution perform well on this task distribution.' The firing-stratum decomposition (§5.2) provides partial evidence against circularity (the lift concentrates where gates fire), but it does not address whether the gate selection itself is fitted to observed violations. If a held-out-task split is feasible within the existing 50-task set (e.g., select gates on 25 tasks, evaluate on the remaining 25), it would substantially de-risk the central claim. If not feasible due to task count, the authors should more prominently flag this as the single

    Authors: We agree that a held-out-task split would substantially de-risk the central claim, and we considered this design. The practical difficulty is that with 50 tasks and the dominant gate firing on 26 of them, a 25/25 split yields very few firing tasks in the held-out set—roughly 13—making the estimate high-variance and difficult to interpret. However, the referee's point is well-taken that the current evaluation cannot fully distinguish between 'gates recover a recurring failure class' and 'gates selected on this distribution perform well on this distribution.' We will address this in two ways in the revision. First, we will more prominently flag the task-overfitting concern as the primary limitation, moving it from its current position as Limitation 8 to a more prominent location in the limitations discussion. Second, we will add the following argument: the cancellation_eligibility gate encodes the full cancellation policy from the domain policy document (fare class, timing, insurance, flown segments), not a subset of conditions selected for firing frequency. Once the gate code is released, readers can verify that the predicate is a complete encoding of the policy rule, not a fitted classifier. This does not fully substitute for a held-out-task evaluation, but it does address the specific concern about whether the gate encoding is fitted to observed violations. We acknowledge that a held-out-task split with adequate power remains the cleaner test and will state this explicitly. revision_made = 'partial' revision: partial

  4. Referee: §5.5, Table 6: The baggage_allowance gate has 5% precision (2 true blocks, 40 false blocks) and its removal improves pass1 by +3. This gate is part of the headline four-gate suite (Table 1). Including a gate with 5% precision in the headline configuration, even if the paper reports the four-gate suite rather than a post-hoc optimum, weakens the claim that the suite represents a principled policy encoding. The paper should either justify why this gate is retained in the headline suite despite 95% false-blocking, or report results for a three-gate suite (excluding baggage_allowance) as an additional configuration.

    Authors: The referee is correct. The baggage_allowance gate has 5% precision (2 true blocks, 40 false blocks) and its removal improves pass1 by +3, meaning it is net-negative on this distribution. Including it in the headline four-gate suite is not justified by the data. We retained it because the four-gate suite was fixed on the budget tier before the per-gate audit was conducted, and we reported the pre-audit configuration rather than a post-hoc optimum. However, the referee's point is well-taken: reporting a headline configuration that includes a gate with 95% false-blocking weakens the claim of principled policy encoding. We will add a three-gate configuration (excluding baggage_allowance) as an additional reported result in the revision. We will retain the four-gate suite in the paper for transparency—because it is the configuration we actually evaluated—but will clearly flag that the three-gate configuration is the better-performing and more principled suite, and will report its pass1, replication, and stratification results alongside the four-gate results. We will also revise §3.3 to explicitly state that the baggage_allowance gate has poor precision and should be treated as a candidate for revision before broader deployment, rather than as part of a principled suite. revision_made = 'yes' revision: yes

Circularity Check

0 steps flagged

No circularity found: gates are deterministic policy predicates, not fitted parameters; success metric comes from the benchmark's own evaluator; firing-stratum decomposition provides a non-circular mechanistic check.

full rationale

The paper's central claim is that deterministic pre-execution gates encoding domain policy rules recover silent policy-violation failures. Walking the derivation chain: (1) Gates are defined as deterministic predicates g(tool_name, args, db_state) → {allow, reject} (§3.2), written from the τ²-bench airline policy document. They are not fitted parameters—they are rule encodings. (2) The success metric (pass1) comes from the benchmark's own final-state evaluator: 'A trial is counted as successful when reward_info.reward > 0' (§4.1), not from gate logic. (3) The firing-stratum decomposition (§5.2, Table 3) provides a non-circular mechanistic check: if the lift were an artifact unrelated to gates, non-firing tasks would move as much as firing tasks. They do not (+19.2pp firing vs +5.0pp non-firing, CI includes zero). (4) The per-gate precision audit (§5.5, Table 6) checks blocked calls against ground-truth trajectories—an external standard. (5) The four-gate headline suite was selected from five candidates on the budget tier, but the paper is transparent about this selection and reports the full audit including the dropped fifth gate. (6) No self-citation is load-bearing: the paper cites external work for the benchmark (τ²-bench [1]), related enforcement systems [3,5,9,10,12], and comparison benchmarks (BFCL [4], WorkBench [8]). The concern that gates were written from the same policy document and evaluated on the same 50-task set is a generalization/external validity concern (acknowledged in Limitation 8), not a circularity: the gates do not define the success metric, the success metric does not define the gates, and no prediction reduces to its input by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities, particles, forces, or dimensions. The gates are standard Python predicates over tool arguments and database state. The free parameters are the gate suite composition and the hand-written predicate logic, both selected on the budget tier and evaluated on the same task set. The axioms are domain assumptions about the benchmark environment (A2-A5 from §2.3) plus one engineering choice (fail-open design). No invented entities are postulated.

free parameters (2)
  • Gate suite composition (four gates selected from five candidates) = cancellation_eligibility, must_read_before_write, baggage_allowance, passenger_count (basic_economy excluded)
    Selected on the budget tier (gpt-4o-mini) and held fixed across model tiers. The selection is justified by the per-gate removal audit (Table 6), but the choice of which gates to write was informed by the same task set used for evaluation.
  • Gate predicate logic (per-gate policy encoding) = Not numerically fitted; hand-written Python predicates from the airline policy document
    Each gate encodes a policy rule as a deterministic predicate. The cancellation_eligibility gate's logic is the most impactful and is described at the intuition level in Table 1, but the exact implementation is not provided.
axioms (5)
  • domain assumption Policy-permissive tools (A2): tools execute any well-formed call even when the state transition is forbidden by domain policy
    Invoked in §1.1 and §3.1 as the root cause. This is a property of the τ²-bench airline domain, not a universal property of all tool environments. The negative controls (§6.1, §6.2) test this assumption.
  • domain assumption State-decidable policy (A3): the policy rule can be expressed as a deterministic predicate over current state and call arguments
    Invoked in §2.3 and §3.2. The gate mechanism requires this. The authors note (Limitation 6) that policies requiring ambiguity resolution, legal interpretation, or human judgment need richer mechanisms.
  • domain assumption Final-state evaluation (A4): the evaluator detects silent wrong states
    Invoked in §2.3. The τ²-bench evaluator checks the final database state against ground truth. Without this, silent wrong-state failures would be undetectable.
  • domain assumption Violation-inducing tasks (A5): the task distribution actually causes agents to violate the policy
    Invoked in §2.3 and §6.4. The authors searched for a second positive domain and did not find one satisfying this criterion, noting WorkBench as a near-miss.
  • ad hoc to paper Fail-open gate design: if a gate raises an exception, the original call is allowed
    Invoked in §3.2. This is an engineering choice that avoids introducing false blocks from gate implementation failures, but it means gate crashes silently revert to the unguarded path.

pith-pipeline@v1.1.0-glm · 13542 in / 3596 out tokens · 303903 ms · 2026-07-09T11:44:57.803631+00:00 · methodology

0 comments
read the original abstract

Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully. In policy-permissive environments, a tool may execute any well-formed call even when the corresponding state transition is forbidden by domain policy. The result is a silent wrong state (a booking cancelled, a passenger count changed, a claim acted on without verification) that neither the tool nor the agent's self-report exposes. We study this failure mode in the $\tau^2$-bench airline domain. On a budget agent, 78% of observed failures are silent wrong-state failures with no tool error, and the aggregate failure rate is reproducible across disjoint seeds, not sampling noise. We then evaluate a lightweight intervention: deterministic, read-only pre-execution gates that inspect the proposed call and current state before allowing a write. A four-gate suite raises full-benchmark success from 29.6% to 42.0% on gpt-4o-mini (+12.4pp; paired task-level bootstrap P=0.0012), and the lift reproduces on a disjoint 15-seed set (+12.3pp; P=0.0008). The effect is concentrated where the gates fire: on the 26/50 firing tasks, success rises by +19.2pp, while movement on the 24 non-firing tasks does not exclude zero. Two negative controls (a self-enforcing retail domain and BFCL) bound the mechanism: gates help when tools are policy-permissive and add little where tools already self-enforce. As suggestive evidence, not a central claim, the same failure mode persists at the frontier: gpt-5.2 at default reasoning still attempts policy-violating writes, and the same suite improves success from 61.2% to 71.6% (+10.4pp; P=0.020; n=5, no replication). The contribution is a bounded evaluation and reliability result: deterministic gates do not guarantee task success, but they can deterministically prevent a known class of silent policy-violating writes at the action boundary.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 7 internal anchors

  1. [1]

    $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

    V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. 2025. 𝜏 2-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Mohammadi, Y

    M. Mohammadi, Y. Li, J. Lo, and W. Yip. 2025. Evaluation and Benchmarking of LLM Agents: A Survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM

    work page 2025

  3. [3]

    Formal Policy Enforcement for Real-World Agentic Systems

    N. Palumbo, S. Choudhary, J. Choi, G. Amir, P. Chalasani, and S. Jha. 2026. Formal Policy Enforcement for Real-World Agentic Systems. arXiv:2602.16708

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  4. [4]

    S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InProceedings of the 42nd International Conference on Machine Learning (PMLR, Vol. 267). PMLR, 48371–48392

    work page 2025

  5. [5]

    L. Qiu, Y. Ye, Z. Gao, X. Zou, J. Chen, Z. Gui, W. Huang, X. Xue, W. Qiu, and K. Zhao. 2025. Blueprint First, Model Second: A Framework for Deterministic LLM Workflow. arXiv:2508.02721

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails

    T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen. 2023. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. arXiv:2310.10501

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. 2023. Re- flexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Styles, S

    O. Styles, S. Miller, P. Cerda-Mardini, T. Guha, V. Sanchez, and B. Vidgen. 2024. WorkBench: A Benchmark Dataset for Agents in a Realistic Workplace Setting. InConference on Language Modeling

    work page 2024

  9. [9]

    H. Wang, C. M. Poskitt, and J. Sun. 2025. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. arXiv:2503.18666

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Winston, C

    C. Winston, C. Winston, and R. Just. 2026. Solver-Aided Verification of Policy Compliance in Tool-Augmented LLM Agents. arXiv:2603.20449

    work page arXiv 2026

  11. [11]

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. 2024. 𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    A. Yuan, Z. Su, and Y. Zhao. 2026. AEGIS: No Tool Call Left Unchecked, A Pre-Execution Firewall and Audit Layer for AI Agents. arXiv:2603.12621 7

    work page arXiv 2026