Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that an LLM agent's skill menu should be filtered by deterministic executability checks before the model sees it, and reports that in production this filtering removed 59.4% of post-semantic candidates and changed model se

desk verdict A careful, honestly scoped production report: the funnel numbers add up and the deterministic-gating pattern is worth knowing, but the soundness claim rests on an asserted invariant the paper never directly demonstrates. read the letter →

arxiv 2608.01050 v1 pith:VD2I22LL submitted 2026-08-02 cs.AI cs.CLcs.SE

Ortal Ashkenazi , Vitalii Kloz , Mykhailo Ulianchenko This is my paper
classification cs.AIcs.CLcs.SE
keywords LLMagentstoolselectionskillroutingdeterministicgatingexecutabilityexit-conditioninversionretrieval-augmenteduseproductiondeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that semantic relevance alone is the wrong criterion for deciding which skills an LLM agent can offer: a skill may match a user's topic yet be impossible to run for the current account, and offering it wastes context and can alter what the model selects. The proposed architecture separates three decisions — topical recall, deterministic executability, and conversational activation — with a gate that evaluates each skill's own hard exit predicates against fresh authoritative state before the skill description enters the model context. The central claim is contract-relative soundness: because the gate and the skill share the same exit predicates, a blocked candidate could not have completed under the same state, so gating removes actions that are known-impossible rather than merely predicted unhelpful. In a production analysis of 756,641 messages, the gate removed 59.4% of post-semantic candidate pairs and 59.1% of skill-description tokens, and a counterfactual replay in which the gate was removed found the model selecting a production-blocked skill in 7.8% of risk-enriched conversations. The authors frame this as a distinct interface problem: non-executable exposure is a property of the account state, not of topical relevance, and should therefore be handled by deterministic infrastructure.

What carries the argument

The central mechanism is exit-condition inversion: for each multi-step skill, the authors identify the account- or site-state predicates that cause the skill's own procedure to stop before completion, and the gate evaluates those same predicates against the current authoritative state. Because the gate and the skill use predicate-equivalent exit conditions, blocking is sound by construction relative to the implemented skill contract — the same condition that stops the skill stops its offer. The mechanism is deliberately deterministic: business-state checks are hoisted out of the stochastic prompt, making candidate exposure reproducible and testable, while the LLM retains only conversational

What would settle it

For a sample of skill–message pairs that the gate removed, invoke the skill under the same authoritative state and record whether it actually exits before completing; a nonzero completion rate would falsify the soundness-by-construction claim. A simpler check: run parity tests over shared state fixtures, including boundary, missing-state, and stale-state cases, and look for gate decisions that disagree with the skill's own exit behavior.

Watch

Extended reading notes

Core claim

The discovery is that executability can be removed from the stochastic LLM decision entirely by inverting each skill's internal abort conditions into deterministic gate predicates. For each skill $s$, let $E_s(x)$ indicate that at least one of its hard exit conditions holds in authoritative state $x$; the visible set passed to the agent is $V(m)=\{s \in C(m) : \neg E_s(x_{m,s})\}$, where $C(m)$ is the candidate set from a state-blind semantic stage. Provided predicate parity is maintained and both the gate and the skill observe sufficiently fresh authoritative state, the gate blocks $s$ exactly when the skill would exit before completion under that state, so a blocked candidate cannot comple

Load-bearing premise

The gate is sound only if each gate predicate and its skill's own exit logic stay equivalent and both read the same sufficiently fresh authoritative state; if that parity drifts or the state is stale, a blocked skill is not guaranteed to have failed.

Editorial extensions

If this is right

  • Semantic retrieval alone leaves a large non-executable residue: 59.4% of post-semantic candidate pairs were blocked by state checks, so retrieval-based tool selection that omits them still exposes many skills that cannot run.
  • Removing blocked skill descriptions before the model sees them shrinks routing context by 90.5% relative to exposing all ten skills to every message, with executability decided outside the model.
  • Gating changes model behavior, not just context size: in a risk-enriched cohort replayed without the gate, the model chose a production-blocked skill in 7.8% of conversations.
  • Candidate exposure becomes reproducible: a deterministic function of authoritative state can be regression-tested, versioned, and rolled back, whereas a prompt-based constraint competes with intent and history in the same stochastic context.
  • The guarantee is limited by design: passing the gate does not predict task success, and predicate drift, stale state, or backend-contract drift would break soundness, so maintaining parity is a release invariant rather than a one-time property.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same exit-condition-inversion pattern were applied to a larger library, the key cost is not inference but maintenance: every skill change becomes a coordinated predicate change, so libraries with many fast-moving skills would need automated parity checks to keep the soundness guarantee.
  • The 7.8% counterfactual selection rate is conditional on a risk-enriched cohort, but it suggests a directly testable extension: run the same replay with downstream tool execution logged, and compare whether gate-blocked skills, when invoked, actually fail at the predicted exit point.
  • The same architecture could absorb authorization and policy checks as hard stop predicates, turning confused-deputy-style risks into deterministic blocks before the LLM can be prompted to act; this extends the paper's scope beyond executability into security.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes a deployed three-stage skill-selection pipeline for Wix's Helpmate customer-care assistant. Stage 1 performs state-blind semantic matching over a ten-skill domain family; Stage 2 applies a deterministic executability gate that removes candidates whose own exit predicates hold in the current authoritative state; Stage 3 lets the LLM choose among the remaining skills. The authors claim a contract-relative soundness guarantee: if predicate parity is maintained and both gate and skill observe fresh authoritative state, a blocked skill would necessarily exit before completion. They report production results on 756,641 messages and 267,612 conversations: the gate removed 59.4% of post-semantic skill-message pairs and 59.1% of skill-description tokens, giving a combined 90.5% context reduction. A counterfactual replay of 1,000 risk-enriched conversations showed the model selected a production-blocked skill in 7.8% of cases when gating was removed. The paper also discusses deployment lessons, failure modes, and limitations.

Significance. The paper addresses a genuine and underappreciated problem in LLM-agent skill selection: semantic relevance does not imply executability, and exposing non-executable skills can mislead the model and create security risk. The proposed deterministic-gating architecture is simple, interpretable, and, if the parity condition holds, provides a stronger guarantee than learned classifiers. The production-scale funnel numbers are large and the internal arithmetic is consistent. The counterfactual replay is a useful behavioral check that goes beyond context-size metrics. The paper is honest about its scope and limitations, and it explicitly avoids overclaiming downstream tool-execution or customer-outcome effects. The main weakness is that the central soundness claim rests on engineering controls—predicate parity and state freshness—that are described but not empirically evidenced; the reported funnel metrics cannot detect false blocks. If the authors can provide even a modest direct validation of parity and blocked-skill behavior, the contribution would be solid.

major comments (3)
  1. [§4.2, §4.4, §6.1–6.2] The soundness guarantee is conditional on predicate parity and fresh authoritative state. The paper asserts these conditions are maintained by regression tests and monitoring, but no parity-test results, staleness statistics, or direct verification that blocked skills actually exit are reported. The funnel metrics in §5.4 count only pairs and tokens; they are logically independent of parity and therefore cannot detect false blocks if predicate drift has occurred. As written, the central claim 'every blocked candidate would be unable to complete' is an unverified engineering assumption. Please add direct evidence: parity-test pass rates over the observation window, an audit sample in which blocked candidates are force-invoked in a sandbox and confirmed to exit, or reconciliation of gate decisions with subsequent skill exit behavior.
  2. [§5.6] The counterfactual replay measures model selection, not tool execution or failure severity. The 7.8% figure thus demonstrates that removing the gate changes model choices, but it does not test whether the production-blocked skills would actually have failed if invoked. The abstract and conclusion phrase the result as 'prevents non-executable candidates from influencing model selection,' which conflates the influence measurement with the unverified non-executability claim. If execution-level validation cannot be added, the paper should state more prominently that the behavioral impact is conditional on the assumed parity, and the replay should be described only as measuring selection, not soundness.
  3. [§4.2] The exit-condition-inversion method is listed as a contribution but is not specified concretely. The paper gives an illustrative connect_domain example and a high-level lifecycle (§6.1), but no anonymized predicate examples, no explanation of how predicates are extracted from multi-step skill implementations, and no formal or operational definition of what a parity test checks. Without this detail, the 'same exit predicates' guarantee is not reproducible or independently assessable. Please include at least one representative predicate definition and the inversion/parity-test procedure, or an appendix with the method.
minor comments (4)
  1. [§5.2] The indicator notation '⊮[E_s]' is non-standard and likely a typographical artifact; use the standard indicator '𝟙[...]' or '1{...}'.
  2. [§1 and §3.2] The connect_domain example is introduced as a concrete skill in §1 but later described as illustrative and 'not tied to any reported per-skill row' (§3.2). Please make the status of this example consistent across the paper.
  3. [§6.2] The discussion of fail-open versus fail-closed policies for unavailable state is useful, but the paper does not state which policy the deployed system used. Since this affects the interpretation of the funnel numbers, a one-sentence clarification would help.
  4. [General] The paper is carefully scoped and the metric discipline is good; however, the phrase 'sound by construction' in the abstract and contributions could be softened to 'sound relative to the maintained skill contract' to align with the conditional nature of the guarantee.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the soundness claim is explicitly definitional and the empirical funnel and replay measurements are independent of it.

full rationale

The derivation chain is self-contained. The gate's contract (Section 4.4, visible set V(m) = {s in C(m) : not E_s(x)}) defines E_s as the skill's own exit conditions, so the statement that every blocked candidate would exit under the same state is an explicitly acknowledged consequence of the implemented contract, not an empirical prediction produced from fitted data. The paper labels this 'soundness by construction' and conditions it on predicate parity and fresh authoritative state, which are engineering assumptions rather than hidden inputs to the reported results. The headline numbers—59.4% of post-semantic candidate pairs removed, 59.1% of skill-description tokens saved, and the 7.8% counterfactual selection rate—are direct measurements over production logs and a replayed cohort; they do not rely on the soundness guarantee being verified. The counterfactual replay is expressly said to measure model selection, not tool execution. No parameters are fitted, no load-bearing self-citation is present, and the related citations to Wix work [6,7] are contextual only. The lack of published parity tests or predicate definitions is a verification gap and a correctness risk, but not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numerical parameters or new entities are introduced. The central claim rests on four operational axioms: predicate parity, state freshness, deterministic hard-stop conditions, and the semantic stage's state-blind recall behavior. These axioms are stated clearly and defended as engineering controls, but their satisfaction is not independently demonstrated.

assumptions (4)
  • domain assumption Predicate parity: gate predicates and skill exit branches are logically equivalent at all times.
    Invoked in Sections 4.2 and 4.4; the contract-relative soundness guarantee and every blocked-skill claim depend on this equivalence, which is asserted to be maintained by regression tests but no test evidence is shown.
  • domain assumption State freshness: the gate reads the same fresh authoritative account state the skill would observe at execution.
    Invoked in Sections 4.2, 4.4, and 7; stale or partial state would break the exact-blocking guarantee.
  • domain assumption Hard exit conditions are expressible as deterministic state predicates; ambiguous intent is excluded.
    Sections 3.2 and 4.2; this restricts the gate's scope and is built into the formal contract, not demonstrated empirically.
  • domain assumption The semantic stage is recall-oriented and state-blind, passing all ten family skills for every matched message.
    Sections 4.1 and 4.4; the funnel arithmetic depends on C(m)=S for matched messages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale." pith.science (2026). https://pith.science/paper/VD2I22LL

@misc{pith2026260801050,
  author       = {Pith},
  title        = {Pith review of: Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VD2I22LL}},
  note         = {Machine review of arXiv:2608.01050}
}
read the original abstract

Production LLM agents that select from large skill libraries face a limitation that semantic relevance alone cannot resolve: a skill may match a user's topic yet be impossible to execute in the current account state. We present a deployed three-stage selection pipeline for Helpmate, Wix's customer-care assistant. First, a recall-oriented semantic matcher identifies messages related to a ten-skill domain family without consulting account state. Second, a deterministic executability gate removes candidates whose internal hard-stop conditions hold. Because the gate and the skill evaluate the same exit predicates, every blocked candidate would be unable to complete under the same account state, provided predicate parity is preserved and both checks observe fresh authoritative state. Finally, the LLM decides whether to invoke one of the remaining candidates. In a post-launch production analysis of 756.6K user messages across 267.6K conversations, semantic matching retained 174,927 messages (23.1%). Within this matched stream, the gate removed 1,039,462 of 1,749,270 skill-message pairs (59.4%), saving 228.8 million skill-description tokens -- 59.1% of the post-semantic skill-description footprint. Together, semantic matching and executability gating reduced skill-description context by 90.5% relative to exposing all ten skills to every message. To test whether this pruning affects model behavior rather than context size alone, we replayed a risk-enriched cohort of 1,000 conversations with all ten skills exposed. The model selected a production-blocked skill in 78 conversations (7.8%). This counterfactual result shows that deterministic gating prevents non-executable candidates from influencing model selection, while not claiming downstream tool execution or customer-outcome effects.

Figures

Figures reproduced from arXiv: 2608.01050 by the authors.

Figure 1
Figure 1. Three-stage selection pipeline. Semantic matching identifies the domain topic family; the deterministic gate removes [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Skill-description token funnel. Semantic matching removes 76.9% of the all-message baseline; the gate then removes [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Anonymous distributions across the ten skills. Points denote skills without identifiers; boxes summarize post-semantic [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 15 canonical work pages

  1. [1]

    Agarwal, G

    A. Agarwal, G. Siyan, Y. Pandya, J. Singh, A. Nambi, and A. Awadallah. 2026. Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use. InProceedings of the 43rd International Conference on Ma- chine Learning (ICML). arXiv:2603.03205

    arXiv 2026

  2. [2]

    R. S. Babu and L. G. Iyer. 2026. Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agents. arXiv:2606.07904

    arXiv 2026

  3. [3]

    R. S. Babu and L. G. Iyer. 2026. ToolChoiceConfusion: Causal Minimal Tool Filtering for Reliable LLM Agents. arXiv:2606.06284

    arXiv 2026

  4. [4]

    R. S. Babu and L. G. Iyer. 2026. ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents. arXiv:2606.15508

    arXiv 2026

  5. [5]

    R. S. Babu and R. Shukla. 2026. GIST-CMTF: Goal-State Inference for Causal Minimal Tool Filtering in LLM Agents. arXiv:2606.16813

    arXiv 2026

  6. [6]

    Cohen, L

    D. Cohen, L. Burg, S. Pykhnivskyi, H. Gur, S. Kovynov, O. Atzmon, and G. Barkan

    work page

  7. [7]

    Cohen, T

    D. Cohen, T. Houri, L. Burg, and G. Barkan. 2025. RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines. arXiv:2505.13538

    work page arXiv 2025

  8. [8]

    R. E. Fikes and N. J. Nilsson. 1971. STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving.Artificial Intelligence2, 3–4 (1971), 189–208. doi:10.1016/0004-3702(71)90010-5

    doi:10.1016/0004-3702(71)90010-5 1971

Show all 31 references
  1. [9]

    Gan and Q

    T. Gan and Q. Sun. 2025. RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation. arXiv:2505.03275

    2025 arXiv

  2. [10]

    D. Guo, Q. Liu, D. Liu, Q. Ren, S. Shao, T. Qiu, H. Li, Y. R. Fung, Z. Ba, J. Dai, et al. 2026. Are Your Agents Upward Deceivers?. InProceedings of the 43rd International Conference on Machine Learning (ICML). arXiv:2512.04864

    2026

  3. [11]

    Gupta, K

    A. Gupta, K. Rossell, E. Alcobaça, J. C. L. Pacheco, C. B. de Lima, S. Tang, L. P. Rabachini, L. Moneda, H. Fei, D. Silva, and R. Ramanath. 2026. Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework. In Proceedings of the 32nd ACM SIGKDD Confe...

    2026

  4. [12]

    Huang and S

    S. Huang and S. Ontañón. 2022. A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. InProceedings of the Thirty-Fifth International Florida Artificial Intelligence Research Society Conference (FLAIRS)

    2022

  5. [13]

    Huang, D

    T. Huang, D. Jung, V. Kumar, M. Kachuee, X. Li, P. Xu, and M. Chen. 2024. Planning and Editing What You Retrieve for Enhanced Tool Learning. InFindings of the Association for Computational Linguistics: NAACL. 975–988. doi:10.18653/v1/ 2024.findings-naacl.61

    2024 doi

  6. [14]

    K. Kate, T. Pedapati, K. Basu, Y. Rizk, V. Chenthamarakshan, S. Chaudhury, M. Agarwal, and I. Abdelaziz. 2025. LongFuncEval: Measuring the Effectiveness of Long Context Models for Function Calling. arXiv:2505.10570

    2025 arXiv

  7. [15]

    M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li. 2023. API- Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. InProceedings of EMNLP. 3102–3116. doi:10.18653/v1/2023.emnlp-main.187

    2023 doi

  8. [16]

    J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, Z. Wang, and R. Pang. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. InFindings of the Association for Computational Linguistic...

    2025

  9. [17]

    Lumer, V

    E. Lumer, V. K. Subbiah, J. A. Burke, P. H. Basavaraju, and A. Huber. 2024. Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases. arXiv:2410.14594

    2024 arXiv

  10. [18]

    Molfetta, G

    L. Molfetta, G. Frisoni, N. Monaldini, and G. Moro. 2025. PORTS: Preference- Optimized Retrievers for Tool Selection with Large Language Models. InPro- ceedings of EMNLP. 10007–10030. doi:10.18653/v1/2025.emnlp-main.507

    2025 doi

  11. [19]

    Y. Ning, J. Jones, Z. Zhang, C. Ye, W. Ruan, J. Li, R. Gupta, and H. Sun. 2026. When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents. InProceedings of the 43rd International Conference on Machine Learning (ICML). arXiv:2602.08995

    2026 arXiv

  12. [20]

    S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez

  13. [21]

    Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. InInternational Conference on Learning Representations (ICLR)

    2024

  14. [22]

    InProceedings of the 42nd In- ternational Conference on Machine Learning (Proceedings of Machine Learning Research, Vol

    The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InProceedings of the 42nd In- ternational Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). 48371–48392

  15. [23]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettle- moyer, N. Cancedda, and T. Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. InAdvances in Neural Information Processing Systems 36 (NeurIPS)

    2023

  16. [24]

    Reddy, S

    V. Reddy, S. R. Challaram, and A. Basu. 2026. Reason Less, Verify More: Deter- ministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents. arXiv:2607.07405

    2026 arXiv

  17. [25]

    K. Song, A. Jayarajan, Y. Ding, Q. Su, Z. Zhu, S. Liu, and G. Pekhimenko. 2025. Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Fail- ures in LLM Agents. arXiv:2508.19504

    2025 arXiv

  18. [26]

    V. Siu, J. He, K. Montgomery, Z. Wang, C. Wang, and D. Song. 2026. Position: Agent Security Needs Redefinition through a Holistic Framework. InProceedings of the 43rd International Conference on Machine Learning (ICML)

    2026

  19. [27]

    Y. Sun, J. Liu, D. Kroening, and J. Xue. 2026. Agentic Model Checking. arXiv:2605.21434

    2026 arXiv

  20. [28]

    South, S

    T. South, S. Marro, T. Hardjono, R. Mahari, C. D. Whitney, A. Chan, and A. Pent- land. 2025. Position: AI Agents Need Authenticated Delegation. InProceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). 82211–82231

    2025

  21. [29]

    C. Ye, J. Cui, and D. Hadfield-Menell. 2026. Prompt Injection as Role Confusion. InProceedings of the 43rd International Conference on Machine Learning (ICML). arXiv:2603.12277

    2026 arXiv

  22. [30]

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. 2025.𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. InThe Thirteenth Interna- tional Conference on Learning Representations (ICLR)

    2025

  23. [2025]

    arXiv:2505.08643

    WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation. arXiv:2505.08643

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.