REVIEW 4 major objections 6 minor 22 references
Policy-as-logic for robust reasoning over rules
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read For decisions governed by objective written rules, policy-as-logic—where an LLM only extracts facts and an answer-set solver reasons—achieves higher accuracy and robustness than prompt-based or code-based LLM methods.
desk verdict A solid but not airtight empirical case for LLM-extraction-plus-ASP in policy reasoning; the main caveat is that the entire accuracy claim rests on one unverified Claude Opus translation of each policy into logic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the grounded answer set program built for each query. A policy is compiled once into an answer set program—a logic program whose stable-model semantics supports default negation—plus a JSON schema and mappings that translate text facts to atoms and answer atoms to decisions. At inference, the LLM fills the schema; grounding maps the JSON into a variable-free propositional program; an answer-set solver returns the stable models; deterministic interpretation converts the winning model to a decision. Default negation lets the program fall back on default outcomes when the query leaves attributes unspecified. Because everything after the LLM call is deterministic, robustness is structurally guaranteed except for extraction errors.
What would settle it
Run the airline benchmark twice on the same translated logic program, once with the original policy wording and once with policy clauses reordered or paraphrased before translation. If accuracy or decisions shift substantially under policy-side perturbation, then the claimed robustness is specific to query perturbations and does not extend to the policy text, which the paper's architecture leaves to a single translation step.
Extended reading notes
Core claim
The central claim is that for policies whose tests are objective—numeric, categorical, or transactional—the right division of labor is extraction by an LLM plus reasoning by an answer-set solver, rather than end-to-end LLM reasoning or executable code. The paper demonstrates this on four domains and four models: PaL achieves 0.94–1.00 accuracy on airline baggage versus 0.01–0.38 for prompt baselines; on tax, prompt-baseline accuracy is at most 0.10 while PaL scores 0.31 across all models; on NBA transactions PaL outperforms baselines by a smaller margin. Robustness follows accuracy, including on tax, where every baseline collapses to 0.00 correctness on perturbed queries while PaL stays near its accuracy. The paper attributes this directly to the pipeline: grounding, solving, and interpretation are deterministic, so the only source of robustness loss is extraction quality under perturbations. On the HR safety domain, which requires judging intent rather than applying rules, the solver adds nothing and baselines match or beat PaL.
Load-bearing premise
The method's accuracy hinges on a single one-time translation of the policy document into a complete set of logical rules and an extraction template that covers every relevant detail; if a rule or field is missed, the deterministic solver returns a confident wrong answer.
Editorial extensions
If this is right
- The accuracy of a small 8B model on airline baggage can be lifted from 0.01 to 0.61 by making it extract facts instead of reason end-to-end.
- In objective-rule domains, robustness to paraphrase, distraction, and sentiment is reduced to one question—whether the query's facts survived extraction—so all six perturbation types succeed whenever the extracted JSON is correct.
- Each answer can be audited by tracing decision atoms back through the grounded program to the extracted JSON facts, giving a concrete explanation path for policy decisions.
- Per-query token use drops by roughly an order of magnitude on long-policy domains because only the schema, not the full policy, is sent to the LLM.
- For subjective, belief-based policies, the pipeline offers no systematic accuracy or robustness gain, marking the boundary of where logical solvers help.
Reading between the lines
- The paper perturbs user queries but not the policy text; a direct extension is to reorder or paraphrase policy clauses before the one-time translation. If decisions change as much as they do for prompt baselines, the single translation step is the exposed point of failure.
- The appendix's failure cases are mostly one-field extraction errors with detectable cross-field anomalies, so a consistency checker over the schema—for example, prior-contract salary cannot exceed the new salary—could recover a large share of the remaining robustness gap.
- Because default negation lets the solver answer incomplete queries, the same architecture can signal when grounding is incomplete and ask the user for the missing attribute instead of silently applying a default.
- For subjective domains, a natural but untested hybrid is to let the solver handle objective rule structure and a separate LLM judge fill the belief atoms, rather than forcing one extraction schema to do both.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Policy-as-Logic (PaL): a natural-language policy is translated once into an Answer Set Program with Claude Opus; at inference time an LLM extracts query facts into a schema-defined JSON, those facts are grounded into atoms, Clingo computes an answer set, and a deterministic interpretation maps the answer set to a decision. The method is evaluated on three RuleArena domains (Airline, Tax, NBA) and the PolyGuard HR domain using four open-weight LLMs and six input-perturbation types, reporting exact-match accuracy and robustness. The headline empirical claim is that separating extraction from deterministic symbolic reasoning yields large accuracy and robustness gains on objective, knowledge-based policies, while providing no systematic advantage on the subjective HR classification task.
Significance. If the central claim holds, PaL is a useful architectural pattern for auditable policy-grounded QA: the deterministic grounding, solving, and interpretation steps mean that decision consistency under perturbation is governed by extraction quality alone, and the paper demonstrates this on external benchmarks with several independent LLMs. The evaluation design is generally sound: ground truth comes from established benchmarks, baselines include policy-as-prompt and published policy-as-code numbers, and decisions are scored by exact match. The paper also clearly identifies the boundary case of subjective policies. However, the accuracy results currently rest on an unverified one-shot LLM-to-ASP translation that the paper itself admits has gaps, and one of the abstract's two headline claims (token reduction) is not supported by the reported table. Both issues need to be addressed before the results can be fully credited.
major comments (4)
- [§3, Semantic parsing] The one-time Claude Opus translation from policy text to ASP is not validated anywhere in the paper, and the paper concedes that 'this LLM-based translation does not guarantee complete policy coverage, and some of the domains in our experiments have known gaps.' Because the solver is deterministic, any omitted or misencoded rule becomes a systematic wrong answer that extraction improvements cannot fix, so Tables 1 and 4 are conditional on an unverified artifact produced by a single undocumented run. Please provide a human audit of each policy.lp, an error decomposition that separates extraction failures from translation failures, or multiple translation runs with variance reporting; without this, the accuracy claim is not yet established.
- [Abstract and §4.2 Token efficiency] The abstract's '~10x reduction in token usage' is not supported by Table 2. For Tax the PaL total of 4,143 tokens is only 2.4x less than the 0-shot baseline's 9,816, and for HR PaL is more expensive (975 vs 502). The claim should be restricted to Airline and NBA (roughly 10x and 7.6x, respectively), and the sentence 'our method needs fewer tokens by an order of magnitude in most domains' in Section 4.2 should be corrected.
- [Tables 1 and 4, Tax rows] PaL accuracy is exactly identical across all four LLMs on Tax (0.31 overall; 0.58/0.28/0.07 by difficulty level), even though each row uses a different extraction model. This requires explanation: either report per-model extraction error rates or clarify why the extractor does not affect the Tax outcome. As presented, the reader cannot tell whether the PaL pipeline is actually sensitive to the choice of extraction model in this domain.
- [§4.2, Robustness] The statement that 'the only source of robustness loss is extraction quality' conflates perturbation consistency with correctness. A deterministic but incorrect ASP program is perfectly robust while being wrong, so the robustness tables do not by themselves validate the fidelity of the logic translation. The robustness claim should be explicitly scoped to queries on a verified policy encoding, or paired with the audit requested above.
minor comments (6)
- [Table 3] The column header 'N|M|= 12 3' is difficult to parse; please reformat to show the counts for |M|=1, |M|=2, and |M|=3 explicitly.
- [Figure 1] The annotation '(20 kg×10)' is unclear; if it denotes conversion from kilograms to a 10x-scaled unit, please state this in the caption.
- [§4.1, Baselines] The sentence 'For the policy-as-prompt baseline, we test 0-shot and 1-shot using prompts from the benchmark' reads as though policy-as-code is also tested locally; please clarify which numbers are produced in this paper and which are taken from prior work.
- [§3, Reproducibility] No code, schemas, or policy.lp artifacts are released; for reproducibility and to address the translation-validation concern, please include them in an artifact appendix.
- [Table 1, HR rows] The HR differences between PaL and the 0-shot baseline are small (0.93 to 0.97) and overlap across models; please report confidence intervals or significance tests for these comparisons.
- [References] References have spacing issues (e.g., 'Pallaet al., 2025 ') and the footnote marker for GPT-OSS-Safeguards is not displayed in the text; please fix in the camera-ready version.
Circularity Check
No circularity: the paper's claims are empirical measurements against external benchmarks, and the cited self-works are background only.
full rationale
The paper's central claims are empirical: PaL accuracy and robustness are measured against the external RuleArena and PolyGuard benchmarks, with ground-truth decisions supplied by those benchmarks and baselines consisting of published policy-as-prompt and policy-as-code methods. No equation in the paper reduces a predicted quantity to a fitted input. The one-time LLM-to-ASP translation is described as a construction step with a disclosed limitation ('this LLM-based translation does not guarantee complete policy coverage'), and the subsequent evaluation is against held-out benchmark queries rather than against the translation prompt or its outputs. The claimed robustness property follows from the architecture (deterministic grounding, solving, and interpretation) and is supported by perturbation experiments; this is an architectural argument, not a circular definition. The citations to the authors' own prior work (EvalAssist, Granite Guardian) appear only as context in the related-work discussion and are not load-bearing for the accuracy or robustness results. There is no self-citation chain invoked to forbid alternatives and no uniqueness theorem imported from prior work. Potential concerns about translation fidelity and completeness affect correctness or external validity, but they are not circularity: the paper does not define its predictions in terms of its outputs, and it does not fit a parameter to the test data and then rename that fit as a prediction. Accordingly, no specific circular step can be identified, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The LLM's one-time translation of each policy document into an ASP program is sufficiently complete and faithful, including exceptions and defaults.
- domain assumption The perturber and the LLM-as-a-judge preserve the semantic meaning of the original queries through six reformulations.
- domain assumption Ground truth labels in RuleArena and PolyGuard are correct and decisions are categorical, so exact-match accuracy is the right metric.
- standard math Clingo's stable-model semantics correctly implements non-monotonic defaults for these programs.
Cite this review
Pith. "Pith review of Policy-as-logic for robust reasoning over rules." pith.science (2026). https://pith.science/paper/VT6SGSFY
@misc{pith2026260811905,
author = {Pith},
title = {Pith review of: Policy-as-logic for robust reasoning over rules},
year = {2026},
howpublished = {\url{https://pith.science/paper/VT6SGSFY}},
note = {Machine review of arXiv:2608.11905}
}
read the original abstract
In many practical applications of generative AI systems, from tax rules to airline baggage allowance, responses to natural language queries must respect written policies or rules. We present a hybrid symbolic approach that expresses policies in formal logic and at inference time exploits the representation power of language models for fact extraction to ground predicates, and an answer set solver for reasoning such that responses are interpretable, auditable, and as we show, accurate and robust under input perturbations. Specifically, we show this separation of extraction and reasoning steps outperforms policy-as-prompt and policy-as-code methods in most cases with ~10x reduction in token usage. The results point to the value of structured reasoning and symbolic solvers in conjunction with generative models to make robust decisions involving objective criteria.
Figures
Reference graph
Works this paper leans on
-
[1]
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages=
Policy-as-prompt: Rethinking content moderation in the age of large language models , author=. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages=
work page 2025
-
[2]
arXiv preprint arXiv:2502.09100 , year=
Logical reasoning in large language models: A survey , author=. arXiv preprint arXiv:2502.09100 , year=
-
[3]
arXiv preprint arXiv:2602.03587 , year=
CL-bench: A Benchmark for Context Learning , author=. arXiv preprint arXiv:2602.03587 , year=
- [4]
-
[5]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Evalassist: Llm-as-a-judge simplified , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[6]
arXiv preprint arXiv:2306.12672 , year=
From word models to world models: Translating from natural language to the probabilistic language of thought , author=. arXiv preprint arXiv:2306.12672 , year=
-
[7]
arXiv preprint arXiv:2604.00414 , year=
Decision-Centric Design for LLM Systems , author=. arXiv preprint arXiv:2604.00414 , year=
-
[8]
European Conference on Information Retrieval , pages=
OrLog: Resolving Complex Queries with LLMs and Probabilistic Reasoning , author=. European Conference on Information Retrieval , pages=. 2026 , organization=
work page 2026
Show all 22 references
-
[9]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Rulearena: A benchmark for rule-guided reasoning with llms in real-world scenarios , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[10]
Judging the judges: A systematic study of position bias in llm-as-a-judge , author=. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics , pages=
-
[11]
arXiv preprint arXiv:2604.04443 , year=
DeonticBench: A Benchmark for Reasoning over Rules , author=. arXiv preprint arXiv:2604.04443 , year=
-
[12]
arXiv preprint arXiv:2504.04377 , year=
Polyguard: A multilingual safety moderation tool for 17 languages , author=. arXiv preprint arXiv:2504.04377 , year=
-
[13]
Advances in neural information processing systems , volume=
Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=
-
[14]
Findings of the association for computational linguistics: ACL 2023 , pages=
Coupling large language models with logic programming for robust and general reasoning from text , author=. Findings of the association for computational linguistics: ACL 2023 , pages=
2023
-
[15]
2023 , month =
Ishay, Adam and Yang, Zhun and Lee, Joohyung , booktitle =. 2023 , month =. doi:10.24963/kr.2023/37 , url =
2023 doi
-
[16]
Logic- LM : Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, Liangming and Albalak, Alon and Wang, Xinyi and Wang, William. Logic- LM : Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.248
2023 doi
-
[17]
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=
LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=
2023
-
[18]
Theory and Practice of Logic Programming , volume=
Multi-shot ASP solving with clingo , author=. Theory and Practice of Logic Programming , volume=. 2019 , publisher=
2019
-
[19]
arXiv preprint arXiv:1405.3694 , year=
Clingo= ASP+ control: Preliminary report , author=. arXiv preprint arXiv:1405.3694 , year=
-
[20]
arXiv preprint arXiv:2410.02736 , year=
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge , author=. arXiv preprint arXiv:2410.02736 , year=
-
[21]
arXiv preprint arXiv:2506.09443 , year=
Llms cannot reliably judge (yet?): A comprehensive assessment on the robustness of llm-as-a-judge , author=. arXiv preprint arXiv:2506.09443 , year=
-
[22]
Journal of Machine Learning Research , volume=
Promptbench: A unified library for evaluation of large language models , author=. Journal of Machine Learning Research , volume=
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.