{"id":"8d1f1051-e985-4c13-84fd-92c136cba7c1","arxiv_id":"2608.11905","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A policy-as-logic pipeline, where an LLM extracts facts and an answer-set solver reasons over them, beats prompt-only and code-only baselines on objective policy benchmarks while using fewer tokens.","lead":"This paper combines a large language model with a classical logic solver to answer natural-language questions governed by written policies, such as baggage fees and tax rules. The result is a decision pipeline that is more accurate, more robust to rephrased inputs, and cheaper per query than asking the model to reason directly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PaL accuracy hinges on one unverified Claude Opus translation of each policy into ASP; missing or misencoded rules yield confident wrong decisions even with perfect extraction.","rationale":"The reader's weakest assumption — that the one-time LLM-to-ASP translation is faithful and complete — is exactly the load-bearing concern. The paper admits the translation has known gaps, yet all downstream components are deterministic, so translation errors propagate directly to decisions. This is not an internal inconsistency but a severe verification gap. The proposed test isolates translation errors from extraction errors and tests replication across translation sources. The reader already imposed a CONDITIONAL verdict; my stress-test does not move that verdict, but it sharpens the condition: the authors must demonstrate that the ASP programs are correct for the full test distribution, or at least quantify the translation-error component. Other issues (token-reduction overclaim, cross-paper policy-as-code baselines, absent code) are real but secondary; they do not threaten the central architectural claim as directly as translation fidelity does. Therefore I agree with the reader and recommend no verdict change beyond keeping the result conditional on artifact verification.","tokens_in":9980,"tokens_out":3183,"duration_ms":30499,"concrete_test":"For each of Airline, Tax, and NBA, feed gold-extracted facts (manually annotated from each query) into the published ASP programs and compute solver-only accuracy per domain. If solver-only accuracy is below 1.0, quantify how many queries are misclassified due to missing or misencoded rules. Additionally, regenerate policy.lp from the same policy text using a second independently prompted LLM (e.g., GPT-5.1) or a human ASP expert, and re-run the full PaL pipeline with the same extraction LLMs. If Airline accuracy drops by more than 0.05 or Tax/NBA change materially, the headline result is contingent on a single unverified translation artifact rather than on the PaL architecture itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that PaL's separation of extraction and solving improves accuracy and robustness on objective policies — assumes the one-time LLM-to-ASP translation is a faithful and complete encoding of each policy. The paper concedes in Section 3 that 'this LLM-based translation does not guarantee complete policy coverage, and some of the domains in our experiments have known gaps.' Because grounding, solving, and interpretation are deterministic, any rule omitted or misencoded by Claude Opus becomes a systematic error that no extraction improvement can fix. The reported accuracies (Airline 0.94–1.00 vs. ≤0.38 baselines) are conditional on this single artifact. The paper provides no validation of the ASP programs against the policy text and no error decomposition separating extraction failures from translation failures. A query hitting a known gap will be answered confidently and incorrectly by the solver, so the headline accuracy could change substantially if the translation is redone or corrected. The robustness claim that 'the only source of robustness loss is extraction quality' concerns perturbation consistency and can hold even for a wrong program, so it does not rescue the accuracy claim. Without independent verification of the ASP programs, the central result is not separable from the skill of one undocumented translation run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Policy-as-Logic (PaL): a natural-language policy is translated once into an Answer Set Program with Claude Opus; at inference time an LLM extracts query facts into a schema-defined JSON, those facts are grounded into atoms, Clingo computes an answer set, and a deterministic interpretation maps the answer set to a decision. The method is evaluated on three RuleArena domains (Airline, Tax, NBA) and the PolyGuard HR domain using four open-weight LLMs and six input-perturbation types, reporting exact-match accuracy and robustness. The headline empirical claim is that separating extraction from deterministic symbolic reasoning yields large accuracy and robustness gains on objective, knowledge-based policies, while providing no systematic advantage on the subjective HR classification task.","tokens_in":10132,"tokens_out":5356,"duration_ms":52672,"significance":"If the central claim holds, PaL is a useful architectural pattern for auditable policy-grounded QA: the deterministic grounding, solving, and interpretation steps mean that decision consistency under perturbation is governed by extraction quality alone, and the paper demonstrates this on external benchmarks with several independent LLMs. The evaluation design is generally sound: ground truth comes from established benchmarks, baselines include policy-as-prompt and published policy-as-code numbers, and decisions are scored by exact match. The paper also clearly identifies the boundary case of subjective policies. However, the accuracy results currently rest on an unverified one-shot LLM-to-ASP translation that the paper itself admits has gaps, and one of the abstract's two headline claims (token reduction) is not supported by the reported table. Both issues need to be addressed before the results can be fully credited.","major_comments":[{"comment":"The one-time Claude Opus translation from policy text to ASP is not validated anywhere in the paper, and the paper concedes that 'this LLM-based translation does not guarantee complete policy coverage, and some of the domains in our experiments have known gaps.' Because the solver is deterministic, any omitted or misencoded rule becomes a systematic wrong answer that extraction improvements cannot fix, so Tables 1 and 4 are conditional on an unverified artifact produced by a single undocumented run. Please provide a human audit of each policy.lp, an error decomposition that separates extraction failures from translation failures, or multiple translation runs with variance reporting; without this, the accuracy claim is not yet established.","section":"§3, Semantic parsing"},{"comment":"The abstract's '~10x reduction in token usage' is not supported by Table 2. For Tax the PaL total of 4,143 tokens is only 2.4x less than the 0-shot baseline's 9,816, and for HR PaL is more expensive (975 vs 502). The claim should be restricted to Airline and NBA (roughly 10x and 7.6x, respectively), and the sentence 'our method needs fewer tokens by an order of magnitude in most domains' in Section 4.2 should be corrected.","section":"Abstract and §4.2 Token efficiency"},{"comment":"PaL accuracy is exactly identical across all four LLMs on Tax (0.31 overall; 0.58/0.28/0.07 by difficulty level), even though each row uses a different extraction model. This requires explanation: either report per-model extraction error rates or clarify why the extractor does not affect the Tax outcome. As presented, the reader cannot tell whether the PaL pipeline is actually sensitive to the choice of extraction model in this domain.","section":"Tables 1 and 4, Tax rows"},{"comment":"The statement that 'the only source of robustness loss is extraction quality' conflates perturbation consistency with correctness. A deterministic but incorrect ASP program is perfectly robust while being wrong, so the robustness tables do not by themselves validate the fidelity of the logic translation. The robustness claim should be explicitly scoped to queries on a verified policy encoding, or paired with the audit requested above.","section":"§4.2, Robustness"}],"minor_comments":[{"comment":"The column header 'N|M|= 12 3' is difficult to parse; please reformat to show the counts for |M|=1, |M|=2, and |M|=3 explicitly.","section":"Table 3"},{"comment":"The annotation '(20 kg×10)' is unclear; if it denotes conversion from kilograms to a 10x-scaled unit, please state this in the caption.","section":"Figure 1"},{"comment":"The sentence 'For the policy-as-prompt baseline, we test 0-shot and 1-shot using prompts from the benchmark' reads as though policy-as-code is also tested locally; please clarify which numbers are produced in this paper and which are taken from prior work.","section":"§4.1, Baselines"},{"comment":"No code, schemas, or policy.lp artifacts are released; for reproducibility and to address the translation-validation concern, please include them in an artifact appendix.","section":"§3, Reproducibility"},{"comment":"The HR differences between PaL and the 0-shot baseline are small (0.93 to 0.97) and overlap across models; please report confidence intervals or significance tests for these comparisons.","section":"Table 1, HR rows"},{"comment":"References have spacing issues (e.g., 'Pallaet al., 2025 ') and the footnote marker for GPT-OSS-Safeguards is not displayed in the text; please fix in the camera-ready version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central idea is worth publishing after the translation-validation and token-claim issues are fixed. I see no circularity: the evaluation uses external benchmarks and independent baselines. The identical Tax numbers across four different extraction models should be probed before acceptance; it may indicate that the extraction step is not actually exercising the models as intended. The authors' own prior work (EvalAssist, Granite Guardian) is cited as background only and does not affect the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. The paper's core idea—separate LLM fact extraction from deterministic answer-set solving—is not new (Yang et al., Logic-LM, LINC all do variants), but applying it to real written policies with a systematic perturbation study is a useful step. On the objective domains the results are genuinely striking: Airline 0.94–1.00 accuracy vs. ≤0.38 for prompt-only baselines, Tax 0.31 vs. ≤0.10. Robustness near accuracy is a nice architectural consequence, and the HR domain is an honest boundary case that shows where the approach doesn't help. The paper also deserves credit for naming its own limitations: known gaps in the LLM-to-ASP translation, extraction failures in the appendix, and a clear explanation of answer-set cardinality.\n\nSoft spots are real but mostly fixable. The abstract's \"~10x reduction in token usage\" is an overstatement—Table 2 shows only Airline reaches that; Tax is ~2.4x and HR is more expensive. That's a minor claim, but it should be corrected. There are no confidence intervals or significance tests anywhere; for the small gaps on NBA and HR that matters. The policy-as-code baseline is borrowed from another paper rather than rerun in the same harness, which introduces uncontrolled differences.\n\nThe bigger issue is the one the stress-test flags: the central accuracy numbers are conditional on a one-time Claude Opus 4.7 translation of each policy into ASP. The paper provides the prompt but not the actual policy.lp files, schemas, or mappings. If that translation is even slightly wrong, the solver will confidently give wrong answers that no extraction improvement can fix. The paper concedes known gaps, but there is no error decomposition separating translation errors from extraction errors. This limits reproducibility and makes the absolute numbers hard to verify. I don't think it kills the paper—the relative superiority over baselines is still meaningful, and the translation prompt is given—but it means the reader should treat the headline accuracies as preliminary.\n\nWho benefits: people building auditable policy automation or neurosymbolic pipelines. It deserves a serious referee, but the referee should insist on releasing the ASP programs and schemas, reducing the token claim, and adding uncertainty quantification. I'd be comfortable accepting it with major revisions.","headline":"A solid but not airtight empirical case for LLM-extraction-plus-ASP in policy reasoning; the main caveat is that the entire accuracy claim rests on one unverified Claude Opus translation of each policy into logic.","tokens_in":10715,"tokens_out":2604,"would_cite":true,"duration_ms":28190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For decisions governed by objective written rules, policy-as-logic—where an LLM only extracts facts and an answer-set solver reasons—achieves higher accuracy and robustness than prompt-based or code-based LLM methods.","keywords":["policy-as-logic","answer set programming","LLM fact extraction","policy compliance","neuro-symbolic reasoning","robustness under perturbations","rule-based decision making","token efficiency"],"falsifier":"Run the airline benchmark twice on the same translated logic program, once with the original policy wording and once with policy clauses reordered or paraphrased before translation. If accuracy or decisions shift substantially under policy-side perturbation, then the claimed robustness is specific to query perturbations and does not extend to the policy text, which the paper's architecture leaves to a single translation step.","tokens_in":9717,"feed_emoji":"⚖️","tokens_out":8349,"duration_ms":79898,"temperature":0.7,"pith_summary":"The paper proposes a hybrid decision pipeline for rules from written policies: translate the policy once into an answer-set program, use a language model only to pull query facts into a fixed schema, and let a deterministic solver do all rule reasoning. On three objective, knowledge-based domains—airline baggage fees, income tax, and NBA salary transactions—this separation lifted accuracy well above policy-as-prompt and policy-as-code baselines, with airline accuracy reaching between 0.94 and 1.00 across four models versus at most 0.38 for the best prompt baseline. Robustness under paraphrased, distracted, and sentiment-shifted queries stayed close to accuracy because every step after fact extraction is deterministic, so the only loss is extraction quality. On a subjective safety-moderation policy the approach gave no systematic advantage, which the paper reads as evidence that the gain comes from separating natural-language understanding from logical reasoning.","feed_headline":"Logic solver lifts policy accuracy from 0.38 to 1.00","feed_subtitle":"On airline, tax, and salary-cap rules, a solver pipeline beats prompts while using 10x fewer tokens.","key_machinery":"The load-bearing object is the grounded answer set program built for each query. A policy is compiled once into an answer set program—a logic program whose stable-model semantics supports default negation—plus a JSON schema and mappings that translate text facts to atoms and answer atoms to decisions. At inference, the LLM fills the schema; grounding maps the JSON into a variable-free propositional program; an answer-set solver returns the stable models; deterministic interpretation converts the winning model to a decision. Default negation lets the program fall back on default outcomes when the query leaves attributes unspecified. Because everything after the LLM call is deterministic, robustness is structurally guaranteed except for extraction errors.","core_discovery":"The central claim is that for policies whose tests are objective—numeric, categorical, or transactional—the right division of labor is extraction by an LLM plus reasoning by an answer-set solver, rather than end-to-end LLM reasoning or executable code. The paper demonstrates this on four domains and four models: PaL achieves 0.94–1.00 accuracy on airline baggage versus 0.01–0.38 for prompt baselines; on tax, prompt-baseline accuracy is at most 0.10 while PaL scores 0.31 across all models; on NBA transactions PaL outperforms baselines by a smaller margin. Robustness follows accuracy, including on tax, where every baseline collapses to 0.00 correctness on perturbed queries while PaL stays near its accuracy. The paper attributes this directly to the pipeline: grounding, solving, and interpretation are deterministic, so the only source of robustness loss is extraction quality under perturbations. On the HR safety domain, which requires judging intent rather than applying rules, the solver adds nothing and baselines match or beat PaL.","pith_inferences":["The paper perturbs user queries but not the policy text; a direct extension is to reorder or paraphrase policy clauses before the one-time translation. If decisions change as much as they do for prompt baselines, the single translation step is the exposed point of failure.","The appendix's failure cases are mostly one-field extraction errors with detectable cross-field anomalies, so a consistency checker over the schema—for example, prior-contract salary cannot exceed the new salary—could recover a large share of the remaining robustness gap.","Because default negation lets the solver answer incomplete queries, the same architecture can signal when grounding is incomplete and ask the user for the missing attribute instead of silently applying a default.","For subjective domains, a natural but untested hybrid is to let the solver handle objective rule structure and a separate LLM judge fill the belief atoms, rather than forcing one extraction schema to do both."],"forward_implications":["The accuracy of a small 8B model on airline baggage can be lifted from 0.01 to 0.61 by making it extract facts instead of reason end-to-end.","In objective-rule domains, robustness to paraphrase, distraction, and sentiment is reduced to one question—whether the query's facts survived extraction—so all six perturbation types succeed whenever the extracted JSON is correct.","Each answer can be audited by tracing decision atoms back through the grounded program to the extracted JSON facts, giving a concrete explanation path for policy decisions.","Per-query token use drops by roughly an order of magnitude on long-policy domains because only the schema, not the full policy, is sent to the LLM.","For subjective, belief-based policies, the pipeline offers no systematic accuracy or robustness gain, marking the boundary of where logical solvers help."],"supporting_citations":[{"why":"Supplies the RuleArena airline, tax, and NBA domains plus the 0-shot and 1-shot prompt baselines that PaL is compared against.","marker":"Zhou et al., 2025"},{"why":"Supplies the PolyGuard HR safety-moderation dataset used as the subjective-policy boundary case.","marker":"Kumar et al., 2025"},{"why":"Provides the policy-as-code baseline numbers on the airline domain and the DeonticBench benchmark used for comparison.","marker":"Dou et al., 2026a"},{"why":"Provides the Clingo answer-set solver implementation that computes the stable models in the pipeline.","marker":"Gebser et al., 2019"},{"why":"Documents the completeness limits of LLM-to-ASP translation, which the paper cites as the source of known policy-coverage gaps.","marker":"Ishay et al., 2023"},{"why":"Establishes the earlier LLM-parsing-plus-ASP-reasoning pipeline that PaL adapts to real-world policy documents.","marker":"Yang et al., 2023"},{"why":"Defines the six language reformulation perturbations used to measure robustness across all domains.","marker":"Ye et al., 2024"},{"why":"Defines the policy-as-prompt paradigm that serves as the main baseline for the comparison.","marker":"Palla et al., 2025"}],"fun_headline_variants":["Separate extraction and reasoning to read rules accurately","Policy-as-logic: 10x fewer tokens, higher accuracy","Hybrid LLM and answer-set solver for robust policy rules","Reason over rules with logic, not just prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's accuracy hinges on a single one-time translation of the policy document into a complete set of logical rules and an extraction template that covers every relevant detail; if a rule or field is missed, the deterministic solver returns a confident wrong answer.","fun_headline_variants_meta":{"raw":{"variants":["Separate extraction and reasoning to read rules accurately","Policy-as-logic: 10x fewer tokens, higher accuracy","Hybrid LLM and answer-set solver for robust policy rules","Reason over rules with logic, not just prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1449,"prompt_tokens":880,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":496,"tokens_out":569,"duration_ms":6554,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:14.292238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the airline benchmark twice on the same translated logic program, once with the original policy wording and once with policy clauses reordered or paraphrased before translation. If accuracy or decisions shift substantially under policy-side perturbation, then the claimed robustness is specific to query perturbations and does not extend to the policy text, which the paper's architecture leaves to a single translation step.","supporting_citations":[],"review_version":1}