Pith. sign in

REVIEW 4 major objections 7 minor 14 references

Frontier LLMs systematically collapse nested “unless” exception chains in eligibility rules, and their accuracy drifts silently under the same model name—so high-stakes rule evaluation needs deterministic execution of authored specs, not li

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 23:37 UTC pith:3PQRCAHW

load-bearing objection Solid empirical systems paper on nested-exception LLM failures and silent alias drift; the architecture is a clean Level-3 fix, but the LegalBench “p≤0.003 vs all three” headline oversells a heterogeneous pool. the 4 major comments →

arxiv 2607.23386 v1 pith:3PQRCAHW submitted 2026-07-25 cs.AI

Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation

Paul Simpson , John Kozak , Lisa Doake This is my paper
classification cs.AI
keywords exception chain collapseexemption anchoringneuro-symbolic architectureSMT rule executionlegal eligibilitymodel driftnested exceptionsLLM rule evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper documents a specific failure mode—exception chain collapse—when frontier language models judge eligibility under nested conditional rules of the form “A is required unless B, unless C overrides B.” On shallow multi-route logic the models can score perfectly, but on three-or-more-level exception stacks they produce systematic, often false-negative errors that do not wash out with temperature, majority vote, or the prompting repairs tested. Worse for regulated use, the same model alias can close failure cells between months with no version bump, so benchmark-time accuracy is a moving compliance boundary. The authors’ answer is a neuro-symbolic split: models author rules from authoritative sources; an SMT-based engine then executes the compiled bundle deterministically. Across a 225-scenario controlled benchmark, a deeper adversarial insurance slice, and 949 held-out LegalBench cases, that execution layer stays consistent with the authored specification and is significantly more accurate than the frontier baselines on the multi-prong tasks that match the failure pattern. The practical claim is architectural: move uncertainty from silent inference to an audited specification boundary.

Core claim

On multi-prong compositional rule evaluation under nested exceptions, frontier LLMs show systematic exemption-anchoring and exception-chain-collapse failures—mostly false negatives at first observation—whose empirical surface is unstable across silent updates of the same model alias. Compiling authored rules to SMT constraints and evaluating them deterministically excludes those two error classes by construction at the execution layer, independent of model drift, reasoning-effort defaults, or prompt format, and yields higher accuracy than end-to-end frontier evaluation on the matching LegalBench multi-prong tasks.

What carries the argument

The Eligibility Module (neuro-symbolic authoring-plus-execution split): LLMs draft typed rule bundles from sources; a constrained DSL compiles alternative legal routes to independent disjunctive branches and nested UNLESS chains to closed-form material-implication constraints; an SMT layer evaluates applicant assignments with fixed semantics, full logical-path provenance, and near-zero marginal cost after compilation.

Load-bearing premise

That the authored rule bundles and expert test fixtures faithfully capture the real statutes and contracts, so “correct execution of the bundle” is the right compliance target—even though the paper itself does not guarantee that upstream formalisation step.

What would settle it

A disclosed LLM-only setup that, on the same 949 held-out LegalBench cases and seed-7 partition, reaches at least about 95% combined accuracy under any stated prompting strategy—or stable, prompt-repair-safe perfect handling of fixed three-level exception-chain suites across silent same-alias model updates without trading false negatives for false positives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Regulated eligibility workflows cannot treat a single benchmark-time frontier score as a fixed compliance boundary; the boundary can move without notice under the same model name.
  • Exemption anchoring and exception-chain collapse can be removed from the trust surface at execution time by evaluating compiled constraints rather than re-inferring the statute on every case.
  • Uncertainty becomes tractable when pushed to the specification boundary: versioned bundles, provenance to source clauses, and SME test gates instead of post-hoc rationales.
  • On multi-prong nested rules the accuracy gap versus end-to-end frontier models is large and detectable; on single-clause classification it shrinks to a small edge, matching where compositional structure exists to exploit.
  • Silent model deprecation and reasoning-default changes stop affecting production determinations once a bundle is compiled, because no frontier model remains in the inference path.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any domain with deep carve-backs and overrides—benefits, tax reliefs, safety certification, clinical pathway gates—should expect the same compositional failure surface unless execution is separated from generation.
  • Procurement and conformity assessment will likely shift from “which model scored highest last quarter” to “which fixed rule bundle passed which expert suite,” because only the latter is certifiable over time.
  • Authoring-time model drift that still passes a thin test suite remains an open residual risk; richer adversarial fixture design becomes the real quality bottleneck once execution is solved.
  • Prompt-format cliffs on classification baselines show a second, separate deployment hazard: live LLM judges can look strong or weak depending on few-shot packaging even when multi-prong reasoning is not the issue.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper documents a failure class in frontier LLMs — "exception chain collapse" and "exemption anchoring" — on multi-prong compositional rule evaluation under nested UNLESS-style exceptions, using a 225-scenario controlled benchmark across four domains (two real UK immigration requirements, one synthetic statute, one synthetic insurance wording) plus a 20-scenario adversarial extension. It presents the Aethis Eligibility Module, a neuro-symbolic system in which LLMs author rules from source text and an SMT layer executes them deterministically, and claims the architecture excludes the documented failure patterns by construction at the execution level (Level 3), while explicitly disclaiming guarantees at the retrieval (L1) and formalisation (L2) levels. External validation on nine LegalBench tasks (949 held-out cases) reports the engine significantly more accurate than Sonnet 4.6, Opus 4.7, and GPT-5.4 (combined McNemar p≤0.003). A secondary thesis is that frontier-model accuracy drifts silently under fixed aliases (GPT-5.4 construction-insurance 96.6%→100% between March and April 2026), making benchmark-time accuracy a moving compliance boundary. The manuscript is unusually self-critical: it withdraws a prior reasoning-effort finding (Finding 5/R5) after failed replication, corrects a harness bug, reports coverage matrices and per-task power, and releases all scenarios, harnesses, and per-case JSONs.

Significance. If the claims hold, the paper makes two useful contributions. First, the silent-drift documentation (§6.5 Findings 4 and 6) — frontier cells moving 96.6%→100% and 89.7%→98.5% under unchanged aliases with no version bump — is a genuinely important observation for regulated deployment and is supported by committed replication artifacts. Second, the Level-3 execution guarantee (§5.4) is stated formally and honestly bounded: the exception-chain compilation is given as a closed-form Boolean expression, and the L1/L2/L3 separation (§5.5) is a model of claim-scoping. The paper ships unusually strong reproducibility apparatus: public per-scenario JSONs, a pre-registered N=66 replication, a verifiable pre-registration git tag for the LegalBench random-sample protocol, an explicit falsifiability criterion (§6.10.6), Wilson CIs throughout, and — rare and creditable — two public corrections of its own earlier errors (the v3.5→v3.6 harness bug and the Finding 5/R5 withdrawal when the result failed to replicate). These are exactly the practices the field asks for, and they materially raise confidence in what remains.

major comments (4)
  1. [§6.10, Tables 6.10.A–B; Abstract] The headline external-validation claim ('significantly more accurate than all three frontier models, combined McNemar p≤0.003', also in the abstract) is a single paired-binomial over a non-homogeneous pool, and the paper's own numbers show the significance is not robustly attributable to the architectural thesis. (a) The GPT-5.4 comparison (b=340, c=35) is dominated by the few-shot positive-class-anchoring artifact that §6.10.4 itself shows is prompt-format-coupled (zero-shot recovers +39pp on cuad_covenant_not_to_sue). Pooling those cells into 'significantly more accurate than GPT-5.4' conflates a protocol sensitivity with a capability difference. (b) Against the Anthropic models, the paper's own §6.10.3 reports random-only combined tests of p=0.24 (Sonnet) and p=0.053 (Opus) — not significant. The Anthropic-side significance in Table 6.10.B is therefore carried by the three curated tas
  2. [§6.10.3, §6.10.5, §6.10.6] 949 cases are counted in the combined test, but by the paper's own engine error analysis (§6.10.6) 850 of them are single-clause classification tasks where 'the engine reduces to the LLM extractor's binary judgement, run through a one-clause rule.' No compositional execution advantage is exercised on those cases, so discordant pairs there do not evidence the exception-chain claim; they evidence extractor-plus-one-clause-rule versus end-to-end prompting, which is closer to a pipeline/prompting comparison than an architectural one. The cross-extractor experiment (§6.10.5) is the right control but is run only on two curated tasks and on the full test split rather than the held-out partition. I would like to see either (a) Table 6.10.B recomputed on the subset of cases where multi-prong conjunction is actually exercised, or (b) the cross-extractor control run held-out across all nine tasks,
  3. [§6.4.1, §6.1 scope note, §6.8] Evidence base (ii) hinges on the E4 carveback-gap scenario: three of four frontier configurations return 'eligible' where the expected value is 'not_eligible', and this is presented as a structural LLM failure ('their disagreements with engine reveal model failures, not authoring errors'). That inference is only as strong as the expected value, and the manuscript states (§6.1 scope note) that construction_car expected outcomes were authored by a non-lawyer and 'have not been independently validated by a qualified construction-insurance professional.' The §6.4.1 mitigation (prose justification authored before consulting the engine) does not fully address this: the prose, the binary, and the fixture were all authored by the same person in the same pass, so a shared misreading of DE3/LEG3 coverage-gap structure would pass all three checks. Three frontier models from two families agreeing ag
  4. [§6.5 Findings 4 and 6; Tables 6 and 8a; §9.4] The silent-drift narrative is the paper's most novel empirical contribution, but the two load-bearing cells are reported as point estimates: GPT-5.4 96.6% [88.3–99.0] → 100% [95.1–100] on construction (Table 8a) and Opus 4.6 89.7% → 98.5% on spacecraft (Table 6). These intervals overlap, runs-per-scenario is 3 with majority vote, and no paired/run-level discordance analysis is presented for the drift deltas, so 'the cell closed silently' is not fully separated from 'the March estimate had ±5–10pp binomial uncertainty and the April estimate landed at the top of it.' The paper's own history (the v3.5→v3.6 GPT-5.3 harness bug; the Finding 5 withdrawal) demonstrates that apparent cell-level movement can be harness artifact rather than model drift, which cuts both ways. Since per-scenario JSONs are public, a McNemar-style comparison of March vs April runs per scenario (or at minimum run-level
minor comments (7)
  1. [§6.4.1] §6.4.1 'Reasoning-effort observation': the description 'GPT-5.4 at low reasoning effort — the configuration that forces the model to deliberate' versus 'default ... engaged no extended reasoning' is counterintuitive relative to the usual semantics of the effort parameter and will confuse readers; please clarify exactly what API configuration was used and reconcile with the withdrawn Finding 5 framing.
  2. [Figures] Figure numbering is out of order: Figure 9 appears in §6.4.1 while Figures 5–8 appear later (§6.7, §6.10). Renumber or reorder.
  3. [Abstract] Abstract: 'McNemar'sp≤0.003' — missing space; also the phrase 'up to +41 percentage points' should be attributed to the curated tasks in the abstract itself, since §6.10.3 shows random-sample margins of 1–3pp.
  4. [§3, References] Several in-text citations are year-only without author names (e.g., 'Chain-of-thought prompting [2022]', 'zero-shot reasoning elicitation [2022]', 'LegalBench [2023]', 'FOLIO [2022]', 'Logic-LM [2023]'); use author-year for readability.
  5. [§6.10.2] The 27 uncorrected per-task McNemar tests in Table 6.10.A are flagged, but the Bonferroni sentence then pivots to the combined test; it would be cleaner to also state explicitly which curated-arm per-task Anthropic comparisons (hearsay vs Sonnet p=0.006, jcrew vs Sonnet p<0.001) survive correction, since those become the primary evidence once the GPT-5.4 cells are discounted.
  6. [§6.7] §6.7 R1: the retention of the 0/70 / 4.19% bound with the v3.8 caveat is good practice, but the caveat is easy to miss; consider moving the qualification into the R1 heading rather than a trailing paragraph.
  7. [§6.4.1, Table 8c] Table 8c: with N=20, the Wilson CIs on all four LLM configurations overlap heavily (20/20 vs 18/20 is within noise); the text should state this explicitly so the adversarial extension is read as an existence proof of failure modes, not a ranking.

Circularity Check

3 steps flagged

Controlled-benchmark engine “accuracy” is agreement with the same gold fixtures it compiles—100% is by construction at Level 3—while LLM-vs-fixture gaps and LegalBench labels remain non-circular.

specific steps
  1. self definitional [§6.1 Methodology; §6.3 Results (engine accuracy definition)]
    "System A (Eligibility Module): Gold-standard rule fixtures compiled to formal SMT constraints. … In all tables below, accuracy figures for the Eligibility Module refer to agreement with the benchmark’s formal rule fixtures — the authored rule specifications used as the benchmark’s ground truth. … The Module achieves complete consistency with the benchmark’s formal rule fixtures across all domains - a consequence of deterministic execution over the authored specification, not empirical tuning."

    Module “accuracy” is defined as matching the same authored fixtures the Module compiles and executes. Correct Level-3 compilation/evaluation makes 100% agreement tautological; it does not independently establish that determinations match statute or external legal ground truth (Level 2), yet tables present Module 100% in parallel with empirical LLM accuracies against that same internal standard.

  2. self definitional [§5.4 compilation; Appendix A.1 life_uk expected-value formula]
    "A multi-level exception chain … compiles to the material implication ¬[(B∧¬C)∨D]⇒A or equivalently (B∧¬C)∨D∨A. … Evaluating it against a fixed applicant assignment and a compiled rule bundle is a deterministic computation. … Expected value formula: eligible = test_passed OR (age < 18) OR (age >= 65) OR medical_exempt OR discretion_applied. … The life_uk expected values are computed by Python code from the section’s authored eligibility formula."

    Exception-chain “exclusion by construction” is the semantics of the compiled formula, not a prediction tested against independent outcomes. For life_uk, fixture labels are generated from the same boolean formula the engine evaluates, so engine–label agreement cannot fail without a bug in the shared formalization/implementation path.

  3. fitted input called prediction [§6.4.1 v3.8 adversarial extension methodology]
    "Each scenario was authored with an independent prose justification for its expected outcome before the engine was consulted. The prose lives in the scenario’s notes: field; the binary expected-value (expect.outcome) was authored in the same pass for consistency. The scenarios were then submitted to the engine … Engine outcome matched authored expectation on 20/20, indicating self-consistency between the prose, the binary, and the rule encoding."

    The 20/20 engine score is framed as stress-test success, but expected binaries were co-authored with the prose “for consistency” against the same canonical bundle encoding; matching then confirms internal coherence of author-constructed labels with the bundle, not an out-of-sample prediction. (LLM disagreements with those labels can still be informative.)

full rationale

The paper’s load-bearing circular step is narrow and mostly disclosed. In the 225-scenario controlled benchmark, System A is the Eligibility Module running gold-standard rule fixtures through SMT compilation; reported Module accuracy is explicitly “agreement with the benchmark’s formal rule fixtures.” Once the compiler/evaluator implements the authored DSL (including the material-implication encoding of UNLESS chains), 225/225 follows by construction and is not an independent empirical prediction about legal outcomes. life_uk expected values are likewise generated from the same authored eligibility formula the fixtures encode. That asymmetry is partially acknowledged (§5.4–§5.5, §6.3: “consequence of deterministic execution… not empirical tuning”; Level 2 formalization is not guaranteed). It does not nullify the LLM side of the comparison: scoring frontier models against fixed fixtures is a coherent test of whether models preserve nested exception structure relative to a stated formalization, and silent drift under fixed aliases is externally observable. The LegalBench arm is not circular in the same sense—bundles are authored from upstream task text against author-written statute tests, and outcomes are scored on held-out LegalBench labels, not on the engine’s own fixtures—though selection/holdout and prompt-format issues are validity threats, not definitional loops. No self-citation uniqueness theorem or ansatz-smuggling chain carries the result. Proportionate score: partial by-construction circularity on engine perfection in evidence base (i), with independent content in LLM failure documentation, drift, and external labels.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

This is an empirical systems paper, not a first-principles derivation. The central claims rest on methodological choices treated as given: fixture-defined ground truth, synthetic statutes as valid probes of real nested-UNLESS structure, generic-prompt deployment as the right baseline, and Level-3 SMT semantics as the compliance-relevant correctness notion. Few numeric free parameters are fitted; risk is in domain assumptions and author-constructed oracles rather than curve-fitting.

free parameters (3)
  • Generic vs enhanced prompt formulations and LegalBench upstream few-shot templates = Generic assessor prompt; one enhanced exemption-focused variant; upstream LegalBench base_prompt.txt
    Prompt text is a design choice that materially moves FN/FP trade-offs and GPT-5.4 classification accuracy; not fitted statistically, but hand-chosen and load-bearing for measured gaps.
  • Held-out partition seed and dev fraction for LegalBench = seed 7, dev_fraction 0.5
    Deterministic seed-7 / 50% split defines which cases enter headline metrics; pre-registered but still an analysis choice.
  • Adversarial scenario selection and depth banding = Author-stratified A–E complexity dimensions; post-hoc adversarial additions
    Spacecraft adversarial suite and v3.8 20-scenario CAR extension were authored to stress known failure modes rather than sampled from a natural case distribution.
axioms (5)
  • ad hoc to paper Correctness of eligibility for the controlled benchmark is defined by agreement with authored formal rule fixtures / expected outcomes.
    Stated throughout §6: Module accuracy is consistency with fixtures; LLM accuracy measured against the same fixtures.
  • domain assumption Quantifier-free SMT theories (bitvectors, LIA/LRA, UFs) plus disjunctive route groups faithfully capture the evaluated statutory/contract logics at execution time.
    §5.3–5.4 compilation of routes and exception chains to material implication / Boolean combinations.
  • domain assumption Synthetic spacecraft statute and synthetic DE3/DE5-style CAR wording are valid instruments for studying nested exception evaluation in real high-stakes texts.
    §6.1 domain disclosure and benchmark intent; external validity partly deferred to LegalBench.
  • domain assumption A generic single-shot legal-assessor prompt represents realistic deployment better than ceiling-seeking bespoke prompt engineering for the primary claim.
    §6.1 benchmark philosophy; robustness section tests only limited prompt repairs.
  • standard math Standard statistical decision procedures (Wilson CIs, McNemar / paired binomial tests) apply to these paired correctness indicators.
    Used in §6 tables and §6.10 aggregates.
invented entities (3)
  • Exception chain collapse (failure pattern) independent evidence
    purpose: Name the dominant multi-level UNLESS mis-evaluation mode in LLM rule following.
    Taxonomic label for observed behavior; useful but not an external physical entity. Evidence is benchmark-internal plus LegalBench multi-prong tasks.
  • Exemption anchoring (failure pattern) independent evidence
    purpose: Name failure to treat alternative routes as independently sufficient after primary-route failure.
    Second named pattern in §4.2; supported by scenario tables but still a paper-introduced category.
  • Aethis Eligibility Module (LLM-authored rules + SMT execution bundles) independent evidence
    purpose: Provide deterministic Level-3 evaluation and provenance for eligibility determinations.
    Productized architecture under evaluation by its builders; public benchmark harness exists, full engine internals partly NDA-gated.

pith-pipeline@v1.2.0-grok45-kimik3 · 38736 in / 4064 out tokens · 81799 ms · 2026-07-30T23:37:07.731171+00:00 · methodology

0 comments
read the original abstract

We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B". The failure reproduces at first observation, but its empirical surface is unstable: between March and April 2026 several failure cells closed silently under the same model alias, with no version bump (GPT-5.4 on construction insurance moved from 96.6% to 100%, same prompt and harness). For regulated workflows, frontier-model accuracy is a moving compliance boundary that shifts without notice. We present the Aethis Eligibility Module, a neuro-symbolic architecture in which LLMs author rules from authoritative sources and an SMT-based layer executes them deterministically, consistent with the authored specification regardless of model drift, reasoning-effort defaults, or prompt format. Three evidence bases: (i) a controlled benchmark of 225 scenarios across four regulatory domains documents the pattern and, in replication, the drift that partially closed it; (ii) a 20-scenario adversarial extension on construction insurance, where the engine scores 20/20, as does one of four frontier configurations (GPT-5.4 at low reasoning effort), while the other three, including Anthropic's strongest model at evaluation time, fail the same coverage-gap edge case; (iii) external validation on nine peer-reviewed LegalBench tasks, 949 held-out cases, where the engine is significantly more accurate than all three frontier models (combined McNemar's p <= 0.003), with margins up to +41 points on the curated multi-prong tasks against the Anthropic models. The contribution is to relocate uncertainty from the inference boundary, where it is silent, to the specification boundary, where it is deliberate and audited. All scenarios, rule encodings, and results are public and reproducible.

Figures

Figures reproduced from arXiv: 2607.23386 by John Kozak, Lisa Doake, Paul Simpson.

Figure 1
Figure 1. Figure 1: End-to-End Pipeline Every generated rule carries provenance linking it to specific source material. Source items are tagged with structured references (e.g., [BNA1981#Schedule1/P1.1]) in the LLM’s context, and the LLM is instructed to cite these tags on each generated criterion and field. A verification step then resolves these citations against the source material, creating anchors with document IDs, sect… view at source ↗
Figure 2
Figure 2. Figure 2: Route Independence in the Eligibility Module [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Spacecraft accuracy with 95% Wilson confidence intervals. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Construction insurance accuracy with 95% Wilson confidence intervals. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 9
Figure 9. Figure 9: v3.8 adversarial extension forest plot. Cross-evaluation note: the carveback-gap scenario (E4) is failed by three of four frontier-LLM configurations across both Anthropic and OpenAI families. The scenario tests the DE3/LEG3 coverage gap explicitly documented in source.md Cl. 7 commentary: with is_access_damage=true and consequence_of_failure=false, the carveback group fails (Route A needs consequence; Rou… view at source ↗
Figure 5
Figure 5. Figure 5: Prompt-repair trade-off (Claude Opus 4.6, spacecraft). [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: LegalBench per-task held-out accuracy with Wilson 95% CIs. [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GPT-5.4 calibration cliff and zero-shot recovery. [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗
Figure 6
Figure 6. Figure 6: Provenance Chain (Two-Stage Verification) [PITH_FULL_IMAGE:figures/full_fig_p036_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 13 linked inside Pith

  1. [2]

    Artur d’Avila Garcez, Luís C

    arXiv:2211.10435. Artur d’Avila Garcez, Luís C. Lamb, and Dov M. Gabbay.Neural-Symbolic Cognitive Reasoning. Springer,

    Pith/arXiv arXiv

  2. [4]

    FOLIO: Natural language reasoning with first-order logic.arXiv preprint arXiv:2209.00840,

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, et al. FOLIO: Natural language reasoning with first-order logic.arXiv preprint arXiv:2209.00840,

    Pith/arXiv arXiv

  3. [8]

    Donald Nute

    doi: 10.1016/j.clsr.2022.105799. Donald Nute. Defeasible logic. InHandbook of Logic in Artificial Intelligence and Logic Programming, volume

    arXiv 2022

  4. [10]

    Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295,

    Pith/arXiv arXiv

  5. [11]

    Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati

    arXiv:2302.00093. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change).arXiv preprint arXiv:2206.10498,

    Pith/arXiv arXiv

  6. [12]

    LLMs still can’t plan; can LRMs? A preliminary evaluation of OpenAI’s o1 on PlanBench.arXiv preprint arXiv:2409.13373,

    42 Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. LLMs still can’t plan; can LRMs? A preliminary evaluation of OpenAI’s o1 on PlanBench.arXiv preprint arXiv:2409.13373,

    Pith/arXiv arXiv

  7. [13]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H

    arXiv:2203.11171. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  8. [14]

    arXiv:2201.11903. 43

    Pith/arXiv arXiv

  9. [1977]

    GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229,

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229,

    Pith/arXiv arXiv

  10. [1994]

    Olausson, Alex Gu, Benjamin Lippert, Cedegao E

    Theo X. Olausson, Alex Gu, Benjamin Lippert, Cedegao E. Si, Yeonbae Tian, Allen Le, et al. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers.arXiv preprint arXiv:2310.15164,

    Pith/arXiv arXiv

  11. [2010]

    Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, et al

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, et al. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models.arXiv preprint arXiv:2308.11462,

    Pith/arXiv arXiv

  12. [2022]

    Gary Marcus

    arXiv:2205.11916. Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence.arXiv preprint arXiv:2002.06177,

    Pith/arXiv arXiv 2002

  13. [2023]

    European Parliament

    arXiv:2305.18654. European Parliament. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI act). Official Journal of the European Union,

    Pith/arXiv arXiv 2024

  14. [2024]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa

    arXiv:2402.01817. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv