Benchmarking LLM Competence on Logical Inference over Probability Operators

Alvin Grissom II; Jack Greff; Nayera Hasan

Only 9 of 29 LLMs beat chance on 'probably'/'might' inference

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 01:38 UTC pith:WQUAFJNJ

load-bearing objection Useful benchmark and a sensible metric, but the headline '9/29' rests on two mislabeled inference templates; the qualitative conclusion still holds. the 3 major comments →

arxiv 2607.27405 v3 pith:WQUAFJNJ submitted 2026-07-29 cs.CL cs.AI

Benchmarking LLM Competence on Logical Inference over Probability Operators

Nayera Hasan , Jack Greff , Alvin Grissom II This is my paper

classification cs.CL cs.AI

keywords probability operatorsepistemic modalslogical inferencelarge language modelsanswer biascompetence floornegation sensitivitybenchmark design

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a benchmark of 14,320 English prompts asking yes/no questions about inferences with graded probability words such as 'probably,' 'might,' and 'must' — for example, whether 'Savir is probably at the party' entails 'Savir might be at the party.' Every logical form is rewritten in multiple ways (five question phrasings, several negation styles, different names and activities), so a model that genuinely reasons should give the same answer when the logic is identical. Evaluating 29 models zero-shot, the paper finds that most are not tracking the logic: accuracy is best explained by a fixed preference for Yes or No, not by the validity of the inference. It summarizes this with a 'competence floor' — the worse of a model's accuracy on questions whose correct answer is Yes versus No — and reports that only 9 of 29 models clear the chance floor of 0.5. The finding matters because LLMs are increasingly used in medicine, law, and finance, where statements like 'probably has a disease' must be combined correctly under uncertainty.

Core claim

In the paper's own terms, the central discovery is that performance on this benchmark is dominated by answer bias: accuracy on questions with a correct answer of Yes and accuracy on questions with a correct answer of No diverge sharply for most models, and the worse of the two (the competence floor) falls at or below chance for 20 of 29 models. The benchmark is built as a 2x2 grid crossing inference validity with question negation, so each cell has a known gold answer; a constant-Yes responder scores near zero on the floor despite 55% overall accuracy. The paper also reports that the finding is negation-independent (the affirmative-only floor correlates r=0.90 with the full floor) and that s

What carries the argument

The central object is the competence floor, Floor = min(Acc(Yes), Acc(No)), where Acc(Yes) averages accuracy on the two cells whose correct answer is Yes and Acc(No) on the two cells whose correct answer is No. It has an absolute random baseline of 0.5: a constant responder scores 0, a coin flipper 0.5, a competent reasoner near 1. The benchmark's design makes the floor meaningful: every inference template is crossed with affirmative and negated question forms, so the same logical content produces prompts with each gold answer; this 'validity-by-negation' structure turns answer bias from a nuisance into a measurable quantity. A secondary mechanism is the systematic variation of question form

Load-bearing premise

The load-bearing premise is that every template's gold answer is unambiguously correct; under the paper's own threshold semantics, at least two 'valid' templates (Chancy Modus Tollens and Conditional-to-Comparative) admit counterexamples, so the 9-of-29 headline rests on those labels being right.

What would settle it

Regrade the models on a version of the benchmark whose gold labels are computed from the paper's own semantics (probably = p>=0.5, material 'if'), marking Chancy Modus Tollens and Conditional-to-Comparative invalid while keeping the three invalid templates invalid; then count how many of 29 models clear the 0.5 floor. If substantially more than 9 or 10 clear it, the claim that models rely on fixed answer bias rather than logic is not supported for those models. A single probability assignment — p(phi and psi)=0.05, p(phi and not psi)=0.5, p(not phi and psi)=0.05, p(not phi and not psi)=0.4 — s

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

Overall accuracy on binary logical-reasoning benchmarks can be inflated by answer bias; the competence floor should be reported alongside it, since the floor has an absolute chance baseline and is not inflated by a constant Yes/No preference.
Current open-weight and frontier LLMs are not reliable for zero-shot inferences over graded epistemic modals: 20 of 29 models score at or below chance on the floor.
Negation strategy is a confound in benchmark design: 'Does it not follow that...?' causes a broad collapse (21 of 29 models drop at least 10 points), and prefix versus word-'not' negation differ by up to 64 points on identical content.
The answer bias is a model-level property carried across all templates, not an artifact of one inference type; restricting to uncontestable inferences still leaves only 10 of 29 models above the floor.
Surface demographics (name nationality, gender, activity scenario) shift accuracy for many models, with the largest effects appearing as task refusal rather than answer flips.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the competence floor became a standard summary for binary-question reasoning benchmarks, many published accuracy numbers would need reinterpretation: a model scoring 55% overall with a 0.10 floor is better described as biased than competent.
The 'Does it not follow' collapse suggests a syntactic processing failure specific to negation inside entailment verbs; a testable extension is to probe logically equivalent negations across other languages or with different presupposition-carrying verbs.
The paper's own probability semantics (probably = probability at least 0.5, with material 'if') implies that at least two of its 'valid' templates admit counterexamples; if gold labels were regraded under that semantics, a model that rejects Chancy Modus Tollens might be reasoning rather than biased.
Answer priors of the kind measured here are in principle removable by calibration; a natural next experiment is to divide out each model's Yes/No prior and ask whether the residual accuracy tracks the logic.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful benchmark and a sensible metric, but the headline '9/29' rests on two mislabeled inference templates; the qualitative conclusion still holds.

read the letter

The core finding—that most LLMs answer these items from a fixed yes/no bias rather than tracking the inference—is probably right, and the competence floor is a genuinely useful summary. But the paper's signature number, 9/29, is not robust to its own semantic commitments.

The benchmark design is careful: 15 templates, 14,320 prompts, systematic variation of question form, negation strategy, and surface content. The competence floor is a clean idea with an absolute baseline. The "does it not follow" collapse is a nice concrete result. The authors are also unusually honest: they re-run the floor on a conservative core and report 10/29.

The soft spots are real. Two gold labels are demonstrably wrong under the paper's own semantics. Chancy Modus Tollens—Pr(φ→ψ), Pr(¬ψ) ⇒ Pr(¬φ)—is not valid with material implication and threshold ≥0.5; the counterexample in the stress test works. Conditional-to-Comparative is likewise invalid. These labels depress model scores and are part of why the headline count is 9/29 rather than 10/29, which is what the conservative core already gives. So the exact "only 9" claim and the "unambiguous correct response" language should not be taken at face value. The paper also provides no confidence intervals or significance tests, and no data or code release, which limits independent verification. These are fixable.

Weighing it: the qualitative conclusion is not in jeopardy. Biased models still fail even the easy invalid templates; the affirmative-only floor still puts most models near or below chance. The problem is the precision of the headline number and the claim that every gold label is unambiguous.

Send to peer review. A good referee will ask for corrected labels, error bars, and released artifacts; with those, this could become a standard benchmark. I'd read it before citing specific numbers.

Referee Report

3 major / 3 minor

Summary. The paper introduces a benchmark for zero-shot inference over English probability operators (probably, might, must), containing 14,320 procedurally generated prompts across fifteen templates (thirteen distinct inference patterns, labeled ten valid and three invalid). It evaluates 29 models and summarizes performance with a 'competence floor' (the minimum of accuracy on Yes-correct and No-correct items), which has an absolute random baseline of 0.5. The central claim is that only 9 of 29 models exceed this floor, indicating that most models answer from a fixed yes/no bias rather than from the logical inference. Secondary findings concern sensitivity to question form, negation strategy, activity, and name nationality/gender.

Significance. The benchmark is well-motivated and large, with systematic controls for question form, negation, and demographic surface features. The competence floor is a simple, parameter-free diagnostic with an absolute baseline, and the yes/no decomposition of accuracy is a useful contribution to LLM evaluation. The robustness checks (conjunction variants, per-template analysis, scale trends) are extensive. If the label and uncertainty issues are fixed, this would be a solid contribution to the literature on LLM reasoning bias. The qualitative conclusion—that answer bias rather than logical inference explains most performance—appears likely to survive, but the exact '9/29' claim and the 'unambiguous correct response' framing are not supported by the current analysis.

major comments (3)

[§3.1.1 / Table 1] Chancy Modus Tollens (schema P r(ϕ→ψ), P r(¬ψ) ⇒ P r(¬ϕ)) and Conditional-to-Comparative (P r(ϕ→ψ) ⇒ ψ⪰ϕ) are marked Valid, but under the semantics fixed in §3.1 (P r(A) iff p(A)≥0.5, material 'if') they are invalid. For CMT, take p(ϕ∧ψ)=0.05, p(ϕ∧¬ψ)=0.5, p(¬ϕ∧ψ)=0.05, p(¬ϕ∧¬ψ)=0.4; then p(ϕ→ψ)=0.5 and p(¬ψ)=0.9 hold while p(¬ϕ)=0.45, so P r(¬ϕ) is false. For CtC, take p(ϕ∧ψ)=0.2, p(ϕ∧¬ψ)=0.4, p(¬ϕ∧ψ)=0.1, p(¬ϕ∧¬ψ)=0.3; then p(ϕ→ψ)=0.6 but p(ψ)=0.3 < p(ϕ)=0.6. The failure is not an artifact of ≥ vs >; strict > gives the same result. Since these items are scored against Yes, a model obeying the paper's own semantics is penalized. The labels must be corrected or the semantics must license these inferences; the claim of 'unambiguous correct responses' (§7) cannot survive as written.
[Abstract / §4.1 / §6 / §7] The headline 'only 9/29 models exceed random baseline' is not stable under the paper's own robustness analysis. §6 reports that the conservative core (three invalid templates plus four basic valid ones) yields 10 of 29 models above 0.5, while removing only the two most contestable templates leaves 9/29. The 9 vs 10 count is small, but the claim is stated as exact. Once the template labels are corrected per comment 1, the count should be recomputed and reported with the exact template set; the qualitative conclusion that most models are below the floor appears robust, but the specific number in the abstract is not.
[§3.4 / §4.1 / Table 3] No uncertainty quantification is provided for the binary 'exceeds chance' classification. Many floors sit just above 0.5 (gemma3:12b 0.518, gpt-oss:120b 0.526), and the number of responses per validity×negation cell is not stated (the paper says '80 surface prompts per template' without saying how this maps to cells). The floor is the minimum of two correlated proportions, so its sampling distribution is not a simple binomial. Report cell-level n, standard errors or bootstrap CIs, and a test (or conservative correction) before deciding which models exceed 0.5. This is needed to make the 9/29 claim quantitative.

minor comments (3)

[§3.1.1] The prose schema for Chancy Modus Tollens uses '¬probably ψ' while Table 1 uses 'P r(¬ψ)'; these are different at p=0.5 and should be disambiguated.
[§2.3] 'This aligns with our results The' is missing a period or conjunction before 'The'.
[§3.2] The phrase 'pining for agreement' is unclear; consider 'angling for agreement' or 'signaling expected agreement'.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's gold labels come from external semantics work, and the competence floor is a defined statistic computed from measured model outputs, not a fitted parameter.

full rationale

The paper's central claim—that most LLMs exhibit a fixed answer bias rather than reliable logical inference—is an empirical measurement, not a derivation from its own definitions. The gold labels for the inference templates are imported from external semantics literature (Yalcin 2010; Kratzer 2012; Lassiter 2017), not from the models' responses, and no parameter is fitted to the evaluation data and then renamed as a prediction. The competence floor (Eq. 4, min of Acc(Yes) and Acc(No)) is an explicitly defined summary statistic with an absolute 0.5 baseline; the '9 of 29 models exceed random chance' result is computed directly from the per-cell accuracies in Table 3, so it is a reported measurement rather than a built-in consequence of a fitted constant. The robustness checks in §4.1 and §6 (affirmative-only floor, conservative template core, removal of contestable axioms) provide independent confirmation rather than circular support. The paper's own acknowledgement that Conditional-to-Comparative and Probably-to-not-probably-not depend on contestable axioms is a validity/robustness caveat, not a circularity: the conservative core still yields the same qualitative conclusion (10/29 above 0.5). The only self-citations (e.g., Shi, Grissom II, and Trinh 2022 in §2.2) are illustrative related work and are not load-bearing for the benchmark's validity or the headline result. No step in the derivation chain reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

Only one hand-set constant enters the definitions (probably ≥0.5); the empirical claims rest on external semantic theories and on methodological choices (first-token parsing, single run, zero-shot) that are stated but not justified with estimates of variance. No new physical or conceptual entities are postulated.

free parameters (1)

Probability threshold for 'probably' = ≥ 0.5
Chosen by hand as 'typical probability assumptions' (§3.1); several template validities (Probably-to-not-probably-not, Conditional-to-Comparative) depend on this threshold and fail at or near the boundary.

axioms (5)

domain assumption 'Probably φ' means p(φ) ≥ 0.5; 'might φ' means p(φ) > 0; 'certain φ' means p(φ) = 1.
Adopted from Lassiter/Yalcin in §3.1; this threshold convention is one of several and changes gold labels at boundaries.
domain assumption Material implication reading of 'if φ then ψ' in Chancy Modus Ponens/Tollens and Conditional-to-Comparative.
Formal schemata in Table 1 use → without stating a semantics; under material implication Chancy Modus Tollens is invalid, so the gold label contradicts the stated semantics.
domain assumption The 10 valid/3 invalid inference patterns from Yalcin (2010) are accepted as gold standard.
The benchmark's correctness labels rest on this external theory; the paper itself notes some templates 'a competent reasoner could contest' (§6).
domain assumption The first token parsed as Yes/No reflects the model's inference; longer generations would not change conclusions.
Stated in Limitations (§6); unconstrained Qwen3:4B required constrained output, and DeepSeek-R1:7b mostly refuses, so first-token parsing conflates refusal with error in some models.
domain assumption Temperature τ=0 and one run per prompt yield stable accuracy.
§3.4; no variance estimate or repeated runs are provided.

reviewed 2026-08-03 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking LLM Competence on Logical Inference over Probability Operators." pith.science (2026). https://pith.science/paper/WQUAFJNJ

@misc{pith2026260727405,
  author       = {Pith},
  title        = {Pith review of: Benchmarking LLM Competence on Logical Inference over Probability Operators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQUAFJNJ}},
  note         = {Machine review of arXiv:2607.27405}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large language models are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a benchmark for reasoning over probability operators--inference over sentences with gradable epistemic modals (e.g., probably, might, must) containing 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the prompts, finding biases across every axis.