Pith. sign in

REVIEW 3 major objections 5 minor 97 references

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs can judge the atoms of a compound answer yet fail to combine them; scoring each atom separately and letting a constraint solver do the composition lifts Macro-F1 from 48 to 77.

desk verdict A clean decomposition-plus-ILP paper with large gains on NEITHER/NOR, but the unverified fidelity of model-authored contrastive hypotheses is the main load-bearing weakness. read the letter →

arxiv 2608.12836 v1 pith:AFJSL53K submitted 2026-08-13 cs.CL cs.AI

Obed Junias , Maria Leonor Pacheco This is my paper · ORCID
classification cs.CLcs.AI
keywords compoundansweroptionslogicalcompositionintegerlinearprogrammingcontrastivehypothesisscoringrelativecalibrationcompositionalitygapNEITHER/NORreasoningLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models often answer multiple-choice questions correctly when the options are simple, yet stumble when the same choice is phrased as two atomic answers joined by AND, OR, or NEITHER/NOR, even when they judge each atom correctly on its own. This paper tries to show that the failure is a composition problem, not a knowledge problem, and that it can be repaired by architecture rather than by more prompting. The proposed framework hides compound options from the model entirely: it splits each option into atoms, elicits contrasting positive and negative judgments about each atom, and hands the calibrated scores to an integer linear program whose constraints encode the operator truth tables and the requirement that exactly one option be valid. The balanced F1 score (Macro-F1) rises from 48.3 to 77.0 on the commonsense benchmark LOGICAL-COMMONSENSEQA and from 47.0 to 75.6 on LOGICAL-SATA, a new reading-comprehension benchmark the paper constructs from SATA-Bench, with NEITHER/NOR — the operator where direct prompting nearly collapses — recovering from about 14 to roughly 77. If the argument holds, part of LLM logical-reasoning failure is a fixable wiring problem in how local judgments are combined, and the same recipe could apply to implication, exclusive or, and longer compounds.

What carries the argument

The carrying mechanism is an operator-constrained integer linear program (ILP, a solver that maximizes a linear objective subject to linear inequalities over binary variables) acting on atomic status variables, fed by paired contrastive hypothesis scoring and an instance-relative calibration. Each unique atomic answer gets one binary variable $y_a$ and each compound option one variable $x_i$; linear inequalities ($x_i \leq y_1$, $x_i \leq y_2$, $x_i \geq y_1 + y_2 - 1$ for AND; the mirror set for OR; the complemented set $x_i \leq 1 - y_1$, $x_i \leq 1 - y_2$, $x_i \geq 1 - y_1 - y_2$ for NEITHER/NOR) make it structurally impossible to mark an option valid unless its atoms satisfy the operator, and the constraint $\sum_i x_i = 1$ forces exactly one choice. The objective maximizes the total calibrated evidence $\sum_{a \in U_C} [s^+_C(a) y_a + s^-_C(a) (1 - y_a)]$, so the solver never generates or paraphrases a judgment — it only selects. The evidence itself comes from a single prompt offering the positive and negative hypotheses as options A and B and normalizing the log probabilities of the first answer tokens, and the new relative calibration adds within-instance features (standardized score, rank among the atoms, gap to the instance maximum) to a logistic mapping, which matters most in MIXED instances where options simultaneously demand accepting and rejecting atoms.

What would settle it

Run the constrained ILP against an unconstrained aggregation that scores each option by a fixed combination of its atoms' calibrated scores (minimum for AND, maximum for OR, negative-product for NEITHER/NOR) and takes the argmax: if the unconstrained version matches the reported 77.0 and 75.6 Macro-F1, the operator constraints are not what produce the gains. In parallel, sample the generated contrastive hypotheses and have annotators check whether each $h^+$ and $h^-$ is logically equivalent to 'the atom holds' and 'the atom does not hold' under the context, because systematic paraphrase drift would mean the scores reward propositions the benchmark never labeled.

Watch

Extended reading notes

Core claim

The paper's central claim is that the compositionality gap in compound answer reasoning is real, separable, and repairable: a model that is handed the individual atomic answers and asked to judge each one in isolation retains far more usable evidence than its performance on the compound option suggests. The authors demonstrate this by decomposing every candidate option into two atomic answers and an explicit operator, collecting positive and negative hypotheses for each atom, normalizing the model's token-level preference between the two into a score in $[0,1]$, calibrating those scores (including a new relative calibration that also encodes each atom's rank and standing within the instance), and then solving an integer linear program whose linear inequalities enforce the exact semantics of AND, OR, and NEITHER/NOR while selecting exactly one option. On Llama-3.1-8B-Instruct the framework lifts Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR (14.0 to 76.8; 12.6 to 73.4), and with near-exact recovery once the gold atomic statuses are supplied to the solver. The conclusion a sympathetic reader should take is that LLM logical failures on these tasks are largely failures of combining local judgments, not of knowing the facts.

Load-bearing premise

The framework trusts that the contrastive hypotheses the model writes for each atomic answer faithfully restate that answer and its negation under the question, so if the model's paraphrases drift, the elicited scores become evidence about a different proposition than the one the benchmark labeled, and every downstream step inherits the drift.

Editorial extensions

If this is right

  • If the framework's results hold, the same model that scored 14 to 48 under direct prompting demonstrably carries the atomic evidence needed to reach roughly 77, so its logical failure is in combination, not in knowing or judging the atoms.
  • Because the ILP encodes operator semantics exactly, the framework cannot be coaxed or generated into violating the logic; every remaining compound error traces to a wrong atomic judgment, turning logical evaluation into a diagnostic of atomic evidence.
  • The recovery pattern — small gains on AND, large on OR, largest on NEITHER/NOR and MIXED — indicates that explicit composition pays off exactly where the possibility structure humans find hardest also defeats the model.
  • The same decomposition-and-constrain recipe transfers across evidence types, commonsense plausibility and passage-grounded reading comprehension, suggesting the compositionality gap is not an artifact of one benchmark or one kind of knowledge.
  • Atoms shared across options are scored once, so the framework guarantees a single consistent judgment for a proposition wherever it appears, something unconstrained generation cannot promise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit: the framework's exact-solver guarantee becomes an upper bound on what a model's atomic evidence can support, so the residual gap between atomic accuracy (about 0.83) and compound accuracy (about 0.76) measures how much the operators themselves demand of the local judgments; one could use it to compare models by how much of the composition their local e
  • Because all results come from one 8-billion-parameter instruction-tuned model, the claim that composition is a distinct burden would be tested by a cross-model replication: if a stronger or differently trained checkpoint shows no NEITHER/NOR collapse under direct prompting, the 'distinct burden' would be model-specific rather than a general property of LLM reasoning.
  • The operator-graded difficulty ordering (AND easiest, NEITHER/NOR hardest) matches the mental-model literature the paper cites; applying the framework to implication, exclusive disjunction, or nested expressions would indicate whether that ordering generalizes beyond the three operators studied.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies multiple-choice questions whose answer options are compounds of two atomic answers joined by AND, OR, or NEITHER/NOR. It proposes a structured framework that deterministically decomposes each option into atoms, uses the same LLM to generate positive and negative contrastive hypotheses for each atom, scores them with paired multiple-choice log-probabilities, calibrates those scores with Platt, isotonic, or a new relative calibration method, and finally combines them with an operator-constrained integer linear program that enforces the truth tables of the operators and selects exactly one option. The framework is evaluated with Llama-3.1-8B-Instruct on LOGICAL-COMMONSENSEQA and on a new benchmark, LOGICAL-SATA, constructed from SATA-Bench. The reported macro-F1 gains over direct prompting are large, especially on NEITHER/NOR (e.g., 14.0 to 76.8 on LOGICAL-COMMONSENSEQA-HV with relative calibration).

Significance. If the main results hold, the paper makes a useful contribution by demonstrating that separating atomic evidence elicitation from logical composition can substantially reduce the compositionality gap on compound answer options, and by providing a new benchmark, LOGICAL-SATA, in this format. The framework is clearly described, the ILP constraints are exact encodings of the operator semantics, the code and datasets are promised publicly, and the comparison across three calibration schemes is systematic. The strength of the empirical claim, however, rests on two assumptions that are not directly tested: that the LLM-generated contrastive hypotheses faithfully preserve the meaning of each atomic answer and its negation, and that the ILP, rather than the decomposition or contrastive scoring alone, is responsible for the reported gains. These are load-bearing because the atomic evidence is the only information that reaches the logical composition layer.

major comments (3)
  1. [Section 3.3, Eq. (1)] The contrastive hypotheses h+ and h- are generated by the same LLM that later scores them, and the pipeline does not verify that the produced h+ is logically equivalent to 'a satisfies C' or that h- is its negation. The prompts in Appendix E.1 instruct the model to use the atomic statement exactly as written and the demonstrations in E.6 are faithful, but nothing checks per-instance equivalence; a generated h+ that drops a modifier or reinterprets an open-ended question would make the elicited scores evidence for a different proposition than the gold-labeled atom. Calibration (Section 3.5) can correct a stationary bias but cannot repair per-instance semantic drift, and the ILP in Section 3.6 only rearranges these already-corrupted scores. Because this assumption is load-bearing for the 48.3-to-77.0 and 47.0-to-75.6 claims, I ask for a control that uses templated or otherwise verified hypotheses (or an NLI/verification step on a sample of instances) to establish that the gains do not depend on paraphrase fidelity.
  2. [Section 5.1, Tables 1 and 2] The main comparison, direct prompting versus structured inference, changes three components at once: option decomposition, contrastive scoring, and ILP-constrained composition. There is no ablation that keeps the atomic evidence fixed and replaces the ILP with a natural-language composition step or with a simple deterministic rule-based combiner over the same calibrated scores. As a result, the paper does not establish that the operator-constrained ILP itself contributes to the observed improvement; the gains could come entirely from decomposition and contrastive scoring. Since the title and introduction emphasize 'operator-constrained ILP inference' as a central contribution, this ablation is needed to support the attribution of the reported gains to the ILP.
  3. [Section 5.1 and Tables 1-6] Most structured-inference rows report a standard deviation of exactly 0.0, while direct-prompting rows show nonzero standard deviations. The paper states that all experiments use temperature 0.7 but does not explain why the structured-inference results have zero variance; presumably log-probability extraction is deterministic whereas direct prompting uses stochastic generation. This should be stated explicitly in the experimental settings, because a reader could otherwise suspect that the 0.0 values reflect an error in the averaging or in the random seeds.
minor comments (5)
  1. [Section 5.1, Paragraph on operator-level gains] The sentence 'paired multiple-choice structured inference raises these to 75.1, and 71.9, respectively, and relative to 76.8 and 73.4' is missing a verb before 'relative' and should be rewritten, for example as 'and relative calibration raises them to 76.8 and 73.4.'
  2. [Section 3.5] The feature vector for relative calibration includes rankC(a), but the ranking convention (ascending versus descending order, and how ties are broken) is not specified; please add this detail to Appendix A or to the main text.
  3. [Appendix E.6] The representative demonstrations show only positive examples (Option A correct) for both benchmarks; adding a negative example or stating how demonstrations are balanced across labels would clarify whether the few-shot demonstrations could bias the model toward selecting A.
  4. [Appendix A] The appendix says all experiments use a random seed of 42 but does not describe how the five runs differ; please specify whether only decoding randomness varies or whether other components (calibration set subsampling, demonstration selection) also vary across runs.
  5. [Section 4] For LOGICAL-SATA, the paper reports that 1,390 eligible source questions are reduced to 1,350, but the selection criterion for dropping the remaining 40 questions is not described; please state the criterion or note that they were dropped by random selection.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the ILP and calibration are trained on labels and evaluated on held-out test sets, and the benchmark claims rest on externally derived data.

full rationale

The core derivation chain is not circular. Atomic scores are produced by a normalized softmax over answer-token log-probabilities in Eq. (1), with no parameter fitted to the test outcome. Calibration, including relative calibration, is explicitly fit on atomic examples from the training set, and all reported test metrics are computed on held-out instances. The ILP in Section 3.6 exactly encodes the operator semantics defined in Section 3.1, and its objective is a sum of calibrated evidence scores; this is a faithful encoding of the task, not a restatement of the answer. The paper even transparently notes that supplying gold atomic statuses gives accuracy 1.00 'by construction,' which is an acknowledged sanity check rather than a disguised prediction. The main self-reference is the use of LOGICAL-COMMONSENSEQA, a benchmark from the authors' prior work, as one of two evaluation sets. That is an independence concern, not a circular derivation: the framework is also evaluated on LOGICAL-SATA, constructed from the external SATA-Bench, and no parameter or constraint is derived from either benchmark's test labels. The concern about LLM-generated contrastive hypotheses h+/h- preserving atomic meaning is a measurement-validity and robustness issue, not a circular reduction, because the hypothesis construction is not defined in terms of the final labels and the pipeline does not assume their equivalence as an input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of LLM-generated hypotheses and atomic labels, plus the exact-one-option benchmark assumption; these are reasonable but should be stress-tested.

free parameters (3)
  • Relative calibration weights w and bias b
    Logistic regression mapping four instance-relative features to the calibrated atomic score, fit on the training set (Sec. 3.5).
  • Platt scaling slope and intercept
    Fitted on the training set for the Platt calibrator (Sec. 3.5).
  • Isotonic calibration mapping
    Non-parametric mapping fitted on the training set (Sec. 3.5).
assumptions (5)
  • domain assumption Each benchmark instance has exactly one valid compound option.
    Used to enforce sum x_i = 1 in the ILP (Sec. 3.6); if this fails, the ILP has no feasible solution or multiple solutions.
  • domain assumption Atomic gold statuses are correct and unambiguous.
    Calibration uses gold atomic labels; errors in labels propagate (Sec. 3.5 and Limitations).
  • domain assumption LLM log-probabilities of first tokens A/B reflect relative evidence for the two hypotheses.
    Score computation in Eq. 1 assumes exp(log-prob) is a sensible weight for the contrastive pair.
  • ad hoc to paper Contrastive hypotheses h+/h- generated by the LLM faithfully represent the atomic answer and its negation.
    Sec. 3.3 relies on model-generated hypotheses; the prompt enforces the exact atomic statement but generation is unverified.
  • standard math Operators AND, OR, NEITHER/NOR follow classical propositional semantics.
    Used to encode ILP constraints in Sec. 3.6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options." pith.science (2026). https://pith.science/paper/AFJSL53K

@misc{pith2026260812836,
  author       = {Pith},
  title        = {Pith review of: From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFJSL53K}},
  note         = {Machine review of arXiv:2608.12836}
}
read the original abstract

Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.

Figures

Figures reproduced from arXiv: 2608.12836 by the authors.

Figure 1
Figure 1. Mental-model representation of the possibili [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed framework. Each compound answer option is decomposed into two atomic [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Structure of LOGICAL-COMMONSENSEQA instances. Operator-specific instances use the same operator across all four options, whereas MIXED instances may contain different operators. • Construct two logically opposing hypotheses. • H+ must state that the atomic satisfies the question constraints. • H- must state that the atomic does not satisfy the question constraints. • Do not determine which hypothesis is correct. Out… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Construction of LOGICAL-SATA from SATA-Bench. Each source instance provides a paragraph, a reading-comprehension question, and independently annotated correct and incorrect atomic answers. Pairs of atomic answers are combined according to the semantics of AND, OR, and …
Figure 5
Figure 5. Figure 5: Representative LOGICAL-COMMONSENSEQA instances covering the AND, OR, NEITHER/NOR, and MIXED settings. Values in parentheses denote ground-truth binary labels for the atomic answers. The highlighted option is the unique gold option. 19 [PITH_FULL_IMAGE:figures/full_fig…
Figure 6
Figure 6. Figure 6: Representative LOGICAL-SATA instances from four source domains, covering the AND, OR, NEI￾THER/NOR, and MIXED settings. Values in parentheses denote ground-truth binary labels for the atomic answers. The highlighted option is the unique gold option. Passage and documen…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 34 canonical work pages

  1. [3]

    The Thirteenth International Conference on Learning Representations , year=

    Logical Consistency of Large Language Models in Fact-Checking , author=. The Thirteenth International Conference on Learning Representations , year=

  2. [4]

    R obust LR : A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners

    Sanyal, Soumya and Liao, Zeyi and Ren, Xiang. R obust LR : A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.653

    doi:10.18653/v1/2022.emnlp-main.653 2022

  3. [7]

    , author=

    Propositional reasoning by model. , author=. Psychological review , volume=. 1992 , publisher=

    1992

  4. [8]

    The Quarterly Journal of Experimental Psychology Section A , volume=

    Are conjunctive inferences easier than disjunctive inferences? A comparison of rules and models , author=. The Quarterly Journal of Experimental Psychology Section A , volume=. 2001 , publisher=

    2001

  5. [9]

    Acta Psychologica , volume=

    The negations of conjunctions, conditionals, and disjunctions , author=. Acta Psychologica , volume=. 2014 , publisher=

    2014

  6. [10]

    Journal of Pragmatics , volume=

    Processing negated sentences with contradictory predicates: Is a door that is not open mentally closed? , author=. Journal of Pragmatics , volume=. 2006 , publisher=

    2006

  7. [11]

    Journal of memory and language , volume=

    How negation is understood: Evidence from the visual world paradigm , author=. Journal of memory and language , volume=. 2014 , publisher=

    2014

  8. [12]

    Journal of Psycholinguistic Research , volume=

    Inhibitory mechanisms in the processing of negations: A neural reuse hypothesis , author=. Journal of Psycholinguistic Research , volume=. 2021 , publisher=

    2021

Show all 97 references
  1. [13]

    PLoS biology , volume=

    Negation mitigates rather than inverts the neural representations of adjectives , author=. PLoS biology , volume=. 2024 , publisher=

    2024

  2. [14]

    arXiv preprint arXiv:2002.05867 , year=

    Transformers as soft reasoners over language , author=. arXiv preprint arXiv:2002.05867 , year=

    2002 arXiv

  3. [15]

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

    Proofwriter: Generating implications, proofs, and abductive statements over natural language , author=. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

    2021

  4. [16]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  5. [17]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  6. [18]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    CONDAQA: A contrastive reading comprehension dataset for reasoning about negation , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  7. [21]

    , author=

    Elimination by aspects: A theory of choice. , author=. Psychological review , volume=. 1972 , publisher=

    1972

  8. [22]

    , author=

    Logical-rule models of classification response times: a synthesis of mental-architecture, random-walk, and decision-bound approaches. , author=. Psychological Review , volume=. 2010 , publisher=

    2010

  9. [23]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) , pages=

    EchoPrompt: instructing the model to rephrase queries for improved in-context learning , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) , pages=

    2024

  10. [24]

    arXiv preprint arXiv:2307.11768 , year=

    Question decomposition improves the faithfulness of model-generated reasoning , author=. arXiv preprint arXiv:2307.11768 , year=

    arXiv

  11. [28]

    2024 , url=

    The Llama 3 Herd of Models , author=. 2024 , url=

    2024

  12. [29]

    A mbig QA : Answering Ambiguous Open-domain Questions

    Min, Sewon and Michael, Julian and Hajishirzi, Hannaneh and Zettlemoyer, Luke. A mbig QA : Answering Ambiguous Open-domain Questions. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.466

    2020 doi

  13. [30]

    2014 , publisher=

    Mental models for the negation of conjunctions and disjunctions , author=. 2014 , publisher=

    2014

  14. [33]

    Psychological science , volume=

    Dual processing in reasoning: Two systems but one reasoner , author=. Psychological science , volume=. 2006 , publisher=

    2006

  15. [34]

    Consciousness and Cognition , volume=

    Evidence that logical reasoning depends on conscious processing , author=. Consciousness and Cognition , volume=. 2008 , publisher=

    2008

  16. [37]

    FOLIO : Natural Language Reasoning with First-Order Logic

    Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and Zhou, Wenfei and Coady, James and Peng, David and Qiao, Yujie and Benson, Luke and Sun, Lucy and Wardle-Solano, Alexander and Szab \'o , Hannah and Zubova, Ekaterina and Burtell, Matthe...

    2024

  17. [40]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  18. [41]

    The eleventh international conference on learning representations , year=

    Least-to-most prompting enables complex reasoning in large language models , author=. The eleventh international conference on learning representations , year=

  19. [42]

    Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    Explaining answers with entailment trees , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages=

    2021

  20. [44]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  21. [46]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  22. [47]

    arXiv preprint arXiv:2203.15556 , year=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    arXiv

  23. [50]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    ConjNLI: Natural language inference over conjunctive sentences , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  24. [51]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    This is not a dataset: A large negation benchmark to challenge large language models , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  25. [52]

    2023 , eprint=

    Decomposed Prompting: A Modular Approach for Solving Complex Tasks , author=. 2023 , eprint=

    2023

  26. [55]

    Advances in Neural Information Processing Systems , volume=

    Satlm: Satisfiability-aided language models using declarative prompting , author=. Advances in Neural Information Processing Systems , volume=

  27. [59]

    AAAI Conference on Artificial Intelligence , year=

    PIQA: Reasoning about Physical Commonsense in Natural Language , author=. AAAI Conference on Artificial Intelligence , year=

  28. [65]

    ArXiv , year=

    Language Models (Mostly) Know What They Know , author=. ArXiv , year=

  29. [73]

    1999 , url=

    Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , author=. 1999 , url=

    1999

  30. [74]

    Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

    Transforming classifier scores into accurate multiclass probability estimates , author=. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

  31. [75]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. https://api.semanticscholar.org/CorpusID:208290939 Piqa: Reasoning about physical commonsense in natural language . In AAAI Conference on Artificial Intelligence

    2019

  32. [76]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  33. [77]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  34. [78]

    Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7358--7370

    2021

  35. [79]

    C Nathan DeWall, Roy F Baumeister, and EJ Masicampo. 2008. Evidence that logical reasoning depends on conscious processing. Consciousness and Cognition, 17(3):628--645

    2008

  36. [80]

    Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston ...

    2024

  37. [81]

    Etienne Fortier-Dubois and Domenic Rosati. 2023. https://doi.org/10.18653/v1/2023.acl-short.72 Using contradictions improves question answering systems . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2023 doi

  38. [82]

    Iker Garc \' a-Ferrero, Bego \ n a Altuna, Javier Alvez, Itziar Gonzalez-Dios, and German Rigau. 2023. This is not a dataset: A large negation benchmark to challenge large language models. In Proceedings of the 2023 conference on empirical methods in natural language processin...

    2023

  39. [83]

    Juan A Garc \' a-Madruga, Sergio Moreno, Nuria Carriedo, Francisco Guti \'e rrez, and Philip N Johnson-Laird. 2001. Are conjunctive inferences easier than disjunctive inferences? a comparison of rules and models. The Quarterly Journal of Experimental Psychology Section A, 54(2...

    2001

  40. [84]

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szab \'o , Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, a...

    2024 doi

  41. [85]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  42. [86]

    de Vries, Maarten de Rijke, and Faegheh Hasibi

    Mohanna Hoveyda, Jelle Piepenbrock, Arjen P. de Vries, Maarten de Rijke, and Faegheh Hasibi. 2026. https://doi.org/10.1007/978-3-032-21289-4_7 Orlog: Resolving complex queries with llms and probabilistic reasoning . In Advances in Information Retrieval: 48th European Conferenc...

    2026 doi

  43. [87]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00407 How can we know when language models know? on the calibration of language models for question answering . Transactions of the Association for Computational Linguistics, 9:962--977

    2021 doi

  44. [88]

    Philip N Johnson-Laird, Ruth M Byrne, and Walter Schaeken. 1992. Propositional reasoning by model. Psychological review, 99(3):418

    1992

  45. [89]

    Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.82 Maieutic prompting: Logically consistent reasoning with recursive explanations . In Proceedings of the 2022 Conferenc...

    2022 doi

  46. [90]

    Obed Junias and Maria Leonor Pacheco. 2026. https://doi.org/10.18653/v1/2026.acl-short.61 LOGICAL - COMMONSENSEQA : A benchmark for logical commonsense reasoning . In Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 2: Short Pa...

    2026 doi

  47. [91]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Thomas Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort,...

    2022 arXiv

  48. [92]

    Nora Kassner and Hinrich Sch \"u tze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7811--7818

    2020

  49. [93]

    Nora Kassner, Oyvind Tafjord, Hinrich Sch \"u tze, and Peter Clark. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.697 B elief B ank: Adding memory to a pre-trained language model for a systematic notion of belief . In Proceedings of the 2021 Conference on Empirical Methods...

    2021 doi

  50. [94]

    Irtaza Khalid, Amir Masoud Nourollah, and Steven Schockaert. 2025. https://doi.org/10.18653/v1/2025.acl-long.433 Large language and reasoning models are shallow disjunctive reasoners . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (...

    2025 doi

  51. [95]

    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. https://doi.org/10.18653/v1/N18-1023 Looking beyond the surface: A challenge set for reading comprehension over multiple sentences . In Proceedings of the 2018 Conference of the North A meri...

    2018 doi

  52. [96]

    Sangeet Khemlani, Isabel Orenes, and Philip N Johnson-Laird. 2014. The negations of conjunctions, conditionals, and disjunctions. Acta Psychologica, 151:1--7

    2014

  53. [97]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. https://arxiv.org/abs/2210.02406 Decomposed prompting: A modular approach for solving complex tasks . Preprint, arXiv:2210.02406

    2023 arXiv

  54. [98]

    Karl Christoph Klauer. 1997. https://doi.org/10.1080/135467897394419 Working memory involvement in propositional and spatial reasoning . Thinking & Reasoning, 3(1):9--47

    1997 doi

  55. [99]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. https://doi.org/10.24963/ijcai.2020/501 Logiqa: A challenge dataset for machine reading comprehension with logical reasoning . In Proceedings of the Twenty-Ninth International Joint Conference on ...

    2020 doi

  56. [100]

    Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.389 Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons . In Proceedings of the 2024 Conference on Empirical Methods in Nat...

    2024 doi

  57. [101]

    Guillermo Macbeth, Eugenia Razumiejczyk, Mar \' a C Crivello, Claudia Bolz \'a n, Carolina I Pereyra Girardi, and Guillermo Campitelli. 2014. Mental models for the negation of conjunctions and disjunctions

    2014

  58. [102]

    Maitrey Mehta, Valentina Pyatkin, and Vivek Srikumar. 2024. https://doi.org/10.18653/v1/2024.naacl-long.7 Promptly predicting structures: The return of inference . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist...

    2024 doi

  59. [103]

    Thorsten Meiser, Karl Christoph Klauer, and Birgit Naumer. 2001. https://doi.org/10.1016/S0001-6918(00)00055-X Propositional reasoning and working memory: the role of prior training and pragmatic content . Acta Psychologica, 106(3):303--327

    2001 doi

  60. [104]

    Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.115 Enhancing self-consistency and performance of pre-trained language models through natural language inf...

    2022 doi

  61. [105]

    Wim De Neys. 2006. Dual processing in reasoning: Two systems but one reasoner. Psychological science, 17(5):428--433

    2006

  62. [106]

    Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.313 LINC : A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers . ...

    2023 doi

  63. [107]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155

    2022 arXiv

  64. [108]

    Maria Leonor Pacheco and Dan Goldwasser. 2021. https://doi.org/10.1162/tacl_a_00357 Modeling content and context with deep relational learning . Transactions of the Association for Computational Linguistics, 9:100--119

    2021 doi

  65. [109]

    Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.198 Plausibly problematic questions in multiple-choice benchmarks for commonsense reasoning . In Findings of the Associatio...

    2024 doi

  66. [110]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.248 Logic- LM : Empowering large language models with symbolic solvers for faithful logical reasoning . In Findings of the Association for Computational Linguistics...

    2023 doi

  67. [111]

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the As...

    2024

  68. [112]

    Matt Pauk and Maria Leonor Pacheco. 2026. https://doi.org/10.18653/v1/2026.eacl-long.160 Mapping the course for prompt-based structured prediction . In Proceedings of the 19th Conference of the E uropean Chapter of the A ssociation for C omputational L inguistics (Volume 1: Lo...

    2026 doi

  69. [113]

    John Platt. 1999. https://api.semanticscholar.org/CorpusID:56563878 Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

    1999

  70. [114]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Measuring and narrowing the compositionality gap in language models . In Findings of the Association for Computational Linguistics: EMNLP 20...

    2023 doi

  71. [115]

    Rajkumar Pujari and Dan Goldwasser. 2019. https://doi.org/10.18653/v1/N19-1403 Using natural language relations between answer choices for machine comprehension . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguist...

    2019 doi

  72. [116]

    Abhilasha Ravichander, Matt Gardner, and Ana Marasovi \'c . 2022. Condaqa: A contrastive reading comprehension dataset for reasoning about negation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8729--8755

    2022

  73. [117]

    Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunctive sentences. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8240--8252

    2020

  74. [118]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  75. [119]

    She, Christopher Potts, Samuel R

    Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023. https://doi.org/10.18653/v1/2023.acl-short.154 S co N e: Benchmarking negation reasoning in language models with fine-tuning and in-context learning . In Proceedings of the 61st Annual Meeting of t...

    2023 doi

  76. [120]

    Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. https://doi.org/10.18653/v1/2021.findings-acl.317 P roof W riter: Generating implications, proofs, and abductive statements over natural language . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 20...

    2021 doi

  77. [121]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  78. [122]

    Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.303 Diagnosing the first-order logical reasoning ability through L ogic NLI . In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan...

    2021 doi

  79. [123]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  80. [124]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  81. [125]

    Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, and Benjamin Van Durme. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.531 Enhancing systematic decompositional natura...

    2024 doi

  82. [126]

    Weijie Xu, Shixian Cui, Xi Fang, Chi Xue, Stephanie Eckman, and Chandan K Reddy. 2025. Sata-bench: Select all that apply benchmark for multiple choice questions. arXiv preprint arXiv:2506.00643

    2025

  83. [127]

    Liang Yao and Yang Yang. 2026. https://doi.org/10.1016/j.eswa.2025.130407 Large language models are contrastive reasoners . Expert Systems with Applications, 301:130407

    2026

  84. [128]

    Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023. Satlm: Satisfiability-aided language models using declarative prompting. Advances in Neural Information Processing Systems, 36:45548--45580

    2023

  85. [129]

    Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326

    2020 arXiv

  86. [130]

    Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694--699

    2002

  87. [131]

    Xiao Zhang, Maria Leonor Pacheco, Chang Li, and Dan Goldwasser. 2016. https://doi.org/10.18653/v1/W16-5906 Introducing DRAIL -- a step towards declarative deep relational learning . In Proceedings of the Workshop on Structured Prediction for NLP , pages 54--62, Austin, TX. Ass...

    2016 doi

  88. [132]

    Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.470 R o MQA : A benchmark for robust, multi-evidence, multi-answer question answering . In Findings of the Association for Computational Linguistics: EMNLP 2023, ...

    2023 doi

  89. [133]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and 1 others. 2022. Least-to-most prompting enables complex reasoning in large language models. In The eleventh international conference ...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.