Pith. sign in

REVIEW 5 major objections 5 minor 50 references

Reasoning Capabilities and Invariability of Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most large language models, including several over 70 billion parameters, perform no better than an always-'no' baseline on simple one-step logical deduction questions, and their answers barely change when the questions are reworded.

desk verdict Useful new shallow-logic benchmark with a robust negative accuracy result, but the headline invariability claim is not verifiable as written because Q1/Q4 negation alignment is never specified. read the letter →

arxiv 2505.00776 v1 pith:K2RVZ7T6 submitted 2025-05-01 cs.CL

Alessandro Raganato , Rafael Peñaloza , Marco Viviani , Gabriella Pasi This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelslogicalreasoningbenchmarkdatasetpromptdependencyinvariabilitychain-of-thoughtpromptingnegationshallow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether current large language models can perform shallow logical reasoning in a stripped-down setting: one-step deductions about geometric figures described entirely in the prompt. Across 24 models and 432 test questions, only a handful—mostly the largest ones—reach around 70 percent accuracy, while most models are statistically indistinguishable from a baseline that always answers 'no.' The paper also argues that model answers are largely invariant under small rewording of the questions, and that chain-of-thought prompting generally helps only when the rationale is requested after the yes/no answer. If the result holds, one-step deductive inference is not a reliably solved capability in current LLMs, and prompt engineering has little leverage on this class of questions.

What carries the argument

The load-bearing object is the four-batch benchmark built on 108 base questions, paired with the McNemar test, a significance test for paired binary outcomes. Each question asks for a yes/no conclusion from two premises in a geometric domain with no real-world priors; the four batches hold the underlying logical content fixed while varying surface form (extra explicit facts, invented shape names, and negation with flipped correct answer). The McNemar test is what turns 'the model answers differently on batch X than batch Y' into a statistical statement: it counts how unbalanced the discordant answer pairs are, and all comparisons in the paper come out with p-values above 0.1. The benchmark's role is to isolate the logical constructor being tested, so a failure can be attributed to reasoning over existence, universal statements, counting, or negation rather than to memorized world knowledge.

What would settle it

Open the published dataset, take a model's Q1 and Q4 answer pairs, invert every Q4 answer, and rerun the McNemar test on the Q1-Q4 pair; if the p-value drops below 0.05, the invariability conclusion is reversed by the correction that the paper appears to omit.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that shallow logical reasoning—one conclusion drawn from two premises, with no case analysis or intermediate memory—is not reliably solved by any of the 24 models tested. The dataset uses 108 base questions about geometric figures, each repeated in four batches: the original phrasing; the same phrasing with explicit facts about shapes; the same questions with invented shape names; and a negated version whose correct answer is flipped. Because 55 of the 108 base questions have 'no' as the correct answer, the baseline that always says 'no' scores 50.93 percent. In zero-shot testing, GPT-4 reaches 72.30 percent overall accuracy, but its confidence interval overlaps with Qwen1.5-72B, Llama-3-70B, Phi-3-small, and recurrentgemma-2b, and the remaining models are, with few exceptions, statistically indistinguishable from that baseline. Few-shot prompting lowers performance, leaving only GPT-4, Qwen72B, Llama70B, and Phi-small distinguishable from the baseline. For the batch pairs not already flagged as different by accuracy, pairwise McNemar tests yield p-values above 0.1, which the paper reads as invariability under language variation, and between-model answer similarity is also high. With chain-of-thought prompting, asking for the rationale before the answer pushes several models far below chance and produces many responses that are not yes/no, while asking for the rationale after the answer generally improves accuracy.

Load-bearing premise

The invariability conclusion assumes that when batch Q4 negates the question and flips the correct answer, the model's Q4 responses were reversed before comparing them with Q1 responses; if the literal answers were compared, a logically correct flip would be counted as variability, and the paper does not state that this reversal was applied.

Editorial extensions

If this is right

  • If the central claim holds, accuracy on this benchmark is a diagnostic of basic logical competence: a model that scores near the always-'no' baseline on one-step deductions cannot be trusted for knowledge-based applications that require checking consistency or certainty.
  • Few-shot prompting made most models worse, so adding a few examples is not a reliable remedy for this class of questions; a deployed system would need a different mechanism.
  • The invariability across Q1-Q4 means a failure is not a quirk of one phrasing: if a model gets a question wrong, semantically equivalent rephrasings are likely to be wrong too, which makes the error harder to avoid by prompt rewriting.
  • Chain-of-thought prompting should be configured to ask for the rationale after the yes/no answer; the before-answer ordering can push several models far below chance and generates many responses that cannot be parsed as yes/no.
  • The dataset's four batches isolate specific logical constructors, so they can be used to identify which expressions (existential, universal, number restrictions, negation) a given model handles and which it does not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In Section V-A, batch Q4 is described as negating the question and flipping its answer, and an invariable model is said to 'provide the same answers in each batch.' The paper never states that Q4 responses were inverted before the McNemar comparison; without that inversion, logically correct flips would be counted as variability, so a reader should verify this before relying on the invariability c
  • The four-batch protocol could be extended to paraphrases generated by another LLM: the paper's invariability claim is about small hand-designed variants, and it is untested whether invariance survives larger surface variation.
  • The Q1/Q4 pair suggests a targeted probe of negation handling: compare accuracy on Q1 with accuracy on Q4 after inverting Q4 labels; a model that scores at chance on both may be systematically flipping its answers, which aggregate accuracy alone cannot reveal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a new benchmark for shallow logical reasoning, consisting of 108 questions about geometric figures under four batches (Q1–Q4). Q1 is the original set; Q2 adds explicit geometric facts; Q3 replaces shape names with invented terms; Q4 negates the questions and flips the correct answers. The authors evaluate 24 LLMs under zero-shot and few-shot prompting, and 22 LLMs under two chain-of-thought variants. They report per-batch and aggregate accuracies, compare models against a 'always no' baseline, and use McNemar tests to claim that model answers are largely invariant across the four batches. The paper's central claims are that LLMs cannot yet reliably solve one-step logical deductions, that performance degrades in the few-shot setting, and that answer behavior is stable under small linguistic variations.

Significance. The dataset is a genuinely useful contribution: the domain-free geometric setting reduces reliance on world knowledge and memorized phrases, and the four-batch design is a sensible way to probe prompt sensitivity. The finding that many LLMs, including large ones, perform far below ceiling on these simple deductions is a meaningful, falsifiable result. The authors also make their data publicly available and report repeated runs with different seeds, which supports reproducibility of the accuracy tables. However, the invariability analysis—a central part of the title, abstract, and conclusions—is not verifiable as written because the handling of the negated Q4 batch in the McNemar comparison is ambiguous, and the filtering of tested pairs is not described. If the authors clarify the recoding and report the full test results, the paper could be a valuable empirical contribution; as it stands, the main invariability claim needs revision.

major comments (5)
  1. [V-A] The invariability test for Q1 vs. Q4 is ambiguous and potentially internally inconsistent. The paper states that the fourth batch 'only negates the question (and its answer)' and then says that an invariable model 'should provide the same answers in each batch.' But negating a question flips its correct answer, so a logically consistent model should produce opposite yes/no responses on Q1 and Q4. If the raw Q1 and Q4 answers were compared in the McNemar test without recoding Q4 responses (yes↔no), correct logical behavior would be counted as variability. The manuscript never states whether such recoding was performed, nor are the raw answer matrices or analysis code provided. As written, the central invariability conclusion is unverifiable.
  2. [V-A] The procedure for selecting pairs for the McNemar test is not a valid basis for the invariability claim. The authors state that the test was applied 'only to the pairs that were not already deemed as statistically different from the accuracy analysis; all other cases clearly yield variable answers.' This ex post filtering removes precisely the pairs where answer changes are detectable and makes the subsequent 'all pairwise statistics yield a p-value over 0.1' statement uninterpretable: the set of tested pairs is not specified, and the rule itself biases the analysis toward finding invariance. The authors should report McNemar results for all pairwise batch comparisons without such filtering, or provide a pre-registered justification.
  3. [I and V] The paper contradicts itself about how many models are distinguishable from the baseline. The Introduction states 'Most other models are indistinguishable from the baseline,' while Section V states 'The other models are mostly statistically distinguishable from the baseline.' Table I shows that several models, such as recurrentgemma-2b (67.98 ± 2.49), Phi-3-mini (65.28 ± 2.59), and Qwen72B (70.60 ± 2.48), have confidence intervals far above the baseline of 50.93. The manuscript should state which claim is intended and support it with explicit statistical comparisons rather than relying on informal reading of confidence intervals.
  4. [IV and V] The statistical comparisons between models rely on the intersection of 95% confidence intervals, e.g., 'its difference with Qwen72B, Llama70B, Phi-small, recurrentgemma-2b are not statistically significant (their confidence intervals intersect).' Overlapping confidence intervals do not imply a non-significant difference; this is a well-known fallacy. Formal tests (or proper simultaneous intervals) are needed for any claim of statistical equivalence or non-inferiority among models.
  5. [V-A and IV-A] The paper does not specify how the three runs with different seeds are used in the McNemar invariability tests. Accuracy tables report means over runs, but the McNemar test requires paired individual answers. The authors should state whether the test uses answers from a single run, a pooled set, or a majority vote, and they should provide the answer-level data to make the analysis reproducible.
minor comments (5)
  1. [V] In the few-shot paragraph, 'as in one-shot prompting' should be 'as in zero-shot prompting'.
  2. [V-A] The manuscript reports that 'all the pair-wise statistics obtained from the McNemar test yield a p-value over 0.1' but does not provide the actual p-values, test statistics, or the number of discordant pairs. Adding a table with these values would substantially improve the empirical support.
  3. [III-B] The example Q4 sentence says 'the fourth batch only negates the question (and its answer)'; this wording is confusing because negating a question does not leave the answer token unchanged. Clarifying the intended semantics and the expected response behavior would remove ambiguity.
  4. [References] Reference [31] lists the venue as 'Proc. EMLP 2023'; this appears to be a typo for EMNLP 2023. Also, the model name 'stablelm-2-1 6b-chat' in Tables I and II has an inconsistent space and should be rendered uniformly.
  5. [V] The text says 'in most cases, significantly' when describing CoTafter improvements, but no statistical test for the CoT comparison is described. Either add the test or soften the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark study with independent baseline and fixed ground-truth answers.

full rationale

This paper is an empirical evaluation, not a derivation. The benchmark questions and their correct binary answers are fixed in advance by the logical truth conditions of the described geometric scenarios; the always-'no' baseline is an explicit independent reference computed from the label distribution (55/108, approximately 51%), and no model response or fitted parameter is used to define the target outcome. Accuracy, confidence intervals, and McNemar tests compare independent model outputs against that fixed ground truth, so the central claims (low accuracy, invariance across batches, CoT effects) are not equivalent to the inputs by construction. The only self-citation present, Viviani and Pasi reference [47], appears in the future-work paragraph about misinformation detection and is not load-bearing for any result. The possible ambiguity about whether Q4 answers were reversed before the Q1-vs-Q4 McNemar comparison is a methodological and interpretational concern about the invariability test, not a circularity: even if the comparison were flawed, the conclusion would be an incorrect empirical inference rather than a tautology derived from the paper's own definitions. No circular step can be quoted because no step reduces a claimed prediction to a fitted input or self-citation chain. The paper is therefore self-contained as an experimental study, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical benchmark study, so there are no fitted free parameters and no invented entities. The load-bearing assumptions are the scoring rule for non-boolean responses, the comparability of Q1 and Q4 in the invariability test, the statistical treatment of repeated runs, and the correctness of the dataset labels. The Q4 comparability assumption is the most fragile because it directly affects the paper's signature invariability claim.

assumptions (4)
  • domain assumption Binary answers not starting with "yes" or "no" are considered wrong, with void responses scored as incorrect.
    Section V notes StableLM responses were void and counted as wrong; this scoring choice can understate accuracy for models with instruction-following failures.
  • ad hoc to paper Q4's negated questions can be compared directly with Q1's original questions in the McNemar invariability analysis.
    Section V-A states the fourth batch negates the question and flips the answer, yet the invariability comparison is described as checking for the same answers; no recoding of Q4 responses before testing is specified. If literal answers were compared, the test measures the wrong thing.
  • domain assumption Repeated runs with different seeds can be treated as independent observations for confidence intervals and McNemar tests.
    Section IV-A reports accuracy means over 3 runs and uses normal approximation; the dependency structure across repeated runs per item is not addressed.
  • domain assumption The ground-truth answers for the 432 generated questions are correct.
    The paper does not describe a validation procedure or independent verification of the labels for the generated questions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Capabilities and Invariability of Large Language Models." pith.science (2026). https://pith.science/paper/K2RVZ7T6

@misc{pith2026250500776,
  author       = {Pith},
  title        = {Pith review of: Reasoning Capabilities and Invariability of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K2RVZ7T6}},
  note         = {Machine review of arXiv:2505.00776}
}
read the original abstract

Large Language Models (LLMs) have shown remarkable capabilities in manipulating natural language across multiple applications, but their ability to handle simple reasoning tasks is often questioned. In this work, we aim to provide a comprehensive analysis of LLMs' reasoning competence, specifically focusing on their prompt dependency. In particular, we introduce a new benchmark dataset with a series of simple reasoning questions demanding shallow logical reasoning. Aligned with cognitive psychology standards, the questions are confined to a basic domain revolving around geometric figures, ensuring that responses are independent of any pre-existing intuition about the world and rely solely on deduction. An empirical analysis involving zero-shot and few-shot prompting across 24 LLMs of different sizes reveals that, while LLMs with over 70 billion parameters perform better in the zero-shot setting, there is still a large room for improvement. An additional test with chain-of-thought prompting over 22 LLMs shows that this additional prompt can aid or damage the performance of models, depending on whether the rationale is required before or after the answer.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 33 canonical work pages

  1. [1]

    Attention is all you need,

    A. V aswani, N. Shazeer et al. , “Attention is all you need,” in NeurIPS, vol. 30. Curran Associates, Inc., 2017

    work page 2017

  2. [2]

    Emergent abilities of large language models,

    J. Wei, Y . Tay et al. , “Emergent abilities of large language models,”

    work page

  3. [3]

    Are emergent ab ilities of large language models a mirage?

    R. Schaeffer, B. Miranda, and S. Koyejo, “Are emergent ab ilities of large language models a mirage?” 2023. [Online]. Availab le: https://arxiv.org/abs/2304.15004

    arXiv 2023

  4. [4]

    Beyond the imitation game: Quantifying and extrap olating the capabilities of language models,

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, and others, “Beyond the imitation game: Quantifying and extrap olating the capabilities of language models,” 2023. [Online]. Avai lable: https://arxiv.org/abs/2206.04615

    arXiv 2023

  5. [5]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” 2023. [Onlin e]. Available: https://arxiv.org/abs/2303.12712

    arXiv 2023

  6. [6]

    Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,

    C. Wang, X. Liu et al. , “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,” 202 3. [Online]. Available: https://arxiv.org/abs/2310.07521

    arXiv

  7. [7]

    Reasoning with language model prompting: A survey,

    S. Qiao, Y . Ou et al. , “Reasoning with language model prompting: A survey,” in Proc. 61st Annual Meeting of the ACL (V olume 1: Long Papers). Toronto, Canada: ACL, Jul. 2023, pp. 5368–5393

    work page 2023

  8. [8]

    The winograd schema chal lenge: evaluating progress in commonsense reasoning,

    L. Morgenstern and C. L. Ortiz, “The winograd schema chal lenge: evaluating progress in commonsense reasoning,” in Proc. AAAI’15 . AAAI Press, 2015, p. 4024–4025

    work page 2015

Show all 50 references
  1. [9]

    Diagnos ing the first-order logical reasoning ability through LogicNLI,

    J. Tian, Y . Li, W. Chen, L. Xiao, H. He, and Y . Jin, “Diagnos ing the first-order logical reasoning ability through LogicNLI,” i n Proc. EMNLP 21, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. ACL, Nov. 2021, pp. 3738–3747

    2021

  2. [10]

    LogiQA 2.0—an improved dataset for logical reasoning in na tural language understanding,

    H. Liu, J. Liu, L. Cui, Z. Teng, N. Duan, M. Zhou, and Y . Zha ng, “LogiQA 2.0—an improved dataset for logical reasoning in na tural language understanding,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 2947–2962, 2023

    2023

  3. [11]

    From lsat: The progress and challenges of complex r ea- soning,

    S. Wang, Z. Liu, W. Zhong, M. Zhou, Z. Wei, Z. Chen, and N. Duan, “From lsat: The progress and challenges of complex r ea- soning,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 30, p. 2201–2216, apr 2022

    2022

  4. [12]

    Exploring self-supervised logic-enhanced training for large langua ge models,

    F. Jiao, Z. Teng, B. Ding, Z. Liu, N. F. Chen, and S. R. Joty , “Exploring self-supervised logic-enhanced training for large langua ge models,” in NAACL. Association for Computational Linguistics, 2024

    2024

  5. [13]

    On the paradox of learning to reason from data,

    H. Zhang, L. H. Li, T. Meng, K.-W. Chang, and G. V . den Broe ck, “On the paradox of learning to reason from data,” 2022

    2022

  6. [14]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Y u, D. Su, Y . Xu, E. Ishii, Y . J . Bang, A. Madotto, and P . Fung, “Survey of hallucination in natural language generation,” ACM Comput. Surv. , vol. 55, no. 12, mar 2023

    2023

  7. [15]

    Working memo ry involve- ment in propositional and spatial reasoning,

    K. C. Klauer, R. Stegmaier, and T. Meiser, “Working memo ry involve- ment in propositional and spatial reasoning,” Thinking & Reasoning , vol. 3, no. 1, pp. 9–47, 1997

    1997

  8. [16]

    Pre-train, prompt, and predict: A systematic survey of prompting metho ds in natural language processing,

    P . Liu, W. Y uan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig , “Pre-train, prompt, and predict: A systematic survey of prompting metho ds in natural language processing,” ACM Comput. Surv. , vol. 55, no. 9, jan

  9. [17]

    Baader, D

    F. Baader, D. Calvanese, D. McGuinness, D. Nardi, and P . Patel- Schneider, Eds., The Description Logic Handbook: Theory, Implemen- tation, and Applications , 2nd ed. Cambridge University Press, 2007

    2007

  10. [18]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q . V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  11. [19]

    Evaluating large language models: A comprehensive survey,

    Z. Guo, R. Jin, C. Liu, Y . Huang, D. Shi, L. Y u, Y . Liu, J. Li , B. Xiong, D. Xiong et al. , “Evaluating large language models: A comprehensive survey,” arXiv preprint arXiv:2310.19736 , 2023

    2023 arXiv

  12. [20]

    Knowledge Engineer ing Using Large Language Models,

    B. P . Allen, L. Stork, and P . Groth, “Knowledge Engineer ing Using Large Language Models,” TGDK, vol. 1, no. 1, pp. 3:1–3:19, 2023

    2023

  13. [21]

    Evaluating language models for knowledge base completion,

    B. V eseli, S. Singhania, S. Razniewski, and G. Weikum, “ Evaluating language models for knowledge base completion,” in Proc. ESWC 2023, ser. LNCS, C. Pesquita, E. Jim´ enez-Ruiz et al. , Eds., vol. 13870. Springer, 2023, pp. 227–243

    2023

  14. [22]

    Assessing t he factual accuracy of generated text,

    B. Goodrich, V . Rao, P . J. Liu, and M. Saleh, “Assessing t he factual accuracy of generated text,” in Proc. 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . New Y ork, NY , USA: ACM, 2019, p. 166–175

    2019

  15. [23]

    How context affects language models’ factual pr edictions,

    F. Petroni, P . Lewis, A. Piktus, T. Rockt¨ aschel, Y . Wu, A. H. Miller, and S. Riedel, “How context affects language models’ factual pr edictions,” in Automated Knowledge Base Construction , 2020. [Online]. Available: https://openreview.net/forum?id=025X0zPfn

    2020

  16. [24]

    Are large language models really good logical reasoners? a comprehen sive eval- uation from deductive, inductive and abductive views,

    F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria, “Are large language models really good logical reasoners? a comprehen sive eval- uation from deductive, inductive and abductive views,” arXiv preprint arXiv:2306.09841, 2023

    2023 arXiv

  17. [25]

    Logicbench: Towards systematic evaluation o f logical reasoning ability of large language models,

    M. Parmar, N. V arshney, N. Patel, M. Luo, S. Mashetty, A. Mitra, and C. Baral, “Logicbench: Towards systematic evaluation o f logical reasoning ability of large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui stics, 2024

    2024

  18. [26]

    Cladder: A benchmark to assess causal reasoning capabilities of langu age models,

    Z. Jin, Y . Chen, F. Leeb, L. Gresele, O. Kamal, Z. Lyu, K. B lin, F. G. Adauto, M. Kleiman-Weiner, M. Sachan, and B. Sch¨ olkopf, “Cladder: A benchmark to assess causal reasoning capabilities of langu age models,” in Proceedings of NeurIPS 2023 , 2023

    2023

  19. [27]

    Llms are prone t o fallacies in causal inference,

    N. Joshi, A. Saparov, Y . Wang, and H. He, “Llms are prone t o fallacies in causal inference,” CoRR, vol. abs/2406.12158, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2406.12158

    2024 doi arXiv

  20. [28]

    Large language models cannot self-correct reason ing yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Y u, X. Song , and D. Zhou, “Large language models cannot self-correct reason ing yet,” in Proc. ICLR’24 , 2024

    2024

  21. [29]

    On the paradox of learning to reason from data,

    H. Zhang, L. H. Li, T. Meng, K.-W. Chang, and G. V an den Broeck, “On the paradox of learning to reason from data,” in Proceedings of IJCAI 2023 , aug 2023. [Online]. Available: http://starai.cs.ucla.edu/papers/ZhangIJCAI23.pdf

    2023

  22. [30]

    Le arning deductive reasoning from synthetic corpus based on formal l ogic,

    T. Morishita, G. Morio, A. Y amaguchi, and Y . Sogawa, “Le arning deductive reasoning from synthetic corpus based on formal l ogic,” in Proc. ICML. PMLR, 2023, pp. 25 254–25 274

    2023

  23. [31]

    This is not a dataset: A large negation benchmark to challen ge large language models,

    I. Garc´ ıa-Ferrero, B. Altuna, J. Alvez, I. Gonzalez-D ios, and G. Rigau, “This is not a dataset: A large negation benchmark to challen ge large language models,” in Proc. EMLP 2023 . Singapore: Association for Computational Linguistics, Dec. 2023, pp. 8596–8615. [ Online]. A...

    2023

  24. [32]

    Lang uage models are not naysayers: an analysis of language models on negation benchmarks,

    T. H. Truong, T. Baldwin, K. V erspoor, and T. Cohn, “Lang uage models are not naysayers: an analysis of language models on negation benchmarks,” in Proc. *SEM 2023 . Toronto, Canada: ACL, Jul. 2023, pp. 101–114. [Online]. Available: https://aclanthology.org/2023.starsem-1.10

    2023

  25. [33]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatira ju, S. Pathak, L. Sifre, M. Rivi` ere, M. S. Kale, J. Love et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  26. [34]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models,

    S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Mu raru, A. Gu, R. Haroun, L. Berrada, Y . Chen, S. Srinivasan et al. , “Griffin: Mixing gated linear recurrences with local attention for efficient language models,” arXiv preprint arXiv:2402.19427 , 2024

    2024 arXiv

  27. [35]

    Stable lm 2 1.6 b technical report,

    M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinsk yi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta et al., “Stable lm 2 1.6 b technical report,” arXiv preprint arXiv:2402.17834 , 2024

    2024 arXiv

  28. [36]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  29. [37]

    Phi- 3 technical report: A highly capable language model locally on your phone,

    M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah , H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl et al. , “Phi- 3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219 , 2024

    2024 arXiv

  30. [38]

    Yi: Open foundation models by 01. ai,

    A. Y oung, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. L i, J. Zhu, J. Chen, J. Chang et al., “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P . Albert, A. Almahairi , Y . Babaei, N. Bashlykov, S. Batra, P . Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [40]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Availab le: https://github.com/meta-llama/llama3/blob/main/MODEL CARD.md

    2024

  33. [41]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L . Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  34. [42]

    Dekking, A Modern Introduction to Probability and Statistics: Under - standing Why and How , ser

    F. Dekking, A Modern Introduction to Probability and Statistics: Under - standing Why and How , ser. Springer Texts in Statistics. Springer, 2005. [Online]. Available: https://books.google.it/books?id =XLUMIlombgQC

    2005

  35. [43]

    Note on the sampling error of the differenc e between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the differenc e between correlated proportions or percentages,” Psychometrika, vol. 12, pp. 153– 157, 1947

    1947

  36. [44]

    A survey of fake news: Fundamen tal theories, detection methods, and opportunities,

    X. Zhou and R. Zafarani, “A survey of fake news: Fundamen tal theories, detection methods, and opportunities,” ACM Computing Surveys, vol. 53, no. 5, pp. 1–40, 2020

    2020

  37. [45]

    Cont ent based fake news detection using knowledge graphs,

    J. Z. Pan, S. Pavlova, C. Li, N. Li, Y . Li, and J. Liu, “Cont ent based fake news detection using knowledge graphs,” in Proceedings of ISWC 2018, Part I . Springer, 2018, pp. 669–683

    2018

  38. [46]

    Evaluation of fake news detection with knowledge-enhanced language mo dels,

    C. Whitehouse, T. Weyde, P . Madhyastha, and N. Komninos , “Evaluation of fake news detection with knowledge-enhanced language mo dels,” in Proc. Intern. AAAI Conf. on web and social media , vol. 16, 2022, pp. 1425–1429

    2022

  39. [47]

    Credibility in social media: op inions, news, and health information—a survey,

    M. Viviani and G. Pasi, “Credibility in social media: op inions, news, and health information—a survey,” Wiley interdisciplinary reviews: Data mining and knowledge discovery , vol. 7, no. 5, p. e1209, 2017

    2017

  40. [48]

    Knowledge engineer ing using large language models,

    B. P . Allen, L. Stork, and P . Groth, “Knowledge engineer ing using large language models,” arXiv preprint arXiv:2310.00637 , 2023

    2023 arXiv

  41. [2022]

    Available: https://arxiv.org/abs/2206

    [Online]. Available: https://arxiv.org/abs/2206. 07682

  42. [2023]

    Available: https://doi.org/10.1145/356 0815

    [Online]. Available: https://doi.org/10.1145/356 0815

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.