Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering

Hanna Abi Akl

REVIEW 4 major objections 6 minor 39 references

Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that small language models can be fed compact formal logic grammars instead of natural language and still reason nearly as well, with CLIF matching NL on Flan-T5-small and falling only slightly behind on Flan-T5-large.

desk verdict Useful new grammar comparison on FOLIO, but the CLIF-vs-NL claim is unproven because the CLGC pipeline is not meaning-preserving and demonstrably corrupts at least one target grammar. read the letter →

arxiv 2509.10249 v1 pith:LULR7HGA submitted 2025-09-12 cs.AI

Hanna Abi Akl This is my paper · ORCID

classification cs.AI

keywords smalllanguagemodelslogicalreasoningknowledgerepresentationCLIFcommonlogicFOLIOgrammarconstructionontologyengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a first attempt to answer whether natural language is the best input for small language models solving first-order logic reasoning tasks. On the FOLIO dataset, with supervised fine-tuning, representing premises and conclusions in the compact formal grammar CLIF matches natural-language accuracy on Flan-T5-small (0.4384 vs 0.4384) and comes close on Flan-T5-large (0.6157 vs 0.6600). The authors take this as evidence that a more compact logical language can substitute for natural language while preserving reasoning performance. If this holds, ontology engineering could be bootstrapped by feeding SLMs formal logical statements rather than verbose English, reducing input length while keeping the model's reasoning abilities intact.

What carries the argument

The machinery is a pipeline that converts first-order logic formulas between grammars: it parses FOL with a BNF grammar, builds a parse tree, maps the tree to a target grammar (CLIF, CGIF, TFL, TFL+, MINIFOL), and regenerates the text. This pipeline plus the Syllogistic Evaluation Framework (SEF), which classifies each FOLIO problem as Disjunctive, Hypothetical, Categorical, or Complex, defines the experimental comparison. The conversion step is load-bearing because all accuracy differences across grammars are attributed to the language itself; if the conversions change logical meaning, the comparison collapses.

What would settle it

Count, over the whole FOLIO test and validation sets, how many transformed formulas differ in satisfiability or truth value from their FOL originals (e.g., the shown MINIFOL output for ¬(Manager(james) ⊕ AppearIn(james, company)) is (manager(james) ∧ appearin(james, company)), which is satisfiable when the original is false). If even a small percentage of inputs are corrupted, the accuracy comparisons across grammars no longer compare the same reasoning problems.

Watch

Extended reading notes

Core claim

The central claim is that the choice of the formal representation matters less than the literature assumes: a compact logical grammar, CLIF, can carry first-order reasoning tokens nearly as well as natural language in small models. The evidence comes from a controlled comparison across six models, seven input languages, and three learning settings, with FOLIO as the fixed reasoning benchmark. In the strongest setting, supervised fine-tuning on an A100 GPU, Flan-T5-large reaches 0.6157 accuracy with CLIF against 0.6600 with natural language, and Flan-T5-small ties natural language exactly at 0.4384. The authors conclude that no single grammar outperforms natural language, but compact formalisms come close enough to be viable substitutes, especially for models under three billion parameters.

Load-bearing premise

The conversion from first-order logic to each alternative grammar must preserve the meaning of every formula, but no equivalence proof or automated verification is supplied, and Table 3 shows a MINIFOL output that is not equivalent to its FOL input.

Editorial extensions

If this is right

Compact formal grammars like CLIF can be used to shorten LM inputs without giving away much reasoning accuracy, so they may be practical for low-resource settings.
The conclusion is not that NL is best, but that a compact grammar is competitive; thus grammar choice should be measured alongside model size and training method.
Grammar prompting helps in zero-shot but not few-shot or fine-tuning, so its benefit is setting-dependent.
Tokenizer re-training on a compact grammar can boost small models (e.g., Flan-T5-small on TFL+) but does not scale to larger models.
SEF-based breakdown shows performance is concentrated in Disjunctive and Hypothetical syllogisms, with Categorical too sparse to evaluate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the CLGC conversions were verified to preserve first-order meaning, the accuracy gap between CLIF and NL would measure representation cost directly; one testable extension is to run the same comparison on datasets like ProofWriter or RuleTaker, where formulas are synthetic and can be checked for equivalence exactly.
The observed CLIF-vs-NL closeness suggests an input-compression effect: the model may be using the same latent reasoning machinery while struggling less with surface forms; this could be tested by measuring accuracy versus the amount of fine-tuning data for CLIF and NL.
For ontology engineering, the practical implication is that SLMs could consume ontology axioms in CLIF directly, shrinking prompts and avoiding paraphrase ambiguity; this is an extension beyond the paper's experiments, which stop at FOLIO-style reasoning.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the take: this is a workshop-grade preliminary study with a genuinely new empirical comparison—seven input grammars for small-model first-order reasoning on FOLIO—but the headline claim that CLIF can substitute for natural language is not yet supported because the pipeline that generated the CLIF data is unverified and demonstrably corrupts at least one target grammar.

The genuinely new thing is the systematic comparison itself, and the CLGC transformation pipeline as a reusable tool. The idea that input representation affects LM reasoning is already known, but nobody has laid out CLIF, CGIF, TFL, TFL+, MINIFOL, FOL and NL side by side on the same benchmark with sub-3B models. The SEF syllogism categorization is a nice addition. The paper is honest about being a first step and describes the setup well enough to reproduce.

What the paper does well: the results tables are informative, and the observation that CLIF is competitive with NL on Flan-T5-large in full supervised fine-tuning (0.6157 vs 0.6600 accuracy) is plausible and worth chasing.

The soft spots are real, though correctable. The biggest is the CLGC pipeline. Table 3 shows a direct counterexample: the FOL formula ¬(Manager(james) ⊕ AppearIn(james, company)) is transformed into MINIFOL as (manager(james) ∧ appearin(james, company)), which is not logically equivalent—XOR negation does not reduce to conjunction. Two other rows also drop negations. That means the pipeline is not meaning-preserving. If the same class of error appears in any nontrivial fraction of the 1,204 generated CLIF examples, the CLIF-vs-NL gap could come from corrupted problem content rather than grammar quality. The authors mention manual grammar revisions but give no equivalence proof, no automated check, and no released transformed data. Second, all head-to-head claims rest on single runs without error bars or significance tests. Third, comparing Flan-T5-large with LoRA (Table 5) to full fine-tunes (Table 6) mixes two variables, so statements about model scaling are muddy. Fourth, some 8-shot tables show identical accuracy across all grammars (e.g., Gemma-2-2b-it at 0.3546), which suggests the model is ignoring the examples; that deserves at least a sentence of discussion.

None of this is fatal for a first look. The research question is sound, and the paper is coherent in what it attempts. The citation pattern is appropriate. I would send this to a serious workshop referee, with the expectation of heavy revision. For my own work, I would not yet cite the accuracy numbers, but I would cite the comparison as prior art if I were writing in this exact niche. It is a good reading-group paper because the pipeline-fidelity problem makes for a sharp methodological discussion.

Referee Report

4 major / 6 minor

Summary. The manuscript investigates whether small language models (SLMs) can reason over first-order logic problems when the input is expressed in formal grammars rather than natural language. The authors introduce the Syllogistic Evaluation Framework (SEF) for classifying FOLIO reasoning pairs and the Common Logic Grammar Construction (CLGC) pipeline that translates the FOLIO FOL annotations into CLIF, CGIF, TFL, TFL+, and a custom MINIFOL. They then evaluate Flan-T5-small/base/large, GPT-2, Phi-3.5-mini-instruct, and Gemma-2-2b-it under supervised fine-tuning (with and without LoRA), zero-shot prompting, and 8-shot prompting, with additional variations including grammar-context passing and tokenizer re-training. The main claim is that CLIF, a compact formal language, can largely substitute for natural language while preserving SLM reasoning performance, based on CLIF matching or closely trailing NL in Tables 5 and 6. The paper concludes that compact formal representations are viable for bootstrapping ontology engineering with SLMs, though it positions the work as preliminary.

Significance. The paper addresses a timely and practical question—whether the input representation of logical problems can be made more compact without hurting SLM performance—with a broad configuration matrix spanning multiple models, grammars, and training regimes on the public FOLIO benchmark. The SEF syllogism-type breakdown in Table 13 is a useful descriptive lens, and the tokenizer re-training comparison in Table 12 is an interesting exploratory direction. The central claim is falsifiable and, if substantiated, would be practically relevant for ontology engineering because it would permit more efficient encoding of logical knowledge for SLMs. However, the empirical evidence as presented does not yet establish the claim: the transformation pipeline that generates the formal-language datasets is not shown to be meaning-preserving, and the headline comparisons lack statistical support. The paper's strengths are its scope and the specificity of the research questions; those strengths would be better leveraged by pairing the wide comparison with a smaller set of rigorously verified and repeated experiments.

major comments (4)

[Section 3.2.3 and Table 3] The CLGC pipeline is not demonstrated to preserve the truth conditions of the FOL formulas it translates. For instance, the formula ¬(Manager(james) ⊕ AppearIn(james, company)) is rendered in MINIFOL as (manager(james) ∧ appearin(james, company)), which is not logically equivalent; the rows for ∀x ((Employee(x)∧(¬In(x, homecountry)))→Work(x, home)) and ∀x (Manager(x) → ¬Work(x, home)) drop the negations on ¬In and ¬Work in both MINIFOL and CGIF. Since all formal-language datasets used in the Section 4 experiments are produced by this pipeline, the accuracy comparisons for CLIF, CGIF, TFL, TFL+, and MINIFOL are only interpretable if every generated formula is verified to be logically equivalent to its FOL source. At minimum, the authors need to provide an automated equivalence check (or a released dataset with per-formula verification) and to rerun or clearly qualify the affected results.
[Sections 4.1-4.2, Tables 5-6] All reported metrics are from single runs with no error bars, confidence intervals, or significance tests. The headline CLIF-vs-NL difference for Flan-T5-large in Table 6 is 0.6600 vs 0.6157 in accuracy; without repeated seeds or a significance test, this gap is within plausible random variation and cannot support the claim that CLIF ties or ranks second-best to NL. The same issue affects the grammar-prompting comparison in Table 8, where only one model is used and metric changes are inconsistent (e.g., CLIF F1 improves while precision drops; TFL+ precision drops from 0.5634 to 0.3618).
[Section 4.2, Table 5] The comparison between Flan-T5-small and the larger models is confounded by the fine-tuning setup. Flan-T5-small rows are fully fine-tuned without LoRA, whereas the Flan-T5-large rows are marked with an asterisk and use PEFT-LoRA. Consequently, the text's claim that the smallest model outperforms larger, fine-tuned models is not supported by the table: Flan-T5-large* achieves a higher accuracy than Flan-T5-small on NL (0.4729 vs 0.4384). Any conclusion about model-size effects must compare models trained with the same fine-tuning procedure.
[Section 4.2, Tables 9-10] Several rows across different models and grammars report exactly identical metrics (e.g., accuracy 0.3546, precision 0.1182, recall 0.3333, F1 0.1745 for many entries in Table 9, and for all entries in Table 10). This strongly suggests that these configurations collapse to a constant prediction, such as always predicting the majority class, and therefore those rows carry no information about the language being tested. The paper should report per-configuration prediction distributions and should not use such rows to draw conclusions about grammar performance, for example the statement in Section 4.2 that CLIF is a more compact contender in the 8-shot setting.

minor comments (6)

[Abstract] The sentence 'Our findings show that it is possible to substitute Natural Language (NL) with a more compact logical language while maintaining a strong performance on reasoning tasks and hope to use these results...' is a fragment; the subject of 'hope' is unclear and the sentence should be rewritten.
[Algorithm 1] Algorithm 1 uses the variable D in the loop over statements without defining it in the Require or Ensure block; please clarify the notation so the loop is unambiguous.
[Section 2.1, reference [29]] Reference [29] (a paper on jailbreaking via language games) does not appear to support the claim that the language in which a model receives a problem affects its success rate; a more directly relevant citation on input representation or prompt language effects would be appropriate.
[Section 4.2, Table 8] The text says that Table 8 shows CLIF outperforming all other languages in the zero-shot setting, but the table only reports Gemma-2-2b-it on CLIF, FOL, TFL+, and TFL; it does not include NL, CGIF, MINIFOL, or the other models, so the statement is stronger than the evidence.
[Table 12] The Tokenizer Re-Train column lists the CLIF condition with vocabulary size 32128 for both 'Yes' and 'No', which is confusing because re-training without resizing should preserve the size; clarify whether the re-trained CLIF tokenizer was resized and unify the notation.
[Section 4.2, Table 14] The prose describing Table 14 says the model reasons 'False' as opposed to 'Uncertain' on a single occasion, but the table shows several label differences across NL, CLIF, and TFL+; describe the error pattern more precisely so the reader can map the text to the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark whose central claim rests on held-out model evaluations, not on a self-citing derivation or a fitted parameter.

full rationale

The paper's central claim—that compact logical languages such as CLIF can substitute for natural language while maintaining strong reasoning performance—is an empirical finding supported by supervised fine-tuning and prompting experiments on held-out splits of FOLIO. The CLGC pipeline transforms existing FOL annotations into alternative grammars; this is data preprocessing, not a derivation that presupposes the conclusion. The grammars come from external standards or literature (CLIF, CGIF, TFL) or are introduced as experimental formats (MINIFOL, TFL+), and no fitted parameter is used to construct the reported accuracy differences. The paper contains no load-bearing self-citation: the cited prior work provides datasets, baselines, and related techniques, but the headline comparison is generated by the paper's own runs. The Table 3 transformation discrepancies (e.g., the MINIFOL rendering of a negated XOR as a conjunction) are a legitimate correctness and validity concern for the preprocessing pipeline, but they are not a circularity: even if the transformations are lossy, the model accuracies are measured on independently held-out examples rather than being forced by the transformation definitions. The reported CLIF-vs-NL gaps therefore are empirical findings that could be wrong or unverified, but they are not circular in structure. The paper is self-contained as an experimental study, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claim rests on the correctness of the FOLIO ground truth, the fidelity of the grammar implementations, and the semantic preservation of the parse-tree transformations. There are no physical or mathematical new entities; MINIFOL and TFL+ are data formats. The paper provides no proof of transformation equivalence, and Table 3 shows a concrete violation, so these assumptions are load-bearing and only partially verified.

free parameters (4)

LoRA configuration for PEFT fine-tuning = r=16, lora_alpha=32, target q,v, dropout 0.05
Chosen by hand for the PEFT runs; affects the Flan-T5-large results in Tables 5, 11, and 12.
SFT training epochs = 5 without LoRA, 10 with LoRA
Chosen per setting; different choices could change the accuracy gap between CLIF and NL.
Re-trained tokenizer vocabulary size = 191 for TFL, 180 for TFL+
Set empirically for the tokenizer re-training runs in Table 12; the headline result for Flan-T5-small TFL+ depends on this choice.
Number of few-shot examples = 8
All few-shot runs use 8 examples; the paper itself notes this may be suboptimal for some models.

assumptions (3)

domain assumption The FOLIO dataset ground truth labels are correct.
Accuracy and F1 are computed against FOLIO annotations; if labels are wrong, all numbers are wrong. Cited from [23], no verification in this paper.
domain assumption The CLIF, CGIF, and TFL grammar implementations in the CLGC pipeline faithfully follow their published definitions.
The pipeline relies on BNF grammars from W3C for Common Logic and from TFL literature; errors would change the input data. The paper gives no independent validation, and the TFL strings in Table 3 are opaque.
ad hoc to paper Parse-tree transformation between grammars preserves the truth conditions of each formula.
Algorithm 1 maps non-terminal symbols between grammars without a correctness proof; the authors manually revise grammars when parsing fails (Section 3.3). Table 3 shows a MINIFOL translation that is not logically equivalent, so this assumption is violated at least once.

invented entities (2)

MINIFOL
purpose: A word-based FOL notation intended to be easier for SLM tokenizers while remaining compact.
Introduced in this paper; no external specification or benchmark. The only shown example contains a logic error, and models trained or prompted on MINIFOL performed poorly (Tables 5, 7, 12).
TFL+
purpose: An extension of Tensor Function Logic with quantifier subscripts and parentheses, used as a mid-complexity grammar in the comparisons.
The paper does not state who created TFL+ or provide an external specification; it appears in this study as a variant of TFL. No independent validation is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering." pith.science (2026). https://pith.science/paper/LULR7HGA

@misc{pith2026250910249,
  author       = {Pith},
  title        = {Pith review of: Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LULR7HGA}},
  note         = {Machine review of arXiv:2509.10249}
}

read the original abstract

Recent advances in Language Models (LMs) have failed to mask their shortcomings particularly in the domain of reasoning. This limitation impacts several tasks, most notably those involving ontology engineering. As part of a PhD research, we investigate the consequences of incorporating formal methods on the performance of Small Language Models (SLMs) on reasoning tasks. Specifically, we aim to orient our work toward using SLMs to bootstrap ontology construction and set up a series of preliminary experiments to determine the impact of expressing logical problems with different grammars on the performance of SLMs on a predefined reasoning task. Our findings show that it is possible to substitute Natural Language (NL) with a more compact logical language while maintaining a strong performance on reasoning tasks and hope to use these results to further refine the role of SLMs in ontology engineering.

Figures

Figures reproduced from arXiv: 2509.10249 by the authors.

**Figure 2.** (Left) Input-to-tree example for Steps 1 and 2 of the CLGC pipeline. (Right) Tree-to-output example for Steps 5, 6 and 7 of the CLGC pipeline. The text in red represents the grammar of choice for the example. pair) in the target language. The output is passed to a formatting function to align the syntax with the target language (e.g. spacing). (7) The pipeline outputs a correctly-formatted premises-conclusions pair … view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 22 canonical work pages

[1]

Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, L. Yu, Y. Liu, J. Li, B. Xiong, D. Xiong, et al., Evaluating large language models: A comprehensive survey, Preprint arXiv:2310.19736 (2023)

arXiv 2023
[2]

Minaee, T

S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, J. Gao, Large language models: A survey. arxiv 2024, Preprint arXiv:2402.06196 (2024)

arXiv 2024
[3]

H. Liu, Z. Fu, M. Ding, R. Ning, C. Zhang, X. Liu, Y. Zhang, Logical reasoning in large language models: A survey, Preprint arXiv:2502.09100 (2025)

arXiv 2025
[4]

W. Wang, Y. Yang, F. Wu, Towards data-and knowledge-driven ai: a survey on neuro-symbolic computing, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

work page 2024
[5]

B. Jin, G. Liu, C. Han, M. Jiang, H. Ji, J. Han, Large language models on graphs: A comprehensive survey, IEEE Transactions on Knowledge and Data Engineering (2024)

work page 2024
[6]

A. S. Lippolis, M. J. Saeedizade, R. Keskisärkkä, S. Zuppiroli, M. Ceriani, A. Gangemi, E. Blomqvist, A. G. Nuzzolese, Ontology generation using large language models, in: European Semantic Web Conference, Springer, 2025, pp. 321–341

work page 2025
[7]

Y. Zhao, B. Zhang, X. Hu, S. Ouyang, J. Kim, N. Jain, J. de Berardinis, A. Meroño-Peñuela, E. Simperl, Improving ontology requirements engineering with ontochat and participatory prompting, in: Proceedings of the AAAI Symposium Series, volume 4, 2024, pp. 253–257

work page 2024
[8]

Y. Zhao, N. Vetter, K. Aryan, Using large language models for ontoclean-based ontology refinement, Preprint arXiv:2403.15864 (2024)

work page Pith review arXiv 2024

Show all 39 references

[9]

Zhao, Leveraging large language models for ontology requirements engineering, in: Extended Semantic Web Conference ESWC, 2025, pp

Y. Zhao, Leveraging large language models for ontology requirements engineering, in: Extended Semantic Web Conference ESWC, 2025, pp. –

2025
[10]

Fathallah, A

N. Fathallah, A. Das, S. D. Giorgis, A. Poltronieri, P. Haase, L. Kovriguina, Neon-gpt: a large language model-powered pipeline for ontology learning, in: European Semantic Web Conference, Springer, 2024, pp. 36–50

2024
[11]

Hou, Neural-symbolic reasoning: Towards the integration of logical reasoning with large language models, Authorea Preprints (2025)

Z. Hou, Neural-symbolic reasoning: Towards the integration of logical reasoning with large language models, Authorea Preprints (2025)

2025
[12]

Schlegel, K

V. Schlegel, K. V. Pavlov, I. Pratt-Hartmann, Can transformers reason in fragments of natural language?, Preprint arXiv:2211.05417 (2022)

2022 arXiv
[13]

Huang, K

J. Huang, K. C.-C. Chang, Towards reasoning in large language models: A survey, Preprint arXiv:2212.10403 (2022)

2022 arXiv
[14]

Srivastava, S

G. Srivastava, S. Cao, X. Wang, Towards reasoning ability of small language models, Preprint arXiv:2502.11569 (2025)

2025
[15]

J. Bi, Y. Wu, W. Xing, Z. Wei, Enhancing the reasoning capabilities of small language models via solution guidance fine-tuning, Preprint arXiv:2412.09906 (2024)

2024 arXiv
[16]

Y. Kim, E. Yi, M. Kim, S.-Y. Yun, T. Kim, Guiding reasoning in small language models with llm assistance, Preprint arXiv:2504.09923 (2025)

2025 arXiv
[17]

Besta, J

M. Besta, J. Barth, E. Schreiber, A. Kubicek, A. Catarino, R. Gerstenberger, P. Nyczyk, P. Iff, Y. Li, S. Houliston, et al., Reasoning language models: A blueprint, Preprint arXiv:2501.11223 (2025)

2025 arXiv
[18]

Kumar, T

K. Kumar, T. Ashraf, O. Thawakar, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, P. H. Torr, F. S. Khan, S. Khan, Llm post-training: A deep dive into reasoning large language models, Preprint arXiv:2502.21321 (2025)

2025 arXiv
[19]

H. R. Faghihi, Q. Guo, A. Uszok, A. Nafar, E. Raisi, P. Kordjamshidi, Domiknows: A library for integration of symbolic domain knowledge in deep learning, Preprint arXiv:2108.12370 (2021)

2021 arXiv
[20]

Kesseli, P

P. Kesseli, P. O’Hearn, R. S. Cabral, Logic. py: Bridging the gap between llms and constraint solvers, Preprint arXiv:2502.15776 (2025)

2025 arXiv
[21]

S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, Y. Tian, Training large language models to reason in a continuous latent space, 2024, URL https://arxiv. org/abs/2412.06769 (2022)

2022 arXiv
[22]

Cheng, H

F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, Z. Lin, Empowering llms with logical reasoning: A comprehensive survey, Preprint arXiv:2502.15652 (2025)

2025 arXiv
[23]

S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, W. Zhou, J. Coady, D. Peng, Y. Qiao, L. Benson, et al., Folio: Natural language reasoning with first-order logic, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 22017–22031

2024
[24]

Zhang, H

Y.-F. Zhang, H. Zhang, L. E. Li, E. Xing, Evaluating step-by-step reasoning through symbolic verification, Preprint arXiv:2212.08686 (2022)

2022 arXiv
[25]

J. Xu, H. Fei, L. Pan, Q. Liu, M.-L. Lee, W. Hsu, Faithful logical reasoning via symbolic chain-of- thought, Preprint arXiv:2405.18357 (2024)

2024 arXiv
[26]

S. Yang, E. Gribovskaya, N. Kassner, M. Geva, S. Riedel, Do large language models latently perform multi-hop reasoning?, Preprint arXiv:2402.16837 (2024)

2024 arXiv
[27]

S. Yang, N. Kassner, E. Gribovskaya, S. Riedel, M. Geva, Do large language models perform latent multi-hop reasoning without exploiting shortcuts?, Preprint arXiv:2411.16679 (2024)

2024 arXiv
[28]

Ghosh, S

B. Ghosh, S. Hasan, N. A. Arafat, A. Khan, Logical consistency of large language models in fact-checking, Preprint arXiv:2412.16100 (2024)

2024 arXiv
[29]

Y. Peng, Z. Long, F. Dong, C. Li, S. Wu, K. Chen, Playing language game with llms leads to jailbreaking, Preprint arXiv:2411.12762 (2024)

2024 arXiv
[30]

Y. Wu, M. Han, Y. Zhu, L. Li, X. Zhang, R. Lai, X. Li, Y. Ren, Z. Dou, Z. Cao, Hence, socrates is mortal: A benchmark for natural language syllogistic reasoning, in: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 2347–2367

2023
[31]

J. F. Sowa, Conceptual graphs, Foundations of artificial intelligence 3 (2008) 213–237

2008
[32]

J. F. Sowa, Conceptual graphs as a universal knowledge representation, Computers & Mathematics with Applications 23 (1992) 75–93

1992
[33]

Sowa, Introduction to common logic, 2011

J. Sowa, Introduction to common logic, 2011

2011
[34]

Sommers, G

F. Sommers, G. Englebretsen, An invitation to formal reasoning: the logic of terms, Routledge, 2017

2017
[35]

J. M. Castro-Manzano, L. I. Lozano-Cobos, P. O. Reyes-Cardenas, Programming with term logic, BRAIN. Broad Research in Artificial Intelligence and Neuroscience 9 (2018) 22–36

2018
[36]

J. M. C. Manzano, An intermediate term functor logic, Argumentos - Revista de Filosofia 11 (2025) 17–31

2025
[37]

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models., ICLR 1 (2022) 3

2022
[38]

B. Wang, Z. Wang, X. Wang, Y. Cao, R. A Saurous, Y. Kim, Grammar prompting for domain-specific language generation with large language models, Advances in Neural Information Processing Systems 36 (2023) 65030–65055

2023
[39]

Borgo, R

S. Borgo, R. Ferrario, A. Gangemi, N. Guarino, C. Masolo, D. Porello, E. M. Sanfilippo, L. Vieu, Dolce: A descriptive ontology for linguistic and cognitive engineering, Applied ontology 17 (2022) 45–69

2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, L. Yu, Y. Liu, J. Li, B. Xiong, D. Xiong, et al., Evaluating large language models: A comprehensive survey, Preprint arXiv:2310.19736 (2023)

arXiv 2023

[2] [2]

Minaee, T

S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, J. Gao, Large language models: A survey. arxiv 2024, Preprint arXiv:2402.06196 (2024)

arXiv 2024

[3] [3]

H. Liu, Z. Fu, M. Ding, R. Ning, C. Zhang, X. Liu, Y. Zhang, Logical reasoning in large language models: A survey, Preprint arXiv:2502.09100 (2025)

arXiv 2025

[4] [4]

W. Wang, Y. Yang, F. Wu, Towards data-and knowledge-driven ai: a survey on neuro-symbolic computing, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

work page 2024

[5] [5]

B. Jin, G. Liu, C. Han, M. Jiang, H. Ji, J. Han, Large language models on graphs: A comprehensive survey, IEEE Transactions on Knowledge and Data Engineering (2024)

work page 2024

[6] [6]

A. S. Lippolis, M. J. Saeedizade, R. Keskisärkkä, S. Zuppiroli, M. Ceriani, A. Gangemi, E. Blomqvist, A. G. Nuzzolese, Ontology generation using large language models, in: European Semantic Web Conference, Springer, 2025, pp. 321–341

work page 2025

[7] [7]

Y. Zhao, B. Zhang, X. Hu, S. Ouyang, J. Kim, N. Jain, J. de Berardinis, A. Meroño-Peñuela, E. Simperl, Improving ontology requirements engineering with ontochat and participatory prompting, in: Proceedings of the AAAI Symposium Series, volume 4, 2024, pp. 253–257

work page 2024

[8] [8]

Y. Zhao, N. Vetter, K. Aryan, Using large language models for ontoclean-based ontology refinement, Preprint arXiv:2403.15864 (2024)

work page Pith review arXiv 2024