Pith. sign in

REVIEW 3 major objections 1 minor 105 references

Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

T0 review · 3 major / 1 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Incorrect FOL formalizations affect 39% of FOLIO and 36% of MALLS entries, and corrections improve LLM accuracy by 9 to 22 points.

desk verdict The paper finds high error rates in two NL-to-FOL benchmarks, releases fixes that shift model scores, and offers a prioritization framework, but the human audit lacks basic reproducibility details. read the letter →

arxiv 2606.02837 v1 pith:2MSWBSYK submitted 2026-06-01 cs.CL cs.AI

Andrea Brunello , Cristian Curaba , Luca Geatti , Michele Mignani , Angelo Montanari , Nicola Saccomanno This is my paper · ORCID
classification cs.CLcs.AI
keywords NL-to-FOLtranslationdatasetauditingannotationerrorsLLM-assistedreviewFirst-OrderLogicNaturalLanguageInferencebenchmarkquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper performs a systematic human audit of the FOLIO validation split and a subset of MALLS test instances, revealing that roughly 39% and 36% of entries contain incorrect FOL ground-truth labels along with notable rates of ambiguous sentences and some incorrect NLI labels. It releases corrected annotations and shows that three state-of-the-art LLMs obtain accuracy gains of 9 to 22 percentage points when evaluated on the fixed labels instead of the originals. To scale future audits, the work introduces an LLM-assisted framework that directs human reviewers to the instances most likely to contain errors, enabling 90% dataset accuracy after inspecting fewer than 24% of entries rather than over 70% under random review.

What carries the argument

An LLM-based framework that scores instances for likely annotation errors and directs human reviewers to the highest-risk subset first.

What would settle it

An independent team re-inspecting a random sample of the released corrections and reporting disagreement rates above 10% on the FOL labels would indicate that the reported error rates and accuracy gains rest on unreliable fixes.

Watch

Extended reading notes

Core claim

Systematic human inspection shows that approximately 39% of FOLIO entries and 36% of sampled MALLS entries have incorrect FOL formalizations as ground truth, accompanied by 16.4% and 48% ambiguous natural-language sentences plus 8.4% incorrect NLI labels in FOLIO; the corrected ground truths raise accuracy for Gemma 4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini by 9 to 22 points, while an LLM-based framework prioritizes error-prone instances so that reviewers reach 90% dataset accuracy after examining under 24% of the data.

Load-bearing premise

The human inspection process correctly and consistently identifies incorrect FOL formalizations and ambiguities without systematic bias or new errors introduced during correction.

Editorial extensions

If this is right

  • All prior model comparisons and leaderboard rankings on FOLIO and MALLS must be recomputed with the corrected labels.
  • Neurosymbolic systems trained or evaluated on these datasets inherit the original label noise and require re-testing.
  • The targeted-review approach cuts the human labor required to produce high-accuracy NL-to-FOL data by more than two-thirds.
  • Any new NL-to-FOL benchmark should incorporate the same inspection step before release.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Un-audited NL-to-FOL or NLI datasets in other domains are likely to contain comparable fractions of label errors.
  • The prioritization logic could be transferred to improve efficiency in other annotation-heavy tasks such as semantic parsing or program synthesis.
  • Public release of the verified annotations creates a reusable reference that future work can treat as a cleaner baseline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The paper audits the validation split of FOLIO and a subset of MALLS for NL-to-FOL translation quality via systematic human inspection, reporting ~39% and ~36% incorrect FOL formalizations (ground truth labels), plus 16.4%/48% ambiguous NL sentences and 8.4% incorrect NLI labels in FOLIO. It releases corrected annotations, shows that re-evaluating three LLMs (Gemma-4 31B-it, Qwen3-30B-A3B, GPT-4o-mini) on the corrected labels yields +9 to +22 pp accuracy gains, and proposes an LLM-assisted framework that directs human review to error-prone instances, achieving 90% dataset accuracy after reviewing <24% of instances versus >70% for unguided review.

Significance. If the human-verified corrections hold, the work demonstrates that annotation errors in prominent NL-to-FOL benchmarks materially distort model evaluations and supplies both corrected data and a practical prioritization framework that reduces human effort. Releasing the verified annotations and framework code strengthens reproducibility and enables follow-on auditing in neurosymbolic AI.

major comments (3)
  1. [Abstract and §3] Abstract and §3 (Human Inspection): The central numerical claims (39%/36% incorrect FOL, 16.4%/48% ambiguous, 8.4% wrong NLI) rest entirely on the authors' human inspection, yet the manuscript supplies no information on inspection protocol, number of annotators, inter-annotator agreement statistics, adjudication procedure for disagreements, or selection criteria for the MALLS subset. This directly undermines the load-bearing error-rate statistics and the downstream accuracy-gain results.
  2. [§4] §4 (Model Evaluation): The reported +9 to +22 pp accuracy gains are computed by comparing LLM performance on the original versus the authors' corrected labels. Without an independent validation of the corrections (e.g., blind re-annotation or external expert review), it is impossible to distinguish genuine error fixes from systematic shifts introduced by the inspection process itself.
  3. [§5] §5 (LLM-assisted Framework): The claim that the framework reaches 90% accuracy after reviewing <24% of instances depends on the same unvalidated human judgments used to define the 'error-prone' instances; any bias in the initial inspection propagates into the prioritization model and the reported efficiency gains.
minor comments (1)
  1. [Abstract] The abstract states results from human inspection but does not reference any supplementary material or appendix that might contain the missing protocol details; if such material exists, it should be explicitly cited in the main text.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The concerns about transparency in the human inspection process and validation of corrections are well-taken. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract and §3] Abstract and §3 (Human Inspection): The central numerical claims (39%/36% incorrect FOL, 16.4%/48% ambiguous, 8.4% wrong NLI) rest entirely on the authors' human inspection, yet the manuscript supplies no information on inspection protocol, number of annotators, inter-annotator agreement statistics, adjudication procedure for disagreements, or selection criteria for the MALLS subset. This directly undermines the load-bearing error-rate statistics and the downstream accuracy-gain results.

    Authors: We agree that the current description of the inspection process is insufficient. In the revised manuscript we will expand §3 with a dedicated subsection that specifies the annotation protocol, number of annotators and their qualifications, the guidelines provided to them, inter-annotator agreement statistics, the procedure used to resolve disagreements, and the exact selection criteria applied to the MALLS subset. These additions will make the reported error rates fully reproducible. revision: yes

  2. Referee: [§4] §4 (Model Evaluation): The reported +9 to +22 pp accuracy gains are computed by comparing LLM performance on the original versus the authors' corrected labels. Without an independent validation of the corrections (e.g., blind re-annotation or external expert review), it is impossible to distinguish genuine error fixes from systematic shifts introduced by the inspection process itself.

    Authors: We acknowledge that the manuscript does not include an independent blind re-annotation by external experts. The corrections were produced through systematic logical comparison of each FOL formula against its NL premise by the authors. In revision we will add an explicit limitations paragraph in §4 that discusses the possibility of systematic bias, reports any internal consistency checks performed, and stresses that the full set of corrected annotations is released publicly so that the community can perform independent verification. We maintain that the observed accuracy gains are driven by the removal of clear logical mismatches, but we will present this as an acknowledged limitation rather than a fully externally validated result. revision: partial

  3. Referee: [§5] §5 (LLM-assisted Framework): The claim that the framework reaches 90% accuracy after reviewing <24% of instances depends on the same unvalidated human judgments used to define the 'error-prone' instances; any bias in the initial inspection propagates into the prioritization model and the reported efficiency gains.

    Authors: We agree that the framework evaluation inherits the same human judgments used to label errors. In the revision we will clarify in §5 how the prioritization model was trained (on features derived from the inspected data), provide additional ablation results that isolate the contribution of the LLM component, and add a discussion of how inspection bias could affect the reported efficiency numbers. We will also release the framework code and the full set of model predictions so that others can re-evaluate the prioritization under alternative label sets. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical audit reports direct observations with no self-referential reductions

full rationale

The paper reports error rates (39%/36% incorrect FOL, etc.) and LLM accuracy gains (+9 to +22 pp) obtained via systematic human inspection of existing dataset instances followed by direct re-evaluation of models on the resulting corrected labels. These quantities are produced by external annotation and testing steps rather than any equation, fitted parameter, or self-citation chain that reduces the outputs to the inputs by construction. No self-definitional, fitted-input-called-prediction, or ansatz-smuggling patterns appear in the abstract or described contributions. The LLM-assisted framework is a separate proposal and does not alter the reported statistics.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claims rest on the assumption that human judgment constitutes reliable ground truth for FOL correctness; no free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption Human annotators can reliably determine whether a given FOL formula is a correct formalization of a natural language sentence
    All reported error rates and accuracy gains derive from this human judgment step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling." pith.science (2026). https://pith.science/paper/2MSWBSYK

@misc{pith2026260602837,
  author       = {Pith},
  title        = {Pith review of: Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MSWBSYK}},
  note         = {Machine review of arXiv:2606.02837}
}
read the original abstract

Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited. Our first contribution is to present a systematic human inspection of the validation split of \textsf{FOLIO} and a subset of \textsf{MALLS} test instances, finding that approximately 39% and 36% of entries, respectively, contain incorrect FOL formalizations (i.e., ground truth labels), with additional rates of ambiguous NL sentences (16.4% and 48%) and incorrect NLI labels in \textsf{FOLIO} (8.4%). Our second contribution is to develop and release corrected ground truths for such datasets, showing that annotation errors distort model evaluation on a reference benchmark task: testing three state-of-the-art LLMs (Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini) with the corrected ground truths yields accuracy gains from +9 to +22 percentage points. Motivated by these findings, we propose an LLM-based framework to support humans in manual reviewing NL-to-FOL datasets. By directing reviewers toward the most error-prone instances, we empirically show that it is possible to achieve 90% dataset accuracy after reviewing fewer than 24% of instances, compared to over 70% required by unguided review. We release all human-verified annotations and the code for our framework.

Figures

Figures reproduced from arXiv: 2606.02837 by the authors.

Figure 1
Figure 1. The two pipelines. Each starts from the Initial Dataset containing triplets (p, Ω, φ), and produces an output with the Formalization Proposal (p, Ω, ψˆ) and the verdict v. Pipeline 1 judges the original formula φ directly. Pipeline 2 first re-generates a candidate φˆ from p and Ω alone, then judges it. • Pipeline 2: Re-generation and V&R. The original formula φ is discarded. The LLM first translates p under Ω into a… view at source ↗
Figure 2
Figure 2. Pipelines comparison across models (horizontally) and datasets (vertically) according to the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Accuracy-human effort curve. Each plot shows Gemma’s performance on FOLIO_validation (left) and MALLS_test (right). The blue band represents Pipeline 1 (min/max and average across the six prompting combinations {B1, B2, B3}×{pv1, pv3}); the red curve shows the best-AUC configuration of Pipeline 2; the black and green line represents respectively the Black and Green Baseline. above 97%. This opens the possibility of … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Accuracy conditioned on the verdict assigned by the judge, for Pipeline 1 (blue) and Pipeline 2 [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Pipeline comparison across models and datasets under the [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Pipeline comparison across models and datasets under the [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Pipeline comparison across models and datasets under the AAG metric. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Model comparison across all three datasets and all four metrics (AUC, AAG, [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Prompting strategies and variants comparison for Pipeline 1. Blue-framed cells mark the [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Prompting strategies and variants comparison for Pipeline 2 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Extension of Figure [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

105 extracted references · 69 canonical work pages

  1. [1]

    13th International Conference on Intelligent Computer Mathematics (CICM) , series =

    Christian Szegedy , title =. 13th International Conference on Intelligent Computer Mathematics (CICM) , series =. 2020 , doi =

    2020

  2. [2]

    2024 , url =

    Long Hei Matthew Lam and others , title =. 2024 , url =

    2024

  3. [4]

    Cox and Robert Dale , title =

    Dave Barker-Plummer and Richard J. Cox and Robert Dale , title =. 2011 , isbn =

    2011

  4. [5]

    CoRR , volume =

    Dalrymple, David "davidad" and Skalse, Joar and Bengio, Yoshua and Russel, Stuart and Tegmark, Max and Seshia, Sanjit and Omohundro, Steve and Szegedy, Christian and Goldhaber, Ben and Ammann, Nora and Abate, Alessandro and Halpern, Joe and Barrett, Clark and Zhao, Ding and Zhi-Xuan, Tan and Wing, Jeannette and Tenenbaum, Joshua , title =. CoRR , volume =...

    2024

  5. [6]

    and Dale, Robert , booktitle=

    Barker-Plummer, Dave and Cox, Richard J. and Dale, Robert , booktitle=. Student translations of natural language into logic:

  6. [7]

    Cox and Robert Dale , year=

    Dave Barker-Plummer and Richard J. Cox and Robert Dale , year=. Tarski’s

  7. [8]

    Yanwei Wang, Nadia Figueroa, Shen Li, Ankit Shah, and Julie Shah

    Wang, Jun and Sundarsingh, David Smith and V. Deshmukh, Jyotirmoy and Kantaros, Yiannis , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.21022 , eprinttype =. 2504.21022 , timestamp =

    work page doi:10.48550/arxiv.2504.21022 2025

  8. [9]

    2021 , url =

    Apurwa Yadav and Aarshil Patel and Manan Shah , title =. 2021 , url =. doi:10.1016/J.AIOPEN.2021.05.001 , timestamp =

    work page doi:10.1016/j.aiopen.2021.05.001 2021

Show all 105 references
  1. [10]

    CoRR , volume =

    Lei Xu and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.06774 , eprinttype =. 2510.06774 , timestamp =

    2025 doi

  2. [11]

    Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs

    Vossel, Felix and Mossakowski, Till and Gehrke, Björn , biburl =. Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs. , url =. CoRR , keywords =

  3. [12]

    Soviet physics

    Binary codes capable of correcting deletions, insertions, and reversals , author=. Soviet physics. Doklady , year=

  4. [13]

    arXiv preprint arXiv:2405.02318 , year=

    Autoformalizing Natural Language to First-Order Logic: A Case Study in Logical Fallacy Detection , author=. arXiv preprint arXiv:2405.02318 , year=

  5. [14]

    QA - N at V er: Question Answering for Natural Logic-based Fact Verification

    Aly, Rami and others. QA - N at V er: Question Answering for Natural Logic-based Fact Verification. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.521

    2023 doi

  6. [15]

    Logical Fallacy Detection

    Jin, Zhijing and others. Logical Fallacy Detection. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. doi:10.18653/v1/2022.findings-emnlp.532

    2022 doi

  7. [16]

    CoRR , volume =

    Yujun Zhou and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.04810 , eprinttype =. 2506.04810 , timestamp =

    2025 doi

  8. [17]

    2024 , url =

    Andrea Brunello and others , title =. 2024 , url =

    2024

  9. [18]

    Lee, Hyemin S

    Ryu, Hyun and Kim, Gyeongman and S. Lee, Hyemin S. and Yang, Eunho , title =. 2025 , url =

    2025

  10. [19]

    Complexity Parameters for First-Order Classes , booktitle =

    Marta Arias and Roni Khardon , editor =. Complexity Parameters for First-Order Classes , booktitle =. 2003 , url =. doi:10.1007/978-3-540-39917-9\_4 , timestamp =

    2003 doi

  11. [20]

    Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence,

    Fengxiang Cheng and others , title =. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence,. 2025 , url =. doi:10.24963/IJCAI.2025/1155 , timestamp =

    2025 doi

  12. [21]

    2025 , url =

    Lovish Madaan and others , title =. 2025 , url =. doi:10.18653/V1/2025.NAACL-LONG.466 , timestamp =

    2025 doi

  13. [22]

    ICLR 2024 Workshop on Secure and Trustworthy Large Language Models , year=

    Enhancing and Evaluating Logical Reasoning Abilities of Large Language Models , author=. ICLR 2024 Workshop on Secure and Trustworthy Large Language Models , year=

    2024

  14. [23]

    NeurIPS 2022, November 28 - December 9, 2022 , year =

    Yuhuai Wu and others , title =. NeurIPS 2022, November 28 - December 9, 2022 , year =

    2022

  15. [24]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),

    Jundong Xu and others , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2024 , url =. doi:10.18653/V1/2024.ACL-LONG.720 , timestamp =

    2024 doi

  16. [25]

    CoRR , volume =

    Benjamin Callewaert and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2501.14540 , eprinttype =. 2501.14540 , timestamp =

    2025 doi

  17. [26]

    Few-Shot Natural Language to First-Order Logic Translation via Code Generation , booktitle =

    Junnan Liu , editor =. Few-Shot Natural Language to First-Order Logic Translation via Code Generation , booktitle =. 2025 , url =. doi:10.18653/V1/2025.NAACL-LONG.547 , timestamp =

    2025 doi

  18. [28]

    2024 , url =

    Xin Quan and others , title =. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.172 , timestamp =

    2024 doi

  19. [29]

    1990 , url=

    Events in the Semantics of English: A Study in Subatomic Semantics , author=. 1990 , url=

    1990

  20. [30]

    CoRR , volume =

    Christopher Hahn and others , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2206.01962 , eprinttype =. 2206.01962 , timestamp =

    2022 doi

  21. [31]

    Parsing the WSJ Using CCG and Log-Linear Models

    Clark, Stephen and Curran, James R. Parsing the WSJ Using CCG and Log-Linear Models. ACL -04. 2004. doi:10.3115/1218955.1218969

    2004 doi

  22. [32]

    2015 , url =

    Johan Bos , title =. 2015 , url =

    2015

  23. [33]

    Yu Pei and others , title =. Trans. Assoc. Comput. Linguistics , volume =. 2025 , url =. doi:10.1162/TACL.A.41 , timestamp =

    2025 doi

  24. [34]

    GCAT 2023 , year=

    Data and Knowledge Engineering for Legal Precedents Using First-Order Predicate Logic , author=. GCAT 2023 , year=

    2023

  25. [35]

    Towards Advanced Mathematical Reasoning for LLM s via First-Order Logic Theorem Proving

    Cao, Chuxue and others. Towards Advanced Mathematical Reasoning for LLM s via First-Order Logic Theorem Proving. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.628

    2025 doi

  26. [36]

    Grammar-Constrained Decoding Makes Large Language Models Better Logical Parsers

    Raspanti, Federico and others. Grammar-Constrained Decoding Makes Large Language Models Better Logical Parsers. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). 2025. doi:10.18653/v1/2025.acl-industry.34

    2025 doi

  27. [37]

    Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance

    Tam, Zhi Rui and others. Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2024. doi:10.18653/v1/2024.emnlp-industry.91

    2024 doi

  28. [39]

    SEMANTiCS 2025, Vienna, Austria, September 3-5, 2025 , series =

    Alexander Beiser and others , title =. SEMANTiCS 2025, Vienna, Austria, September 3-5, 2025 , series =. 2025 , url =

    2025

  29. [40]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),

    Mihir Parmar and others , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2024 , url =. doi:10.18653/V1/2024.ACL-LONG.739 , timestamp =

    2024 doi

  30. [41]

    Into The Limits of Logic: Alignment Methods for Formal Logical Reasoning

    Lopez-Ponce, FernandoFrancisco and Bel-Enguix, Gemma. Into The Limits of Logic: Alignment Methods for Formal Logical Reasoning. MathNLP 2025. 2025. doi:10.18653/v1/2025.mathnlp-main.8

    2025 doi

  31. [42]

    Diagnosing the First-Order Logical Reasoning Ability Through L ogic NLI

    Tian, Jidong and others. Diagnosing the First-Order Logical Reasoning Ability Through L ogic NLI. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.303

    2021 doi

  32. [43]

    CoRR , volume =

    Thatikonda, Ramya Keerthy and Han, Jiuzhou and Buntine, Wray and Shareghi, Ehsan , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2409.16461 , eprinttype =. 2409.16461 , timestamp =

    2024 doi

  33. [44]

    2025 , url =

    Chengwen Qi and others , title =. 2025 , url =

    2025

  34. [45]

    NeurIPS 2023, December 10 - 16, 2023 , year =

    Ye, Xi and Chen, Qiaochu and Dillig, Isil and Durrett, Greg , title =. NeurIPS 2023, December 10 - 16, 2023 , year =

    2023

  35. [46]

    Generating Predicate Logic Expressions from Natural Language , year=

    Levkovskyi, Oleksii and Li, Wei , booktitle=. Generating Predicate Logic Expressions from Natural Language , year=

  36. [47]

    Educational Data Mining , year=

    Dimensions of Difficulty in Translating Natural Language into First-Order Logic , author=. Educational Data Mining , year=

  37. [48]

    CoRR , volume =

    Singh, Hrituraj and Aggarwal, Milan and Krishnamurthy, Balaji , title =. CoRR , volume =. 2020 , url =. 2002.06544 , timestamp =

    2020

  38. [49]

    Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning

    Lu, Xuantao and others. Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning. Proceedings of the 29th International Conference on Computational Linguistics. 2022

    2022

  39. [50]

    Findings of the Association for Computational Linguistics:

    Akshay Chaturvedi and Nicholas Asher , title =. Findings of the Association for Computational Linguistics:. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-EMNLP.390 , timestamp =

    2024 doi

  40. [51]

    Faithful Chain-of-Thought Reasoning

    Lyu, Qing and others. Faithful Chain-of-Thought Reasoning. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. d...

    2023 doi

  41. [52]

    CoRR , volume =

    Qingchuan Li and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2410.21779 , eprinttype =. 2410.21779 , timestamp =

    2024 doi

  42. [53]

    2023 , url =

    Olausson, Theo and Gu, Alex and Lipkin, Ben and Zhang, Cedegao and Solar-Lezama, Armando and Tenenbaum, Joshua and Levy, Roger , title =. 2023 , url =. doi:10.18653/V1/2023.EMNLP-MAIN.313 , timestamp =

    2023 doi

  43. [54]

    CoRR , volume =

    Peizhang Shao and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2507.07748 , eprinttype =. 2507.07748 , timestamp =

    2025 doi

  44. [55]

    2025 , url =

    Bowen Jiang and others , title =. 2025 , url =. doi:10.18653/V1/2025.NAACL-LONG.186 , timestamp =

    2025 doi

  45. [56]

    Frontiers Comput

    Laura Orynbay and others , title =. Frontiers Comput. Sci. , volume =. 2025 , url =. doi:10.3389/FCOMP.2024.1486581 , timestamp =

    2025 doi

  46. [57]

    Findings of the Association for Computational Linguistics:

    Pan, Liangming and Albalak, Alon and Wang, Xinyi and Yang Wang, William , title =. Findings of the Association for Computational Linguistics:. 2023 , url =. doi:10.18653/V1/2023.FINDINGS-EMNLP.248 , timestamp =

    2023 doi

  47. [58]

    CoRR , volume =

    Shashank Kirtania and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2407.02514 , eprinttype =. 2407.02514 , timestamp =

    2024 doi

  48. [59]

    Logic-Thinker: Teaching Large Language Models to Think more Logically

    Wen, Chengyao and others. Logic-Thinker: Teaching Large Language Models to Think more Logically. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.696

    2025 doi

  49. [60]

    CoRR , volume =

    Koushik Viswanadha and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.18383 , eprinttype =. 2506.18383 , timestamp =

    2025 doi

  50. [61]

    2024 , url =

    Fangzhi Xu and others , title =. 2024 , url =. doi:10.18653/V1/2024.ACL-LONG.707 , timestamp =

    2024 doi

  51. [62]

    2025 , url =

    Ruikang Hu and others , title =. 2025 , url =

    2025

  52. [63]

    CoRR , volume =

    Hannah Bansal and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.17377 , eprinttype =. 2509.17377 , timestamp =

    2025 doi

  53. [64]

    2024 , url =

    Zhaofeng Wu and others , title =. 2024 , url =. doi:10.18653/V1/2024.NAACL-LONG.102 , timestamp =

    2024 doi

  54. [65]

    Findings of the Association for Computational Linguistics:

    Oyvind Tafjord and others , title =. Findings of the Association for Computational Linguistics:. 2021 , url =. doi:10.18653/V1/2021.FINDINGS-ACL.317 , timestamp =

    2021 doi

  55. [66]

    Transformers as Soft Reasoners over Language , booktitle =

    Peter Clark and others , editor =. Transformers as Soft Reasoners over Language , booktitle =. 2020 , url =. doi:10.24963/IJCAI.2020/537 , timestamp =

    2020 doi

  56. [67]

    CoRR , volume =

    Debargha Ganguly and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2409.17270 , eprinttype =. 2409.17270 , timestamp =

    2024 doi

  57. [68]

    Findings of the Association for Computational Linguistics:

    Simeng Han and others , title =. Findings of the Association for Computational Linguistics:. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-EMNLP.966 , timestamp =

    2024 doi

  58. [69]

    CoRR , volume =

    Qianxi He and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.19907 , eprinttype =. 2502.19907 , timestamp =

    2025 doi

  59. [70]

    CoRR , volume =

    Shokhrukh Ibragimov and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.14180 , eprinttype =. 2502.14180 , timestamp =

    2025 doi

  60. [71]

    CoRR , volume =

    Navapat Nananukul and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.01530 , eprinttype =. 2510.01530 , timestamp =

    2025 doi

  61. [72]

    2025 , eprint=

    From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translation , author=. 2025 , eprint=

    2025

  62. [73]

    CoRR , volume =

    Yue Zhang and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.21281 , eprinttype =. 2505.21281 , timestamp =

    2025 doi

  63. [74]

    Ontology learning towards expressiveness: A survey , journal =

    Pauline Armary and others , keywords =. Ontology learning towards expressiveness: A survey , journal =. 2025 , issn =. doi:https://doi.org/10.1016/j.cosrev.2024.100693 , url =

    2025 doi

  64. [75]

    CoRR , volume =

    Rick Du and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2404.14991 , eprinttype =. 2404.14991 , timestamp =

    2024 doi

  65. [76]

    Zhengkun Di and others , title =. Knowl. Based Syst. , volume =. 2025 , url =. doi:10.1016/J.KNOSYS.2025.114140 , timestamp =

    2025 doi

  66. [77]

    arXiv preprint arXiv:2509.24765 , year=

    From Ambiguity to Verdict: A Semiotic-Grounded Multi-Perspective Agent for LLM Logical Reasoning , author=. arXiv preprint arXiv:2509.24765 , year=

    arXiv

  67. [78]

    2024 , eprint=

    uto val: Autonomous Assessment of LLMs in Formal Synthesis and Interpretation Tasks , author=. 2024 , eprint=

    2024

  68. [79]

    Learning First-Order Logic Rules for Argumentation Mining

    Sun, Yang and others. Learning First-Order Logic Rules for Argumentation Mining. ACL 2025. 2025. doi:10.18653/v1/2025.acl-long.691

    2025 doi

  69. [80]

    Transformer models for translating natural language sentences into formal logical expressions , school=

    Deveci, İbrahim Ethem , year=. Transformer models for translating natural language sentences into formal logical expressions , school=

  70. [81]

    2025 , url =

    Samuele Germiniani and others , title =. 2025 , url =. doi:10.1109/ACCESS.2025.3551607 , timestamp =

    2025 doi

  71. [82]

    CoRR , volume =

    Ali Mohammadjafari and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2410.01066 , eprinttype =. 2410.01066 , timestamp =

    2024 doi

  72. [83]

    CoRR , volume =

    Ke Weng and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.23486 , eprinttype =. 2505.23486 , timestamp =

    2025 doi

  73. [84]

    CoRR , volume =

    Agnieszka Mensfelt and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.09810 , eprinttype =. 2509.09810 , timestamp =

    2025 doi

  74. [85]

    Symmetry , volume =

    Peng Jiang and Xiaodong Cai , title =. Symmetry , volume =. 2024 , url =. doi:10.3390/SYM16091201 , timestamp =

    2024 doi

  75. [86]

    doi:10.20944/preprints202512.1848.v1 , url =

    Francis Frydman , title =. doi:10.20944/preprints202512.1848.v1 , url =

    doi

  76. [87]

    Computational linguistics , volume=

    A survey and classification of controlled natural languages , author=. Computational linguistics , volume=. 2014 , publisher=

    2014

  77. [88]

    arXiv preprint arXiv:1812.00978 , year=

    A survey on semantic parsing , author=. arXiv preprint arXiv:1812.00978 , year=

    arXiv

  78. [89]

    CoRR , volume =

    Zishen Wan and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2401.01040 , eprinttype =. 2401.01040 , timestamp =

    2024 doi

  79. [90]

    First-Order Logic , year =

    Raymond Merrill Smullyan , editor =. First-Order Logic , year =

  80. [91]

    ACL 2002 , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. ACL 2002 , pages=

    2002

  81. [92]

    arXiv preprint arXiv:2312.06942 , year=

    AI control: Improving safety despite intentional subversion , author=. arXiv preprint arXiv:2312.06942 , year=

  82. [93]

    2201.11903 , archiveprefix =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. 2201.11903 , archiveprefix =

    arXiv

  83. [94]

    2203.11171 , archiveprefix =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. 2203.11171 , archiveprefix =

    arXiv

  84. [95]

    2005.14165 , archiveprefix =

    Language Models are Few-Shot Learners , author =. 2005.14165 , archiveprefix =

    2005 arXiv

  85. [96]

    2024 , eprint=

    Meta Prompting for AI Systems , author=. 2024 , eprint=

    2024

  86. [97]

    2022 , eprint=

    Self-critiquing models for assisting human evaluators , author=. 2022 , eprint=

    2022

  87. [98]

    Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation , booktitle =

    Yuan Yang and Siheng Xiong and Ali Payani and Ehsan Shareghi and Faramarz Fekri , editor =. Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation , booktitle =. 2024 , url =. doi:10.18653/V1/2024.ACL-LONG.375 , timestamp =

    2024 doi

  88. [99]

    2024 , url =

    Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and Zhou, Wenfei and Coady, James and Peng, David and Qiao, Yujie and Benson, Luke and Sun, Lucy and Wardle. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.1229 , timestamp =

    2024 doi

  89. [100]

    International conference on Tools and Algorithms for the Construction and Analysis of Systems , pages=

    Z3: An efficient SMT solver , author=. International conference on Tools and Algorithms for the Construction and Analysis of Systems , pages=. 2008 , organization=

    2008

  90. [101]

    26th International symposium on temporal representation and reasoning (TIME 2019) , pages=

    Synthesis of LTL formulas from natural language texts: State of the art and research directions , author=. 26th International symposium on temporal representation and reasoning (TIME 2019) , pages=. 2019 , organization=

    2019

  91. [102]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Do LLMs Really Struggle at NL-FOL Translation? Revealing Their Strengths via a Novel Benchmarking Strategy , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  92. [103]

    2024 , note =

    GPT-4o-mini , howpublished =. 2024 , note =

    2024

  93. [104]

    Towards a Common Framework for Autoformalization , booktitle =

    Agnieszka Mensfelt and David Tena Cucala and Santiago Franco and Angeliki Koutsoukou. Towards a Common Framework for Autoformalization , booktitle =. 2026 , url =. doi:10.1609/AAAI.V40I48.42132 , timestamp =

    2026 doi

  94. [105]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , booktitle =

    Lianmin Zheng and Wei. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , booktitle =. 2023 , url =

    2023

  95. [106]

    CoRR , volume =

    An Yang and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.09388 , eprinttype =. 2505.09388 , biburl =

    2025 doi arXiv

  96. [107]

    2024 , note =

    Gemma 4 , howpublished =. 2024 , note =

    2024

  97. [108]

    TechRxiv , volume =

    Jindong Li and Yali Fu and Fengxiang Cheng and Yang Yang and Jiahong Liu and Hongce Zhang and Yutao Yue and Menglin Yang and Haoxuan Li and Liangming Pan and Zhouchen Lin , title =. TechRxiv , volume =. 2025 , doi =. https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.176538331...

    2025 doi

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.