Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoning

Alexander Beiser; David Penz; Nysret Musliu

REVIEW 4 major objections 5 minor 31 references

Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Formal language choice drives neurosymbolic LLM reasoning: first-order logic leads, logic programming lags.

desk verdict Useful empirical result on intermediate-language choice for neurosymbolic reasoning, but the causal claim is under-supported by the experiment as reported. read the letter →

arxiv 2509.04083 v1 pith:5BTGFUQW submitted 2025-09-04 cs.AI

Alexander Beiser , David Penz , Nysret Musliu This is my paper

classification cs.AI

keywords neurosymbolicreasoninglargelanguagemodelsintermediatechallengefirst-orderlogicanswersetprogrammingin-contextlearninglogicalautoformalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neurosymbolic reasoning lets an LLM translate a natural-language problem into a formal language and hands the translation to a symbolic solver. This paper's aim is to show that the choice of that formal language—the intermediate language—is a major performance factor, not a neutral implementation detail. It introduces the intermediate language challenge and compares four target languages (Pyke, ASP, NLTK, first-order logic) over three logical-reasoning datasets and seven LLMs. The reported result is that first-order logic performs best on average, followed by NLTK, then ASP, with Pyke last; the gap between best and worst is roughly twenty percentage points. This matters because it gives practitioners a previously unexamined lever that can change a neurosymbolic system's accuracy substantially, especially at the small-model scale.

What carries the argument

The load-bearing object is the intermediate language—the formal representation an LLM must produce before a symbolic solver evaluates it. The paper's controlled comparison keeps the prompting architecture and solver family fixed while varying only the formal target language (Pyke, ASP, NLTK, FOL), so any systematic difference is attributed to the language. Two root causes motivate the challenge: the same logical problem can be encoded in multiple formal languages, and the particular syntax of those encodings is known to change LLM performance even when semantics are identical. The central identity is Overall-Acc = Exec-Rate × Exec-Acc, which separates the two ways a language can fail—the LLM

What would settle it

Take one dataset, write several independent ICL examples for each of the four languages, and re-run the full LLM set. If FOL does not consistently outrank Pyke across examples, or if matching all languages to a common execution-rate eliminates FOL's advantage, the claim that the formal language itself drives reasoning performance is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the formal target language changes both the syntax and the semantics of what an LLM can be made to produce: switching languages shifts how often the model emits a parseable formula and how often that formula is actually the right answer. The authors formalize the choice as the intermediate language challenge—given a natural-language problem P and a set of candidate formal languages L, pick the l that yields high reasoning accuracy. Across 644 experiments covering Pyke, ASP, NLTK, and FOL on ProntoQA, ProofWriter, and FOLIO with seven LLMs, the average ordering is FOL first, NLTK second, ASP third, Pyke last. They decompose overall accuracy into execution-rat

Load-bearing premise

The comparison assumes that a single hand-written in-context example and one dedicated parser per language are fair representatives of that formal language; if example quality or parser strictness varies with the language, the ranking might reflect those differences rather than the language itself.

Editorial extensions

If this is right

Average overall accuracy on ProntoQA and ProofWriter is about 65% with FOL and 46% with Pyke, so the choice of target language alone can account for a roughly twenty-point swing.
The two components of overall accuracy separate parseability from semantic correctness, so reporting only overall accuracy hides which step of the pipeline is failing.
Because FOL and NLTK win on execution-accuracy and ASP on execution-rate, the ranking is not driven by a single mechanism.
Small models (around 8B parameters) show the largest variation across languages, meaning language choice is a high-impact variable precisely in the resource-constrained setting.
On individual LLMs the best language is not always FOL, so model-specific language selection matters for top performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The single hand-crafted ICL example per dataset and per language is a confounder; sampling many independently written examples would test whether FOL's lead survives the change.
The execution-rate/execution-accuracy split suggests two separable failure modes—syntax repair for low parseability and semantic verification for low correctness—so one need not wait for a single language that is best on both axes.
If the authors' training-data-exposure hypothesis is right, the methodology could be inverted to design custom intermediate languages tuned to a specific model rather than adopting an existing formal language.
Because ASP and Pyke were excluded from FOLIO, the empirical ranking is bounded to reasoning problems expressible in standard logic programming; tasks involving classical negation may produce a different ordering.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper up front: it runs a big benchmark — four formal languages, three datasets, seven LLMs — and reports that FOL beats NLTK, ASP, and Pyke on overall accuracy for neurosymbolic reasoning. That result is plausible and the paper is worth engaging with, but the headline claim is softer than the abstract suggests. The largest per-LLM differences are often within one SEM, and several LLMs favor NLTK or ASP over FOL. So the robust message is: language choice matters and its effect is model-dependent, with FOL somewhat ahead on average. The paper overclaims by treating a hand-picked aggregate as a decisive ordering.

The genuinely new thing here is the comparison itself. Prior work like Logic-LM and LINC picks an intermediate language without justifying it, and nobody has systematically varied that axis across many models. The authors name the problem — the intermediate language challenge — which is a useful framing for practitioners. They also transparently separate execution-rate from execution-accuracy, which helps show whether the bottleneck is translation or solving. That decomposition is the best part of the paper.

The soft spots are real but not fatal. The stress-test note about ICL-example quality and parser strictness is on point: one hand-crafted example per dataset/language, and a dedicated parser per language, can masquerade as language-driven differences. The paper even lists the parser-strictness phenomena (missing line breaks in Pyke, unquoted 'not' in ASP, parentheses/arity issues in NLTK/FOL). Because prompts, examples, and parsers are not released, you cannot separate language effects from artifact effects. Also, the aggregate excludes FOLIO for ASP/Pyke with no sensitivity check, and there are no significance tests anywhere.

I read the longer preprint [9] as already containing the core comparison, so the short paper is mostly a summary plus a per-LLM breakdown. That does not bother me much — a workshop paper can be a condensed version — but it puts a limit on novelty.

Who is this for? Someone building a neurosymbolic pipeline who wants a quick rule of thumb (start with FOL, expect variation) and a reason to treat the formal language as a design decision. It is not a rigorous causal study and should not be cited as one. I would send it to review if I edited a venue that wants this kind of empirical work; the experiments are big and the subject is underserved. But I would ask for artifacts and a more careful statement of what the data can support, and I would probably require the authors to temper the abstract. My own verdict: engage it, but treat the ranking as evidence of a phenomenon, not proof of FOL superiority.

For the record, I disagree with the reader on one point: the claim is not definitively undermined by the per-LLM reversals, since aggregating across multiple LLMs is a legitimate way to make a general statement. The larger threat is the unverified fairness of the ICL examples and parsers, which is an internal-validity concern that the authors can partly answer by releasing their materials.

I would bring this to a reading group focused on neurosymbolic reasoning, and I would cite it as an empirical datapoint that intermediate language matters, with a caveat about the confounds. It deserves a serious referee, not a desk reject, but the referee should be told to push on artifacts.

Referee Report

4 major / 5 minor

Summary. The paper studies whether the choice of formal intermediate language (Pyke, ASP, NLTK, FOL) affects the performance of neurosymbolic LLM reasoning, where an LLM translates a natural-language problem into a formal language and a symbolic solver computes the answer. The authors run a large benchmark (3 datasets, 7 LLMs, 8 prompting styles) and report execution-rate, execution-accuracy, and overall accuracy. Their headline result is an aggregate ordering FOL > NLTK > ASP > Pyke, with large per-LLM variation. The paper introduces the 'intermediate language challenge' as a framing and claims the formal language affects both syntactic and semantic reasoning capabilities.

Significance. If the reported ranking were robust, the paper would make a useful contribution: it identifies a largely overlooked design choice in neurosymbolic LLM systems, provides a substantial empirical database (305,760 queries), and decomposes accuracy into execution-rate and execution-accuracy, which is informative for diagnosing where translation fails. The explicit discussion of per-LLM differences and the use of several model sizes are also strengths. However, as presented, the central quantitative claim is not yet established. The single hand-crafted ICL example per dataset/language and the dedicated parser per language are confounds that can explain the ordering independently of the formal language itself, and the aggregate differences are not backed by significance testing. The paper is therefore best read as a hypothesis-generating study rather than a conclusive demonstration.

major comments (4)

[Section 5, Table 1 (left top)] The aggregate ordering FOL > NLTK > ASP > Pyke is not statistically supported. The FOL-vs-NLTK gap is only 65.29 - 60.36 = 4.93 points with SEMs 2.52 and 2.92; a naive comparison gives z ≈ 1.3, far from conventional significance. The authors themselves note in the per-LLM discussion that values are 'often in the range of the SEM, therefore, inconclusive.' Moreover, the ordering reverses across models: DeepSeek-V3 prefers NLTK (84.07) over FOL (76.54), DeepSeek-32B prefers ASP (63.74) over FOL (61.11), and DeepSeek-R1 (20480) prefers ASP (88.82) over FOL (85.45). A paired significance test across LLM × prompting-style × dataset conditions, or at least effect sizes with confidence intervals, is needed before claiming that FOL 'achieves the best results.'
[Section 4.2] The use of 'one partly hand-crafted ICL-example per dataset/formal language' is a severe confound. The eight prompting styles vary only formatting, not the logical content of the example, so a single example that happens to be clearer, shorter, or more representative for one language will shift all eight styles for that language. The resulting language ranking could be an artifact of example quality rather than of the formal language itself. The authors should release the exact prompts and ICL examples, and ideally run a control with multiple alternative examples per language (e.g., the same natural-language content encoded in all four languages) to show the ordering is stable to example choice.
[Sections 4.1 and 4.5] The execution-rate is determined by each language's dedicated parser (Pyke engine, Clingo, Prover9/NLTK, custom FOL parser), and Overall-Acc = Exec-Rate × Exec-Acc. If one parser is more lenient (accepts near-miss formulas) or stricter (rejects minor formatting deviations), that language's overall accuracy changes independently of the LLM's translation ability. The paper's own error analysis reports exactly such parser-strictness phenomena: missing line breaks in Pyke, unquoted 'not' in ASP, and parentheses/arity errors in NLTK/FOL. To support the claim that the formal language, not the parser implementation, drives performance, the authors need to define acceptance criteria for each parser and show the ranking is robust to the acceptance threshold.
[Section 4.2 and Section 5] FOLIO is excluded from the aggregate results in Figure 2 and from the per-LLM table: the aggregate uses only ProntoQA and ProofWriter, both generated datasets. The authors justify excluding ASP and Pyke from FOLIO because FOLIO requires classical-logic constructs, but this substantially narrows the scope of the headline claim. Either report FOLIO results separately for NLTK and FOL, or explicitly state in the abstract and conclusion that the conclusions apply to the two generated datasets only.

minor comments (5)

[Section 3] The argument that 'syntax affects LLMs' reasoning performance' is extrapolated from content effects in natural language (wumpus vs. cat scenarios). This is a useful motivating analogy, but it is not evidence about formal-language syntax. Presenting it as a conclusion overstates the support.
[Figure 1 caption] The sentence 'Subsequently, a symbolic reasoner subsequently computes a solution' contains a duplicated adverb. Please rephrase.
[Section 5 (last paragraph)] Typo: 'DeepSeeek-32B' should be 'DeepSeek-32B'.
[Section 4.2] The phrase 'training data' for the ICL example is potentially confusing, since the LLM weights are not updated. Consider using 'in-context example' throughout.
[General] The manuscript does not state whether prompts, ICL examples, parsers, or raw outputs will be released. For an empirical paper with potential confounds, an artifact/appendix is essential for reproducibility and for third parties to test the parser-strictness and example-quality concerns.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark with fixed inputs; ranking is measured, not constructed.

full rationale

No circularity found. The paper's central claim is an empirical observation: FOL/NLTK/ASP/Pyke accuracy across LLMs and datasets. The derivation chain is: define languages and ICL setup, run LLM translation, parse with language-specific solver, and measure execution-rate, execution-accuracy, and overall-accuracy. None of these quantities is fitted from the result; the hand-crafted ICL example per language is a fixed input, not a parameter tuned on test data. The 'intermediate language challenge' is a framing definition, not a conclusion assumed in advance. The only self-citation [9] is the long version of the same study and is not used as evidence. Possible confounds (example quality, parser strictness) are internal-validity threats, not circular reductions. Therefore score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper contains no fitted parameters or new postulated entities. The central claim is empirical, so the ledger mainly records experimental assumptions about fair comparison, determinism, and the correctness of the symbolic tools.

assumptions (4)

domain assumption LLM outputs under temperature 0 are treated as near-deterministic next-token predictions.
Section 4.3 sets temperature to 0 and treats the LLM as an immutable black box, so output variability is assumed not to change the ranking.
ad hoc to paper The hand-crafted ICL example and the same prompting styles are fair representatives of each formal language.
Section 4.2 says one partly hand-crafted ICL-example is used per dataset/language; this is a study-specific choice that the comparison assumes does not bias one language over another.
domain assumption The parsers and solvers (Clingo, Prover9, Pyke, NLTK/FOL parser) correctly implement the intended formal semantics.
Section 4.1 and 4.5 rely on these tools to decide what counts as syntactically correct and to compute the answer.
domain assumption Execution-rate and execution-accuracy cleanly separate syntactic from semantic capability.
Section 4.5 defines execution-rate as parse success and execution-accuracy as correct answers among parseable outputs; this assumes parser strictness does not confound the syntactic measure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoning." pith.science (2026). https://pith.science/paper/5BTGFUQW

@misc{pith2026250904083,
  author       = {Pith},
  title        = {Pith review of: Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5BTGFUQW}},
  note         = {Machine review of arXiv:2509.04083}
}

read the original abstract

Large language models (LLMs) achieve astonishing results on a wide range of tasks. However, their formal reasoning ability still lags behind. A promising approach is Neurosymbolic LLM reasoning. It works by using LLMs as translators from natural to formal languages and symbolic solvers for deriving correct results. Still, the contributing factors to the success of Neurosymbolic LLM reasoning remain unclear. This paper demonstrates that one previously overlooked factor is the choice of the formal language. We introduce the intermediate language challenge: selecting a suitable formal language for neurosymbolic reasoning. By comparing four formal languages across three datasets and seven LLMs, we show that the choice of formal language affects both syntactic and semantic reasoning capabilities. We also discuss the varying effects across different LLMs.

Figures

Figures reproduced from arXiv: 2509.04083 by the authors.

**Figure 1.** Neurosymbolic LLM reasoning: A problem formulated in natural language is translated by using in-context-learning into a formal language. Subsequently, a symbolic reasoner subsequently computes a solution to the problem, which is followed by the re-translation of the solution. on reasoning performance. We thereby differ from other related approaches like Logic-LM [7], Logic-LM++ [10], and LINC [8]. 2.3. Related Work … view at source ↗

**Figure 2.** Left: We show the effects of the formal languages, averaged across all prompting styles, LLMs, and the ProntoQA and ProofWriter datasets. Error bars show the SEM 𝑛 = 112. Right: Scatter plots comparing execution-rate to execution-accuracy for the formal languages. A single dot shows an average across prompting styles averaged over ProntoQA and ProofWriter datasets and all LLMs (𝑛 = 14). Contour lines show overall-ac… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 14 canonical work pages

[9]

Beiser, D

A. Beiser, D. Penz, N. Musliu, Intermediate Languages Matter: Formal Choice Drives Neurosymbolic LLM Reasoning, 2025. URL: https://arxiv.org/abs/2502.17216

arXiv 2025
[1]

Saparov, H

A. Saparov, H. He, Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought, ICLR23 (2023)

work page 2023
[2]

A. K. Lampinen, I. Dasgupta, S. C. Y. Chan, H. R. Sheahan, A. Creswell, D. Kumaran, J. L. McClelland, F. Hill, Language models, like humans, show content effects on reasoning tasks, PNAS Nexus 3 (2024). doi: 10.1093/pnasnexus/pgae233

doi:10.1093/pnasnexus/pgae233 2024
[3]

Panas, S

D. Panas, S. Seth, V. Belle, Can Large Language Models Put 2 and 2 Together? Probing for Entailed Arithmetical Relationships, in: NeSy24, 2024, pp. 258–276. doi: 10.1007/ 978-3-031-71170-1_21

work page 2024
[4]

B. Y. Lin, S. Lee, R. Khanna, X. Ren, Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models, in: EMNLP20, 2020, pp. 6862–6868. doi:10.18653/v1/2020.emnlp-main.557

doi:10.18653/v1/2020.emnlp-main.557 2020
[5]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain- of-Thought Prompting Elicits Reasoning in Large Language Models, in: NeurIPS22, 2022, pp. 24824–24837

work page 2022
[6]

Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, C. Callison-Burch, Faithful Chain-of-Thought Reasoning, in: IJCNLP23, 2023, pp. 305–329. doi: 10.18653/ v1/2023.ijcnlp-main.20

work page 2023
[7]

L. Pan, A. Albalak, X. Wang, W. Wang, Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning, in: EMNLP23, 2023, pp. 3806–3824. doi:10.18653/v1/2023.findings-emnlp.248

doi:10.18653/v1/2023.findings-emnlp.248 2023

Show all 31 references

[8]

Olausson, A

T. Olausson, A. Gu, B. Lipkin, C. Zhang, A. Solar-Lezama, J. Tenenbaum, R. Levy, LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers, in: EMNLP23, 2023, pp. 5153–5176. doi: 10.18653/v1/2023. emnlp-main.313

2023 doi
[10]

Kirtania, P

S. Kirtania, P. Gupta, A. Radhakrishna, LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations, in: ACL24, 2024, pp. 56–63. doi: 10.18653/v1/2024.nlrse-1.6

2024 doi
[11]

M. Geva, A. Gupta, J. Berant, Injecting Numerical Reasoning Skills into Language Models, in: ACL20, 2020, pp. 946–958. doi:10.18653/v1/2020.acl-main.89

2020 doi
[12]

Coppolillo, F

E. Coppolillo, F. Calimeri, G. Manco, S. Perri, F. Ricca, LLASP: Fine-tuning Large Language Models for Answer Set Programming, in: KR24, 2024, pp. 834–844. doi: 10.24963/kr. 2024/78

2024 doi
[13]

Shanahan, Talking about Large Language Models, Com

M. Shanahan, Talking about Large Language Models, Com. ACM 67 (2024) 68–79. doi: 10. 1145/3624724

2024
[14]

B. Sel, A. Al-Tawaha, V. Khattar, R. Jia, M. Jin, Algorithm of thoughts: enhancing explo- ration of ideas in large language models, in: ICML24, 2024

2024
[15]

A. d. Garcez, L. C. Lamb, Neurosymbolic AI: the 3rd wave, Artif Intell Rev 56 (2023) 12387–12406. doi:10.1007/s10462-023-10448-w

2023 doi
[16]

Badreddine, A

S. Badreddine, A. d’Avila Garcez, L. Serafini, M. Spranger, Logic Tensor Networks, AI 303 (2022) 103649. doi:10.1016/j.artint.2021.103649

2022
[17]

Eiter, T

T. Eiter, T. Geibinger, N. Higuera, J. Oetsch, A logic-based approach to contrastive explain- ability for neurosymbolic visual question answering, in: IJCAI23, 2023, pp. 3668–3676. doi:10.24963/ijcai.2023/408

2023 doi
[18]

Y. Wu, A. Q. Jiang, W. Li, M. N. Rabe, C. Staats, M. Jamnik, C. Szegedy, Autoformalization with Large Language Models, 2022. URL: http://arxiv.org/abs/2205.12615

2022 arXiv
[19]

J. Liu, S. Cao, J. Shi, T. Zhang, L. Hou, J. Li, How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering, in: ACL24, 2024

2024
[20]

N. Li, P. Liu, Z. Liu, T. Dai, Y. Jiang, S.-T. Xia, Logic-of-thought: Empowering large language models with logic programs for solving puzzles in natural language, 2025. URL: https://arxiv.org/abs/2505.16114

2025 arXiv
[21]

Tafjord, B

O. Tafjord, B. Dalvi, P. Clark, ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language, in: IJCNLP21, 2021, pp. 3621–3634. doi: 10.18653/v1/ 2021.findings-acl.317

2021 doi
[22]

S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, W. Zhou, J. Coady, D. Peng, Y. Qiao, L. Benson, e. al., FOLIO: Natural Language Reasoning with First-Order Logic, in: EMNLP24, 2024, pp. 22017–22031. doi:10.18653/v1/2024.emnlp-main.1229

2024 doi
[23]

Frederiksen, Applying Expert System Technology to Code Reuse with Pyke, 2008

B. Frederiksen, Applying Expert System Technology to Code Reuse with Pyke, 2008. URL: https://pyke.sourceforge.net/PyCon2008-paper.html

2008
[24]

Gelfond, N

M. Gelfond, N. Leone, Logic programming and knowledge representation—The A-Prolog perspective, AI 138 (2002) 3–38. doi: 10.1016/S0004-3702(02)00207-2

2002 doi
[25]

Schaub, S

T. Schaub, S. Woltran, Special Issue on Answer Set Programming, Künstliche Intell. 32 (2018) 101–103. doi:10.1007/s13218-018-0554-8

2018 doi
[26]

Kaminski, T

R. Kaminski, T. Schaub, On the Foundations of Grounding in Answer Set Programming, TPLP23 23 (2023) 1138–1197. doi:10.1017/S1471068422000308

2023 doi
[27]

Gebser, R

M. Gebser, R. Kaminski, B. Kaufmann, M. Ostrowski, T. Schaub, P. Wanko, Theory Solving Made Easy with Clingo 5, ICLP16 52 (2016) 1–15. doi: 10.4230/OASICS.ICLP.2016.2

2016 doi
[28]

S. Bird, E. Klein, E. Loper, Natural Language Processing with Python, O’Reilly Media Inc., 2009

2009
[29]

McCune, Prover9 and Mace4, 2010

W. McCune, Prover9 and Mace4, 2010. URL: http://www.cs.unm.edu/~mccune/Prover9

2010
[30]

Fahle, S

T. Fahle, S. Schamberger, M. Sellmann, Symmetry breaking, in: T. Walsh (Ed.), CP01, 2001, pp. 93–107

2001
[31]

Beiser, M

A. Beiser, M. Hecher, K. Unalan, S. Woltran, Bypassing the ASP Bottleneck: Hybrid Grounding by Splitting and Rewriting, in: IJCAI24, 2024, pp. 3250–3258. doi:10.24963/ ijcai.2024/360

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [9]

Beiser, D

A. Beiser, D. Penz, N. Musliu, Intermediate Languages Matter: Formal Choice Drives Neurosymbolic LLM Reasoning, 2025. URL: https://arxiv.org/abs/2502.17216

arXiv 2025

[2] [1]

Saparov, H

A. Saparov, H. He, Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought, ICLR23 (2023)

work page 2023

[3] [2]

A. K. Lampinen, I. Dasgupta, S. C. Y. Chan, H. R. Sheahan, A. Creswell, D. Kumaran, J. L. McClelland, F. Hill, Language models, like humans, show content effects on reasoning tasks, PNAS Nexus 3 (2024). doi: 10.1093/pnasnexus/pgae233

doi:10.1093/pnasnexus/pgae233 2024

[4] [3]

Panas, S

D. Panas, S. Seth, V. Belle, Can Large Language Models Put 2 and 2 Together? Probing for Entailed Arithmetical Relationships, in: NeSy24, 2024, pp. 258–276. doi: 10.1007/ 978-3-031-71170-1_21

work page 2024

[5] [4]

B. Y. Lin, S. Lee, R. Khanna, X. Ren, Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models, in: EMNLP20, 2020, pp. 6862–6868. doi:10.18653/v1/2020.emnlp-main.557

doi:10.18653/v1/2020.emnlp-main.557 2020

[6] [5]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain- of-Thought Prompting Elicits Reasoning in Large Language Models, in: NeurIPS22, 2022, pp. 24824–24837

work page 2022

[7] [6]

Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, C. Callison-Burch, Faithful Chain-of-Thought Reasoning, in: IJCNLP23, 2023, pp. 305–329. doi: 10.18653/ v1/2023.ijcnlp-main.20

work page 2023

[8] [7]

L. Pan, A. Albalak, X. Wang, W. Wang, Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning, in: EMNLP23, 2023, pp. 3806–3824. doi:10.18653/v1/2023.findings-emnlp.248

doi:10.18653/v1/2023.findings-emnlp.248 2023