Pith. sign in

REVIEW 4 major objections 3 minor 75 references

Do LLMs Dream of Discrete Algorithms?

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LLMs fail at strict discrete reasoning and that constraining the LLM to compose Prolog predicates and foreign functions into plans improves precision, coverage, and documentation on the DABStep multi-step reasoning…

desk verdict Claims empirical validation on DABStep but the manuscript contains no experimental results; a clear architecture proposal that overstates its evidence. read the letter →

arxiv 2506.23408 v1 pith:FTYOZFBZ submitted 2025-06-29 cs.LG cs.LO

Claudionor Coelho Jr , Yanen Li , Philip Tee This is my paper · ORCID
classification cs.LGcs.LO
keywords neuro-symbolicAIPrologLLMagentsfirst-orderlogicmulti-stepreasoningDABStepbenchmarkplanninginterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that probabilistic LLMs, however fluent, cannot be trusted for the strict logical reasoning that multi-step planning and safety-critical decisions demand, and that the fix is to constrain them with a discrete logic layer. The proposed architecture asks the LLM to act as a planner that writes Prolog programs by stitching together a fixed library of facts, rules, and foreign functions, so each step is grounded in first-order logic instead of statistical association. The paper claims this neurosymbolic approach reduces hallucination and wrong step decomposition, and that experiments on the DABStep benchmark show improved precision, coverage, and system documentation. A sympathetic reader would care because the proposal offers a concrete route to agents that are both flexible and verifiable, an engineering trade-off that purely neural systems have not resolved.

What carries the argument

Prolog treated as a first-order logic proof system is the load-bearing component. Facts express domain knowledge with no probability attached, rules compose those facts, and negation-as-failure gives the system expressive power that knowledge graphs and plain SQL-style traversal lack. On top of Prolog, the paper defines a typed vocabulary of foreign functions, around thirty entries such as calculating transaction fees, analyzing ACI usage, and filtering complex transactions, and a planner prompt that requires the LLM to output a JSON plan with an explanation, gaps, findings, plan, actions, result, and a self-assigned evaluation score. The mechanism works by forcing the LLM to stitch together these components into a Prolog program, so correctness rests on the components rather than on the model's probabilistic guesses.

What would settle it

Run the proposed planner on the 456 DABStep tasks and record both the self-assigned evaluation score and the exact-match correctness of the final answer; the central claim stands or falls on whether the self-score tracks correctness and on whether the Prolog-composed answers beat an LLM-only planner on the same exact-match metric.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the right unit of LLM reasoning is not the free-form answer or the arbitrary code snippet but the composition of a small set of predefined logical components. Casting the planner's task as generating Prolog code, with predicates for querying data, applying filters, aggregating, computing rates, and generating recommendations, turns an open-ended generation problem into a constrained grammar problem, which is what LLMs do best. The same move makes the plan inspectable: facts, rules, and negation are explicit, and the LLM is prompted to score its own plan against a rule-based evaluation metric. The paper states that on DABStep, a benchmark of 456 exact-match multi-step reasoning tasks over financial payment data, this approach improved precision, coverage, and system documentation, and thereby restored engineering rigor to LLM-driven agents.

Load-bearing premise

The approach assumes that a small fixed vocabulary of about thirty Prolog predicates and foreign functions can express every step needed for all 456 DABStep questions, and that the LLM's own confidence score is a reliable stand-in for exact-match correctness.

Editorial extensions

If this is right

  • If the approach works as claimed, LLM agents can solve multi-step, exact-match tasks by composing a trusted library of logic functions rather than generating free-form code.
  • The constrained grammar lowers the ceiling on hallucination because the LLM selects and orders known-good components instead of inventing steps.
  • Plans written in Prolog are inspectable and auditable, so system documentation, gap analysis, and effort estimation can be derived from the same artifacts that produce the answer.
  • Because the LLM's job becomes grammar-level parsing and composition, smaller or cheaper models may be sufficient for planning.
  • The MVC-based expansion of seed questions into full question sets provides a repeatable way to scope an agent before implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's framing implies that the practical ceiling of the architecture is set by the completeness of the predefined toolset; an obvious test is to measure exact-match coverage as a function of the number of predicates and to see whether missing functions are the dominant failure mode.
  • The self-assigned evaluation score in the prompt is a natural candidate to be calibrated against the benchmark's exact-match metric and then used as a reward signal for fine-tuning, which would turn the described pipeline into a learning loop rather than a prompted heuristic.
  • If the grammar constraint is the real source of reliability, the same benefit should appear with much weaker open models, which would be a testable consequence of the paper's claim that the LLM is mainly exercising pattern-matching on a known grammar.
  • The meet-in-the-middle position suggests a third axis for agent design, between fixed workflows and arbitrary code generation, that the paper sketches but leaves open: how to grow the toolset dynamically without losing the safety and verifiability that motivated Prolog.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper argues that LLMs are unreliable for discrete logical reasoning and proposes a neurosymbolic architecture in which an LLM planner composes a fixed library of Prolog predicates and foreign functions. The manuscript motivates this design with examples of facts, rules, and negation in Prolog, presents an agent architecture with memory, planning, and tools, and describes a prompt-based planning mechanism in which the LLM is asked to maximize a self-assigned 'evaluation' score. The paper claims in the abstract and in Section 9 that experiments on the DABStep benchmark validated the approach and improved precision, coverage, and system documentation. However, the manuscript contains no experimental results: Section 8 is a dataset overview, and no exact-match accuracies, baselines, error bars, or comparisons to other methods are reported anywhere. The only evaluative device described is the self-scoring prompt in Section 7.1, whose relationship to DABStep's exact-match metric is not established.

Significance. The high-level proposal—constraining LLM planners to compose verified, modular logic components—is a plausible direction for improving reliability in agentic systems, and the DABStep dataset overview in Section 8 is potentially useful context. The paper also clearly identifies relevant failure modes such as hallucination and incorrect step decomposition. However, the paper's stated significance rests entirely on the claimed experimental validation, and that validation is absent from the text. There are no machine-checked proofs, no released implementation, no reproducibility artifacts, and the only described evaluation mechanism is an LLM self-score whose calibration to the benchmark's exact-match metric is unverified. As submitted, the manuscript is a position or proposal piece, not a validated systems paper.

major comments (4)
  1. [Abstract and Section 9] The central empirical claim—that experiments on DABStep 'validated the effectiveness of this approach' and showed improved precision, coverage, and system documentation—is unsupported by any experimental data in the manuscript. Section 8 provides dataset statistics and sample questions but no result tables, no exact-match accuracy numbers, no baseline comparisons, and no error bars. This claim is load-bearing for the paper's conclusions, and it cannot be checked or reproduced from the text.
  2. [Section 7.1] The only evaluation mechanism presented is a prompt that asks the LLM to assign itself an 'evaluation' score, starting at 1.0 and subtracting fixed amounts (e.g., 0.2 for LLM-written assert predicates and 0.4 for certain Prolog calls). No evidence is given that this self-score is calibrated to the DABStep exact-match metric described in Section 8.1. Because the LLM is explicitly instructed to maximize this score, any reported improvement in it would measure conformity with the prompt, not correctness on the benchmark; the claimed validation therefore rests on a circular measure.
  3. [Table 4 and Section 8] The feasibility of the proposed approach depends on the completeness of a fixed set of roughly thirty functions for the 456 DABStep tasks, but the paper explicitly states that the full interfaces of these functions are omitted 'due to lack of space' and provides no coverage analysis showing which tasks the toolset can actually solve. Without such an analysis, the assumption that this fixed toolset is sufficient for exact-match answers is unverified.
  4. [Algorithm 1 and Section 7] The proposed pipeline requires manual validation of all enlarged questions and MVC components, as stated in the discussion of Algorithm 1, but the manuscript reports no measure of the human effort involved or of how much of the pipeline is automated. The Section 9 claims of scalability and 'systematic identification of functional gaps' are therefore not supported by the evidence presented.
minor comments (3)
  1. [Section 7.1] The prompt contains a typo: 'acccoding' should be 'according'.
  2. [Section 8.1] In the fees.json description, the monthly-fraud-level entry contains an unterminated example value ('For example ’7.7'); please fix the formatting and complete the sentence.
  3. [Sections 2–6] There are several spacing and typographical errors, such as 'Historical memoryserves' in Section 5.1.2, 'of of LLMs' in Section 6, and 'Gringrotts' in Section 4.1; a careful proofreading pass is needed.

Circularity Check

2 steps flagged · score 7.0 of 10

The central 'experiments validated DABStep' claim is not backed by any external measurement; the only scoring mechanism in the paper is a hand-authored, LLM-self-scored rubric, and the sufficiency of the toolset is delegated to the authors' prior work.

  1. self definitional [Section 7.1, 'Planning by Stitching Together Logical Components']
    "You will interpret the user's query and generate a prolog code that best matches the user's intent to the prolog code. In doing so, you will evaluate the prolog code accoding to the 'evaluation' metric, and you will try to maximize the 'evaluation' metric. ... start with an evaluation score of 1.0 after generating the code. - if you utilize an input to a foreign function before instantiating its value, set evaluation score to 0.0. - if you use an LLM written assert predicate, reduce the evaluation score by 0.2."

    The numeric 'evaluation' score is entirely defined by the authors' hand-set penalties inside the prompt, and the LLM is explicitly instructed to maximize it. This score is never calibrated to DABStep's exact-match metric or to any ground truth. The paper's Section 9 conclusion ('Our experiments, conducted using the DABStep benchmark, validated the effectiveness of this approach') therefore has no independent basis in the text: any improvement in this self-scored metric would measure the model's compliance with the authors' rubric, not correctness on the benchmark. The claimed validation reduces to the prompt's own reward definition.

  2. self citation load bearing [Section 7, 'The Role of Prolog to Assist Planner Activities'; Section 8.2, Table 4]
    "To answer the first question, we will refer the user to [20]. In this paper, we have shown that software engineering has changed from a structured design process to giving a sample of questions by product managers when using LLMs as interfaces. ... We follow the steps defined in [20] to create the tools we need to solve a given set of problems, i.e., starting from a set of seed questions from the user, we enlarge these questions ... generate a set of MVC components ..."

    The paper relies on [20] to justify that the fixed set of tools is sufficient: 'how many functions is enough' is answered by referring the reader to [20], and Table 4 is said to be inferred by Algorithm 1 from [20]. Reference [20] shares the first author and is not independently verified against the DABStep benchmark; the paper also states 'we have not listed here the full interface of the functions due to lack of space' and gives no coverage analysis showing that these roughly thirty functions suffice for all 456 tasks. Thus the load-bearing premise that the toolset can solve the benchmark rests on a self-citation chain rather than on external evidence.

full rationale

The paper is best read as an architecture proposal: use Prolog predicates and composable foreign functions, orchestrated by an LLM, to solve multi-step data tasks. That proposal is not itself circular. However, the paper's central empirical claim is stated in the abstract and Section 9: experiments on DABStep 'validated the effectiveness of this approach' and showed 'improved precision, coverage, and system documentation.' No experimental results, exact-match scores, baselines, or error bars appear anywhere in the manuscript; Section 8 is only a dataset overview. The only evaluative device present is the Section 7.1 prompt, in which the authors define a numeric 'evaluation' score with hand-selected penalties and ask the LLM to maximize it. Because this self-score is uncalibrated to the benchmark's exact-match metric, any validation derived from it would be circular: success would be defined as adherence to the authors' own rubric. Additionally, the sufficiency of the fixed function set—the load-bearing assumption that the toolset can cover the 456 DABStep tasks—is delegated to the authors' prior paper [20] rather than demonstrated with coverage or full function interfaces. These two issues make the claimed validation unsupported and partially circular, so the score is 7 rather than 0. The absence of data is also an empirical completeness problem, but it is intertwined with the circularity because the one metric the paper defines is itself authored by the system designers.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small set of hand-picked parameters (question-generation count, self-evaluation baseline and penalties) and on assumptions that Prolog negation and the authors' self-scoring prompt faithfully capture logical correctness. No new physical or algorithmic entities are introduced.

free parameters (3)
  • N (generated questions per seed query) = 40
    Table 4 caption states 'using 40 questions/query' in Algorithm 1; this number is chosen by the authors and affects the MVC extraction and tool coverage.
  • Evaluation score baseline = 1.0
    Section 7.1 prompt instructs the LLM to start with score 1.0 after generating code; this baseline is an arbitrary anchor.
  • Evaluation penalties = -0.2 for LLM-written assert; -0.4 for native Prolog functions on header+data
    Section 7.1 defines hand-crafted penalties to adjust the self-evaluation score; these weights are not derived from data.
assumptions (3)
  • domain assumption Prolog negation-as-failure (\+ ) is a faithful implementation of first-order-logic negation.
    Section 4.2 says 'Being able to express negation is what differentiates Prolog from traversals in knowledge graphs' and builds the planner on this equivalence.
  • ad hoc to paper The hand-crafted self-evaluation prompt produces scores that correlate with task correctness.
    Section 7.1 asks the LLM to maximize the evaluation metric; no calibration against the DABStep exact-match metric is provided.
  • domain assumption The DABStep benchmark's exact-match metric measures 'engineering rigor' and 'system reliability'.
    The abstract and Section 8 use DABStep as the basis for claims of improved precision and coverage, but no exact-match results are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLMs Dream of Discrete Algorithms?." pith.science (2026). https://pith.science/paper/FTYOZFBZ

@misc{pith2026250623408,
  author       = {Pith},
  title        = {Pith review of: Do LLMs Dream of Discrete Algorithms?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTYOZFBZ}},
  note         = {Machine review of arXiv:2506.23408}
}
read the original abstract

Large Language Models (LLMs) have rapidly transformed the landscape of artificial intelligence, enabling natural language interfaces and dynamic orchestration of software components. However, their reliance on probabilistic inference limits their effectiveness in domains requiring strict logical reasoning, discrete decision-making, and robust interpretability. This paper investigates these limitations and proposes a neurosymbolic approach that augments LLMs with logic-based reasoning modules, particularly leveraging Prolog predicates and composable toolsets. By integrating first-order logic and explicit rule systems, our framework enables LLMs to decompose complex queries into verifiable sub-tasks, orchestrate reliable solutions, and mitigate common failure modes such as hallucination and incorrect step decomposition. We demonstrate the practical benefits of this hybrid architecture through experiments on the DABStep benchmark, showing improved precision, coverage, and system documentation in multi-step reasoning tasks. Our results indicate that combining LLMs with modular logic reasoning restores engineering rigor, enhances system reliability, and offers a scalable path toward trustworthy, interpretable AI agents across complex domains.

Figures

Figures reproduced from arXiv: 2506.23408 by the authors.

Figure 1
Figure 1. Trade-off between fixed functionality and complete freedom to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Midnight sun in Sweden, during the summer, according to [31]. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. User Story to Order Pizza for a Food Delivery App [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: AI Agent from [50] 5.1.1 Agent Core The agent core is a crucial component within an AI Agent that plays a central role in orchestrating the agent’s overall functionality. It receives a query from the user. Consequently, it manages the decision-making pro￾cesses, commun…
Figure 5
Figure 5. Figure 5: AI Agent Representation From Interaction View [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Model-View-Controller system • Projection: List[str]: list of names from the table TableName that we will return to the user, according to the field names in the database. • Header: List[str]: list of field names from TableName or Projection if the projected list is no…
Figure 7
Figure 7. Figure 7: Model-View-Controller system for DABset expanded questions [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 67 canonical work pages

  1. [20]

    Enhancing llm reasoning through prolog: A breakthrough in symbolic logic processing, March 2025

    Chris Clark. Enhancing llm reasoning through prolog: A breakthrough in symbolic logic processing, March 2025

    work page 2025

  2. [1]

    Breaking down complex queries into smaller, logically coherent steps

    work page

  3. [2]

    Selecting or constructing functions to perform each step

    work page

  4. [3]

    4 Current LLM architectures, while powerful in generating plausible text, often lack the explicit structure needed for rigorous logical reasoning and reliable problem decomposition

    Composing these steps into an overall solution that is both correct and interpretable. 4 Current LLM architectures, while powerful in generating plausible text, often lack the explicit structure needed for rigorous logical reasoning and reliable problem decomposition. In the following sections, we examine why this is the case and how integrating formal lo...

    work page

  5. [4]

    Restaurant database that can be searched by location and by type of food

    work page

  6. [5]

    Menu database, where user can search for types of food served by the restaurant

    work page

  7. [6]

    Algorithm that computes the delivery time from the restaurant to your location. The reader should notice that this use case implements a single type of user interaction, and if we decide to modify the interaction, we will need to change the user story, or create another implementation that accommodates a different user story. In general, given as sample o...

    work page

  8. [7]

    Can this restaurant deliver food in 20 min?

    work page

Show all 75 references
  1. [8]

    Give me the list of all restaurants that deliver gourmet pizza in 20 min

  2. [9]

    The reader can easily see that the first question requires just a simple yes/no answer

    Give me the 20 top evaluated restaurants that can deliver gourmet pizza in 20 minutes. The reader can easily see that the first question requires just a simple yes/no answer. The second question requires a summarization or visual- ization agent to provide the answer. The third...

  3. [10]

    To answer the first question, we will refer the user to [20]

    whether it is possible to dynamically instantiate functions in this infras- tructure. To answer the first question, we will refer the user to [20]. In this paper, we have shown that software engineering has changed from a structured design process to giving a sample of questio...

  4. [11]

    Expand the list of questions from the users using the original sample questions as seed

  5. [12]

    explanation

    Extract the model-view-components from the questions, always reusing the previous components whenever possible The full algorithm proposed in [20] is described in Algorithm 1. It is worth noting that as LLMs can hallucinate, and since extraction of the specification is execute...

  6. [13]

    In the past [52], people have resorted to the specification of the user stories determine how the whole interaction with the user is driven, such as in the case of the Pizza ordering App, as presented in Figure 3

  7. [14]

    As advantage, it connects directly to Python infrastruc- ture

    In [13], we are able to dynamically generate, and keep trying until code passes. As advantage, it connects directly to Python infrastruc- ture. However, because the Python language is extremely complex (e.g. users have to specify which libraries they intend to use)

  8. [15]

    As advantage, this allows one to have better control over hallucination and security, and even use smaller or sim- pler models to generate the code

    We provide a large number of small components, and expect the LLM to ”compose” a solution by stitching together small portions to com- prise a full solution. As advantage, this allows one to have better control over hallucination and security, and even use smaller or sim- pler...

    2023

  9. [16]

    Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K. Reddy. H-star: Llm-driven hybrid sql-text adaptive reasoning on tables, 2025

    2025

  10. [17]

    The ai scientist: Towards fully automated open-ended scien- tific discovery

    Sakana AI. The ai scientist: Towards fully automated open-ended scien- tific discovery. https://github.com/SakanaAI/AI-Scientist, 2024. Accessed: 2025-06-20

    2024

  11. [18]

    Ai agents are disrupting automation: Current approaches, market solutions and rec- ommendations, 2024

    Praveen Akkiraju, Sophie Beshar, and Hunter Korn. Ai agents are disrupting automation: Current approaches, market solutions and rec- ommendations, 2024

    2024

  12. [19]

    Piantadosi

    Nasim Borazjanizadeh and Steven T. Piantadosi. Reliable reasoning beyond natural language, 2024

    2024

  13. [21]

    Clocksin

    William F. Clocksin. Programming in Prolog: Using The Iso Standard . Springer, 5th edition, 2003

    2003

  14. [22]

    Crewai: Orchestrate autonomous ai agents as a crew

    CrewAI Contributors. Crewai: Orchestrate autonomous ai agents as a crew. https://github.com/joaomdmoura/crewai, 2024. Accessed: 2025-06-20

    2024

  15. [23]

    Experimenting with planning and reasoning in ad 33 hoc teamwork environments with large language models

    Polyana Costa, Pedro Santos, Jos´ e Boaro, Daniel Moraes, J´ ulio Duarte, and Sergio Colcher. Experimenting with planning and reasoning in ad 33 hoc teamwork environments with large language models. In Proceedings of the 16th International Conference on Agents and Artificial I...

    2024

  16. [24]

    Se- cure by design alert: Eliminating os command injection vul- nerabilities

    Cybersecurity and Infrastructure Security Agency (CISA). Se- cure by design alert: Eliminating os command injection vul- nerabilities. https://www.cisa.gov/resources-tools/resources/ secure-design-alert-eliminating-os-command-injection-vulnerabilities ,

  17. [25]

    Data visualization reference guides, 2025

    Randy Krum. Data visualization reference guides, 2025. Accessed: 2025-06-20

    2025

  18. [26]

    Z3: an efficient smt solver,

    Leonardo de Moura and Nikolaj Bjørner. Z3: an efficient smt solver,

  19. [27]

    Pierrehumbert

    Fangru Lin, Emanuele La Malfa, Valentin Hofmann, Elle Michelle Yang, Anthony Cohn, and Janet B. Pierrehumbert. Graph-enhanced large language models in asynchronous plan reasoning, 2024. 35

    2024

  20. [28]

    The Lean Language Ref- erence Manual

    Leonardo de Moura, Sebastian Ullrich, et al. The Lean Language Ref- erence Manual. Lean Community, 2024. Accessed: 2025-06-20

    2024

  21. [29]

    Reasoning and planning with large language models in code development

    Hao Ding, Ziwei Fan, Ingo Guehring, Gaurav Gupta, Wooseok Ha, Jun Huan, Linbo Liu, Behrooz Omidvar-Tehrani, Shiqi Wang, and Hao Zhou. Reasoning and planning with large language models in code development. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery ...

    2024

  22. [30]

    Smolagents: Agents that think in code

    Hugging Face. Smolagents: Agents that think in code. https: //github.com/huggingface/smolagents, 2024. Accessed: 2025-06- 20

    2024

  23. [31]

    A survey on rag meeting llms: Towards retrieval-augmented large language models, 2024

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language models, 2024

    2024

  24. [32]

    D’Avila Garcez, Luis C

    Artur S. D’Avila Garcez, Luis C. Lamb, and Dov M. Gabbay. Neural- Symbolic Cognitive Reasoning (Cognitive Technologies). Springer, 2009

    2009

  25. [33]

    The 4 degrees of anthropomorphism of generative ai

    Sarah Gibbons, Tarun Mugunthan, and Jakob Nielsen. The 4 degrees of anthropomorphism of generative ai. Nielsen Norman Group , October 2023

    2023

  26. [34]

    On the potential of logic and reasoning in neurosymbolic systems using owl-based knowledge graphs

    David Herron, Ernesto Jim´ enez-Ruiz, and Tillman Weyde. On the potential of logic and reasoning in neurosymbolic systems using owl-based knowledge graphs. Neurosymbolic Artificial Intelligence , 1:29498732251320043, 2025. 34

    2025

  27. [35]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transac- tions on I...

    2025

  28. [36]

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition and applications. arXiv:2002.00388 [cs.CL], 2020

    2002 arXiv

  29. [37]

    Coelho Jr, Hanchen Xiong, Tushar Karayil, Sree Ko- ratala, Rex Shang, Jacog Bollinger, Mohamed Shabar, and Syam Nair

    Claudionor N. Coelho Jr, Hanchen Xiong, Tushar Karayil, Sree Ko- ratala, Rex Shang, Jacog Bollinger, Mohamed Shabar, and Syam Nair. Effort and size estimation in software projects with large language model-based intelligent interfaces. arXiv:2402.07158 [cs.SE], 2024

    2024 arXiv

  30. [38]

    Dan Jurafsky and James H. Martin. Speech and Language Process- ing. https://web.stanford.edu/ jurafsky/slp3, 3rd draft edition, Jan- uary 2025

    2025

  31. [39]

    (how) do reasoning models reason? Annals of the New York Academy of Sciences, April 2025

    Subbarao Kambhampati, Kaya Stechly, and Karthik Valmeekam. (how) do reasoning models reason? Annals of the New York Academy of Sciences, April 2025

    2025

  32. [40]

    Stop anthropomorphizing intermediate toakens as reasoning/thinking traces arXiv:2504.09762 [cs.AI], 2025

    Subbarao Kambhampati, Kaya Stechly, Karthik Valmeekam, Lu- cas Saldyt, Siddhant Bhambri, Vardhan Palod, Atharva Gundawar, Soumya Rani Samineni, Durgesh Kalwar, and Upasana Biswas. Stop anthropomorphizing intermediate toakens as reasoning/thinking traces arXiv:2504.09762 [cs.AI], 2025

    2025 arXiv

  33. [41]

    Software is changing (again), 2025

    Andrej Karpathy. Software is changing (again), 2025. Keynote at AI Startup School, published by Y Combinator. Accessed: 2025-06-20

    2025

  34. [42]

    On the self-verification limitations of large language models on reasoning and planning tasks, 2024

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks, 2024

    2024

  35. [43]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨ aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  36. [44]

    Thought-like-pro: Enhancing reasoning of large language models through self-driven prolog-based chain-of- thought, 2024

    Xiaoyu Tan, Yongxin Deng, Xihe Qiu, Weidi Xu, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. Thought-like-pro: Enhancing reasoning of large language models through self-driven prolog-based chain-of- thought, 2024

    2024

  37. [45]

    Large language mod- eling

    Kyle Lo, Akshita Bhagia, and Nathan Lambert. Large language mod- eling. Tutorial at Neural Information Processing Systems (NeurIPS), December 2024

    2024

  38. [46]

    Llm and prolog: the logical alternative to chain-of-thought reasoning, April 2025

    Lu Mao. Llm and prolog: the logical alternative to chain-of-thought reasoning, April 2025

    2025

  39. [47]

    Rossi, Handong Zhao, Ruiyi Zhang, Puneet Mathur, Nedim Lipka, Yu Wang, Trung Bui, Franck Dernoncourt, and Tianyi Zhou

    Dang Nguyen, Viet Dac Lai, Seunghyun Yoon, Ryan A. Rossi, Handong Zhao, Ruiyi Zhang, Puneet Mathur, Nedim Lipka, Yu Wang, Trung Bui, Franck Dernoncourt, and Tianyi Zhou. Dynasaur: Large language agents beyond predefined actions, 2024

    2024

  40. [48]

    Visit swe- den, 2025

    Sweden’s official website for tourism and travel information. Visit swe- den, 2025

    2025

  41. [49]

    A Brief History of Computing

    Gerard O’Regan. A Brief History of Computing . Springer, 3rd edition, 2021

    2021

  42. [50]

    Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning, 2023

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning, 2023

    2023

  43. [51]

    Lee, and Eunho Yang

    Hyun Ryu, Gyeongman Kim, Hyemin S. Lee, and Eunho Yang. Divide and translate: Compositional first-order logic translation and verifica- tion for complex logical reasoning, 2025

    2025

  44. [52]

    Prolog’s role in the llm era – part 1, August 2024

    Eugene Sahara. Prolog’s role in the llm era – part 1, August 2024

    2024

  45. [53]

    Cot is not true reasoning, it is just a tight constraint to imitate: A theory perspective, 2025

    Jintian Shao and Yiming Cheng. Cot is not true reasoning, it is just a tight constraint to imitate: A theory perspective, 2025

    2025

  46. [54]

    Ho, Carl Yang, and May Dongmei Wang

    Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C. Ho, Carl Yang, and May Dongmei Wang. EHRA- gent: Code empowers large language models for few-shot complex tabu- lar reasoning on electronic health records. In Yaser Al-Onaizan, Mohit Bansal, a...

    2024

  47. [55]

    The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025

    Parshin Shojaee* †, Iman Mirzadeh*, Keivan Alizadeh, Maxwell Hor- ton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. 36

    2025

  48. [56]

    The ripple effect: How innovation disrupts, 2024

    Jen Singleton and Jim Hare. The ripple effect: How innovation disrupts, 2024

    2024

  49. [57]

    Step into the future of zdx with 3 exciting new fea- tures: Zdx copilot, data explorer, and hosted monitoring

    Vikas Srivastava. Step into the future of zdx with 3 exciting new fea- tures: Zdx copilot, data explorer, and hosted monitoring. Zscaler Blog, April 2024

    2024

  50. [58]

    Asana warns mcp ai feature exposed customer data to other orgs, 2025

    BleepingComputer News Staff. Asana warns mcp ai feature exposed customer data to other orgs, 2025. Accessed: 2025-06-20

    2025

  51. [60]

    The Art of Prolog: Advanced Pro- gramming Techniques (Mit Press Series in Logic Programming)

    Leon Sterling and Ehud Shapiro. The Art of Prolog: Advanced Pro- gramming Techniques (Mit Press Series in Logic Programming) . MIT Press, 2nd edition, 1986

    1986

  52. [62]

    Hol - interactive theorem prover, 2025

    HOL team. Hol - interactive theorem prover, 2025

    2025

  53. [63]

    Robust, mature, free

    SWI-Prolog team. Robust, mature, free. prolog for the real world

  54. [64]

    Log2NS: Enhancing deep learning based analysis of logs with formal to prevent survivorship bias

    Charanraj Thimmisetty, Praveen Tiwari, Didac Gil de la Iglesia, Nan- dini Ramanan, Marjorie Sayer, Viswesh Ananthakrishnan, and Clau- dionor Nunes Coelho, Jr. Log2NS: Enhancing deep learning based analysis of logs with formal to prevent survivorship bias. https: //arxiv.org/ab...

    2021 arXiv

  55. [65]

    Priyesh Vakharia, Abigail Kufeldt, Max Meyers, Ian Lane, and Leilani H. Gilpin. ProSLM: A Prolog Synergized Language Model for ex- plainable Domain Specific Knowledge Based Question Answering , page 291–304. Springer Nature Switzerland, 2024

    2024

  56. [66]

    Logic and Structure

    Dirk van Dalen. Logic and Structure . Springer-Verlag, 5th edition, 2013

    2013

  57. [67]

    Introduction to llm agents

    Tanay Varshney. Introduction to llm agents. https://developer. nvidia.com/blog/introduction-to-llm-agents/ , November 2023. Accessed on December 19, 2023. 37

    2023

  58. [68]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. https://arxiv.org/abs/1706.03762, 2017

    2017 arXiv

  59. [69]

    Voorhees

    D.P. Voorhees. Guide to Efficient Software Design: An MVC Ap- proach to Concepts, Structures, and Models. Texts in Computer Science. Springer International Publishing, 2021

    2021

  60. [70]

    From human memory to ai memory: A survey on memory mechanisms in the era of llms, 2025

    Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to ai memory: A survey on memory mechanisms in the era of llms, 2025

    2025

  61. [71]

    The rise and potential of large language model based agents: A survey, 2023

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shi- han Dou, Rongxiang Weng,...

    2023

  62. [72]

    Arithmetic reasoning with llm: Prolog generation & permutation, 2024

    Xiaocheng Yang, Bingsen Chen, and Yik-Cheung Tam. Arithmetic reasoning with llm: Prolog generation & permutation, 2024

    2024

  63. [73]

    Ratt: A thought structure for coherent and correct llm reasoning, 2024

    Jinghan Zhang, Xiting Wang, Weijieying Ren, Lu Jiang, Dongjie Wang, and Kunpeng Liu. Ratt: A thought structure for coherent and correct llm reasoning, 2024

    2024

  64. [74]

    Learning to reason without external rewards, 2025

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards, 2025

    2025

  65. [75]

    Kg4diagnosis: A hierarchical multi-agent llm framework with knowledge graph enhance- ment for medical diagnosis, 2025

    Kaiwen Zuo, Yirui Jiang, Fan Mo, and Pietro Lio. Kg4diagnosis: A hierarchical multi-agent llm framework with knowledge graph enhance- ment for medical diagnosis, 2025. 38

    2025

  66. [2008]

    Accessed: June 5, 2025

    2025

  67. [2024]

    Accessed: 2025-06-20

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.