Pith. sign in

REVIEW 3 major objections 4 minor 17 references

LP-LM: No Hallucinations in Question Answering with Logic Programming

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LP-LM claims that parsing questions into Prolog terms and retrieving answers by unification against a knowledge base eliminates hallucination in question answering.

desk verdict Solid little grounded-QA demo whose advertised 'no hallucination' guarantee only holds for a trusted KB; the paper's own false-fact example undercuts the headline claim. read the letter →

arxiv 2502.09212 v1 pith:LM3ZBYGF submitted 2025-02-13 cs.AI cs.CL

Katherine Wu , Yanhong A. Liu This is my paper · ORCID
classification cs.AIcs.CL
keywords logicprogrammingquestionansweringhallucinationdefiniteclausegrammarprobabilisticcontext-freeknowledgebasePrologsemanticparsing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LP-LM is a question-answering system built on a simple design: every known fact and every question is converted into the same formal representation, so answering becomes a matching operation rather than a statistical guess. The paper claims this eliminates hallucination, because the answer is read off a knowledge base through Prolog unification and cannot be invented. The authors show that current LLMs give plausible but wrong answers on simple, context-dependent prompts, while LP-LM returns the stored fact. The payoff, if the claim holds, is a concrete demonstration that reliability in question answering does not require a larger model, only a representation that makes answers verifiable. The system is also efficient, with definite clause grammars and tabling giving linear-time parsing for grammars with many production rules.

What carries the argument

The machinery is a probabilistic definite clause grammar pipeline running under tabling, with unification as the retrieval step. The PCFG supplies probabilities so that each sentence has a well-defined most probable parse; the DCG rules in Prolog make those productions executable, building both a constituency tree and a semantic term on the way. The semantic term always takes the verb's root form as its functor, which gives different surface sentences the same retrievable shape. Tabling memoizes repeated subgoals during the search for the best parse, which the paper credits for linear-time behaviour on grammars with many production rules. Unification then plays the role of query execution: a question term containing variables is matched against stored fact terms, and the resulting substitution is the answer.

What would settle it

Insert a false statement that the grammar accepts, such as 'Fir trees can grow in human lungs', then ask 'What can grow in human lungs?' and observe whether LP-LM answers 'fir trees'. If it does, the system's reliability is inherited entirely from the truthfulness of the knowledge base, and the unconditional no-hallucination claim fails in exactly the way the paper's comparison suggests LLMs fail.

Watch

Extended reading notes

Core claim

LP-LM's central claim is that hallucination can be avoided by construction rather than reduced by better statistics. A probabilistic context-free grammar assigns probabilities to English sentence structures; LP-LM parses an input with that grammar, selects the most probable constituency parse, and translates it into a Prolog term whose functor is the root form of the verb and whose arguments name the entities involved. Statements are inserted into a knowledge base as such terms, and questions are turned into terms with variables that are unified against the knowledge base: a wh-question gets its answer by substitution, and a yes/no question by checking whether the fully instantiated term is stored. Because the answer must be an exact match or a substitution instance of a stored fact, the system cannot produce a response unsupported by the knowledge base. The paper's experiments with well-known LLMs show that these models give plausible but wrong answers on the same simple prompts, while LP-LM returns the stored fact.

Load-bearing premise

The no-hallucination guarantee assumes the knowledge base contains only true facts and the hand-written grammar correctly converts every question into the intended internal query; if a user inserts a false statement or the grammar mis-parses a question, LP-LM returns a wrong answer while still being described as reliable.

Editorial extensions

If this is right

  • For any question the grammar can parse and whose fact is in the knowledge base, LP-LM returns exactly the stored answer and cannot produce a conflicting invented one.
  • Because statements are inserted as explicit facts, the full set of possible answers is transparent, and every returned answer can be traced to the specific fact that produced it.
  • The linear-time parsing result means the no-hallucination behaviour does not come at the cost of scalability for grammars with many production rules.
  • Extending the grammar to cover more sentence patterns extends the reliable-answering guarantee to those patterns without changing the retrieval mechanism.
  • The comparison with LLMs implies that small, deterministic retrieval systems can outperform much larger generative models on simple relational question answering when the relevant facts are already available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'always reliable' wording is conditional in a way the paper's framing obscures: LP-LM is only as trustworthy as the knowledge base it is given, because a false inserted statement is returned just as faithfully as a true one.
  • The LLM comparison is illustrative rather than a systematic benchmark; a natural next step is a large accuracy and coverage study that measures both how often LP-LM is correct and how many valid questions its grammar can even accept.
  • A hybrid architecture is an obvious extension: use an LLM to clean and normalise messy input into canonical statements, then let LP-LM perform the retrieval, which would widen the class of processable sentences while keeping the no-invention guarantee for the distilled facts.
  • The acceptance of false statements such as 'fir trees can grow in human lungs' suggests that real deployments need a separate curation or verification step, because the method itself cannot distinguish true from false entries.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces LP-LM, a question-answering system that combines a probabilistic context-free grammar (PCFG), Definite Clause Grammars (DCGs), and tabling in XSB Prolog to parse natural-language statements and questions, store statements as Prolog terms in a knowledge base (KB), and answer questions by unifying the parsed query term against KB facts. The authors claim that this architecture 'always produces answers that are reliable' and that LLMs 'hallucinate on even simple questions, unlike LP-LM.' They support these claims with four hand-selected example prompts comparing LP-LM with GPT-4o, GPT-4o mini, and Gemini, and with efficiency experiments on 12 synthetic grammars that they interpret as showing linear-time parsing for large grammars.

Significance. If the claims were properly scoped, the paper would provide a useful, transparent demonstration that a logic-programming-based retrieval mechanism can give machine-checkable grounding for answers to simple English questions. The strengths are real: the implementation is publicly available, the Prolog-based parsing and unification pipeline is described concretely, and the efficiency experiments include reproducible measurements. However, the headline claims are substantially overstated. The 'no hallucinations' guarantee is definitional once one assumes a truthful KB and a perfect hand-written grammar, and the paper's own Table 1 shows the system emitting a false statement as an answer. The LLM comparison also appears to mislabel correct, appropriately cautious LLM responses as hallucinations. The contribution is better characterized as a small, well-executed demonstration of KB-faithful retrieval than as a system that eliminates hallucination in any practically meaningful sense.

major comments (3)
  1. [Section 4, Table 1] The abstract's claim that LP-LM 'always produces answers that are reliable' is contradicted by the paper's own example in Table 1, second row. After the false statement 'Fir trees can grow in human lungs' is inserted via add_kb, LP-LM answers 'What can grow in human lungs?' with exactly that false sentence. Since Section 1 defines hallucination as generating information that is 'factually incorrect,' this output is a hallucination under the paper's own definition. The no-hallucination property therefore holds only under an unstated assumption that the user-populated KB contains only true facts, and only for inputs the hand-written PCFG parses correctly. The authors must either revise the 'always reliable' and 'no hallucination' claims to state these conditions explicitly, or provide a mechanism that validates inserted statements against ground truth.
  2. [Section 3, Table 1] The comparison with LLMs is misleading and appears to invert the standard notion of hallucination. In the first three rows, GPT-4o, GPT-4o mini, and Gemini answer 'What causes temporary hearing loss?' with general causes such as loud noise, ear infections, and ototoxicity; these responses do not assert the user's specific premise 'Furosemide causes temporary hearing loss' and are at worst incomplete, not hallucinated. In the fourth row, GPT-4o mini explicitly says 'fir trees cannot grow in human lungs,' which is factually correct and appropriately contradicts the false premise; labeling this as a hallucination is not defensible. The authors need a transparent evaluation rubric that distinguishes unsupported claims, contextually appropriate corrections, and refusals, and they should report results over multiple runs and a broader set of queries before claiming that LLMs 'hallucinate on even simple questions.'
  3. [Section 4.3, Figures 2-5] The linear-time claim is not established by the evidence presented. The experiments cover only 12 hand-constructed grammars with 3-5 input sentences of length 1-50 per grammar, with no statistical analysis or error bars, and the comparison is against NLTK's Viterbi parser described as a 'greedy heuristic' rather than a state-of-the-art baseline. The phrase 'runs in linear time in the size of input sentences' in the abstract is therefore too strong. The authors should either provide a formal complexity argument (for example, showing that with tabling and a fixed finite grammar the number of table entries is linear in sentence length) or weaken the claim to an observed empirical trend on the tested grammars.
minor comments (4)
  1. [Section 4.2] The sentence 'The term run(X), where X is a variable, will be unified with a matching rule in the KB' should say 'matching fact' or 'matching term,' since the KB contains terms representing facts, not Prolog rules.
  2. [Section 4.3] The description of the x-axis in Figures 2-5 is vague: 'Higher numbered test cases represent sentences with longer lengths' would be clearer if each figure listed the actual sentence lengths or included a table of test-case parameters.
  3. [Section 5] The limitations paragraph acknowledges grammar coverage and the lack of reasoning capabilities, but it does not acknowledge the much more fundamental limitation that the system is only as reliable as the user-populated KB; this should be stated explicitly alongside the other limitations.
  4. [Section 2, Figure 1] The caption says 'The two Prolog versions are equivalent,' but the DCG rules and the hand-written Prolog clauses are not literally identical in all respects (e.g., argument order and the explicit difference lists); consider saying 'semantically equivalent' for clarity.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'always reliable / no hallucination' guarantee is definitional: LP-LM returns exactly the KB facts it was given, so the claim holds only under an unstated truthful-KB assumption, as Table 1's false-fact example shows.

  1. self definitional [Abstract; Section 4, workflow bullets and Table 1]
    "Abstract: 'LP-LM ... grounds answers to questions in known facts contained in a knowledge base (KB), facilitated through semantic parsing in Prolog, and always produces answers that are reliable.' Section 4: 'For statements, insertions into the KB are performed: the term is inserted dynamically into the KB. For questions, retrievals from the KB are performed: the term is matched against the KB and an answer is obtained by unification.'"

    The system's only source of answers is the KB it was given: statements are parsed into Prolog terms and added via add_kb, and questions are answered by unifying their query term against those same terms. Hence the output is, by construction, the input fact that matches the query. The 'reliable' / 'no hallucination' property is therefore not derived or measured; it is equivalent to the assumption that every inserted statement is true. The paper's own Table 1 demonstrates the collapse: after the false statement 'Fir trees can grow in human lungs' is inserted, LP-LM answers 'What can grow in human lungs?' with exactly that falsehood. No component checks factual correctness, and Section 5 concedes the system handles only simple retrieval.

full rationale

The technical retrieval mechanism is transparent and the efficiency experiments with DCG parsing and tabling are independent, self-contained results; there are no fitted parameters and no load-bearing self-citation chains. However, the paper's central advertised property, 'always produces answers that are reliable' and 'no hallucinations,' reduces by construction to the contents of the user-populated KB. Because statements are inserted verbatim as Prolog terms and answers are obtained by unification against those terms, the answer is the input fact itself. Table 1 makes this explicit: inserting the false statement 'Fir trees can grow in human lungs' causes LP-LM to answer with that same false statement, while the LLM's refusal to endorse the false premise is labeled a hallucination. This is partial circularity in the headline claim, though not in the underlying implementation, so the score is 6 rather than higher.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The system's guarantees are conditional on a manually authored grammar, a user-supplied knowledge base, and the correctness of a semantic term representation. There are no fitted numerical predictions, so the main risk is not parameter fitting; it is the narrowness of the assumptions.

free parameters (1)
  • PCFG production probabilities = Hand-set constants such as 0.25, 0.2, 0.09, 0.05 in Section 4.1
    The probabilities determine the most probable parse tree and are manually authored, not learned from a corpus or validated. For ambiguous inputs, different probabilities would select different parses and answers.
assumptions (5)
  • standard math Prolog DCG and tabling correctly implement PCFG parsing and preserve the claimed runtime behavior.
    The paper relies on XSB tabling to argue for linear-time parsing in Section 4.3; this is background logic-programming semantics, not proven in the paper.
  • domain assumption The semantic term representation, with the root verb as functor and arguments for subject and object, faithfully represents English sentence meaning.
    Section 4.1 constructs terms like runs(Bob); if this mapping is wrong, unification retrieves semantically wrong answers.
  • domain assumption The knowledge base is the source of truth and every statement inserted by the user is accepted as a true fact.
    Section 4 says statements are inserted into the KB; the example with "Fir trees can grow in human lungs" shows the system returns user-supplied false premises as answers.
  • domain assumption User inputs relevant to the system are generated by the hand-written PCFG.
    The no-hallucination and always-answer claims are limited to in-grammar sentences; Section 5 concedes generalization is a limitation.
  • ad hoc to paper The synthetic grammar benchmark of 12 grammars and a few sentences is representative of real parsing workloads.
    Section 4.3 uses hand-constructed grammars to claim linear time; no proof or broader benchmark is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LP-LM: No Hallucinations in Question Answering with Logic Programming." pith.science (2026). https://pith.science/paper/LM3ZBYGF

@misc{pith2026250209212,
  author       = {Pith},
  title        = {Pith review of: LP-LM: No Hallucinations in Question Answering with Logic Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LM3ZBYGF}},
  note         = {Machine review of arXiv:2502.09212}
}
read the original abstract

Large language models (LLMs) are able to generate human-like responses to user queries. However, LLMs exhibit inherent limitations, especially because they hallucinate. This paper introduces LP-LM, a system that grounds answers to questions in known facts contained in a knowledge base (KB), facilitated through semantic parsing in Prolog, and always produces answers that are reliable. LP-LM generates a most probable constituency parse tree along with a corresponding Prolog term for an input question via Prolog definite clause grammar (DCG) parsing. The term is then executed against a KB of natural language sentences also represented as Prolog terms for question answering. By leveraging DCG and tabling, LP-LM runs in linear time in the size of input sentences for sufficiently many grammar rules. Performing experiments comparing LP-LM with current well-known LLMs in accuracy, we show that LLMs hallucinate on even simple questions, unlike LP-LM.

Figures

Figures reproduced from arXiv: 2502.09212 by the authors.

Figure 1
Figure 1. An example Prolog DCG and a parse. The two Prolog versions are equivalent. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Plots for left-recursive grammars of increasing size [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Running times for right-recursive grammars of increasing size [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Running times for unambiguous grammars of increasing size [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Running times for ambiguous grammars of increasing size [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

  1. [1]

    Proceedings of the Conference Pacific Association for Computational Linguistics, pp

    Tony Abou-Assaleh, Nick Cercone & Vlado Keselj (2003):Expressing Probabilistic Context-Free Grammars in the Relaxed Unification Formalism. Proceedings of the Conference Pacific Association for Computational Linguistics, pp. 29–36

    work page 2003

  2. [2]

    Hossein Bahak, Farzaneh Taheri, Zahra Zojaji & Arefeh Kazemi (2023): Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models , doi:10.48550/arXiv.2312.07592

    arXiv doi:10.48550/arxiv.2312.07592 2023

  3. [3]

    In: Proceedings of the 24th International Workshop on Functional and (Constraint) Logic Programming, EPTCS 234, pp

    Stefan Brass & Heike Stephan (2017): Experiences with Some Benchmarks for Deductive Databases and Implementations of Bottom-Up Evaluation. In: Proceedings of the 24th International Workshop on Functional and (Constraint) Logic Programming, EPTCS 234, pp. 57–72, doi:10.4204/EPTCS.234.5

    work page doi:10.4204/eptcs.234.5 2017

  4. [4]

    Brown University Department of Computer Science

    Glenn Carroll & Eugene Charniak (1992): Two experiments on learning probabilistic dependency grammars from corpora. Brown University Department of Computer Science

    work page 1992

  5. [5]

    New York University

    John Cocke (1969): Programming languages and their compilers: Preliminary notes. New York University

    work page 1969

  6. [6]

    Jay Earley (1970): An efficient context-free parsing algorithm . Commun. ACM 13(2), p. 94–102, doi:10.1145/362007.362035

    arXiv 1970

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang & Haofen Wang (2024): Retrieval-Augmented Generation for Large Language Models: A Survey , doi:10.48550/arXiv.2312.10997

    arXiv doi:10.48550/arxiv.2312.10997 2024

  8. [8]

    Proceedings of the International Confer- ence RANLP, pp

    Christian Theil Have (2009): Stochastic definite clause grammars. Proceedings of the International Confer- ence RANLP, pp. 139–143. Available at https://aclanthology.org/R09-1027/

    work page 2009

Show all 17 references
  1. [9]

    ACM Computing Surveys 55(12), p

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto & Pascale Fung (2023): Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12), p. 1–38, doi:10.1145/3571730. Katherine Wu & Yanhong A. Liu 77

    2023 doi

  2. [10]

    Technical Report, Air Force Cambridge Research Lab, Bedford, MA

    Tadao Kasami (1965): An efficient recognition and syntax-analysis algorithm for context-free languages . Technical Report, Air Force Cambridge Research Lab, Bedford, MA

    1965

  3. [11]

    Young (1990): The estimation of stochastic context-free grammars using the inside- outside algorithm

    Karim Lari & Steve J. Young (1990): The estimation of stochastic context-free grammars using the inside- outside algorithm. Computer speech & language 4(1), pp. 35–56, doi:10.1016/0885-2308(90)90022-X

    1990 doi

  4. [12]

    Warren (1994): XSB as an Efficient Deductive Database Engine

    Konstantinos Sagonas, Terrance Swift & David S. Warren (1994): XSB as an Efficient Deductive Database Engine. In: Proceedings of the 1994 ACM SIGMOD International Conference on Management of Data , ACM, pp. 442–453, doi:10.1145/191839.191927

    1994

  5. [13]

    In: EATCS Monographs on Theoretical Computer Science, doi:10.1007/978-3-642-61345-6

    Seppo Sippu & Eljas Soisalon-Soininen (1988): Parsing Theory - Volume I: Languages and Parsing . In: EATCS Monographs on Theoretical Computer Science, doi:10.1007/978-3-642-61345-6

    1988 doi

  6. [14]

    Smith & John G

    Tony C. Smith & John G. Cleary (1997): Probabilistic unification grammars. Australasian Natural Language Processing Summer Workshop

    1997

  7. [15]

    Warren, Konstantinos Sagonas, Juliana Freire, Prasad Rao, Baoqiu Cui, Ernie John- son, Luis de Castro, Rui F

    Theresa Swift, David S. Warren, Konstantinos Sagonas, Juliana Freire, Prasad Rao, Baoqiu Cui, Ernie John- son, Luis de Castro, Rui F. Marques, Diptikalyan Saha, Steve Dawson & Michael Kifer (2022): The XSB System Version 5.0,x. http://xsb.sourceforge.net. Latest release May 12, 2022

    2022

  8. [16]

    In: Inter- national Conference on Logic Programming, TPLP, doi:10.1017/S1471068423000169

    Yuheng Wang, Paul Fodor & Michael Kifer (2023): Knowledge Authoring for Rules and Actions. In: Inter- national Conference on Logic Programming, TPLP, doi:10.1017/S1471068423000169

    2023 doi

  9. [17]

    Younger (1967): Recognition and parsing of context-free languages in time n3

    Daniel H. Younger (1967): Recognition and parsing of context-free languages in time n3 . Information and Control 10(2), pp. 189–208, doi:10.1016/S0019-9958(67)80007-X

    1967 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.