{"id":"c7fa59d1-e51a-49c9-b1ad-1534cb4eb7db","arxiv_id":"2502.09212","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LP-LM uses Prolog DCG parsing and unification over a knowledge base to answer simple questions, so answers are grounded in stored facts rather than sampled from a language model.","lead":"LP-LM answers natural-language questions by parsing them into Prolog terms and looking up facts in a knowledge base, so every answer is grounded in stored facts rather than generated. It is a small demonstration that retrieval-style answers avoid hallucination, but only within the narrow grammar and knowledge base it is given.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'always reliable' / 'no hallucinations' claim is undercut by the paper's own example: inserting the false statement 'Fir trees can grow in human lungs' into the KB makes LP-LM output that falsehood, so the guarantee holds only for a truthful KB, not as advertised.","rationale":"Good-faith reading: the system is a small, transparent logic-programming QA prototype with public code, and its retrieval mechanism is sound for in-grammar questions over a trusted KB. The reader's CONDITIONAL verdict captures the main gap. My stress-test focuses on the specific way the paper's own evidence contradicts its central claim. The load-bearing assumption is not merely that the KB happens to be true; the system is designed to accept arbitrary user statements as facts, and the paper's own Table 1 uses a false statement as an example. That makes the 'no hallucinations' claim internally inconsistent, not just overbroad. The concrete test would settle whether the authors can honestly claim reliability in the ordinary sense. I recommend keeping the CONDITIONAL verdict: the core mechanism may be worth publishing as a prototype, but the advertised guarantee must be scoped with a truthful-KB assumption and a corrected comparison.","tokens_in":6923,"tokens_out":4909,"duration_ms":44519,"concrete_test":"Run the public repository's add_kb/query_kb on the exact fir-tree example from Table 1: add_kb('Fir trees can grow in human lungs'), then query_kb('What can grow in human lungs?'). Check whether LP-LM returns the false sentence. If it does, the advertised 'always reliable' claim fails under the standard factual definition of reliability; the paper would need to restrict the claim to KB-faithfulness and state the truthful-KB assumption explicitly before any no-hallucination claim is made.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LP-LM 'always produces answers that are reliable' (abstract) and, unlike LLMs, does not hallucinate (Section 3). The system's only source of answers is a user-populated KB of facts (Section 4). No mechanism validates the truth of inserted statements: add_kb dynamically inserts any parsed statement, including false ones. The paper's own Table 1, second row, demonstrates the consequence: after the user inserts 'Fir trees can grow in human lungs', LP-LM answers the query 'What can grow in human lungs?' with exactly that false sentence. Under the usual meaning of 'hallucination'—producing factually incorrect content—this is a hallucination, or at minimum an unreliable answer. The paper's claim is therefore only defensible under an unstated truthful-KB assumption, and even then only for sentences the hand-written PCFG parses correctly (a limitation acknowledged in Section 5). Furthermore, the comparison with LLMs is misleading: in that same row, GPT-4o's refusal to endorse the false premise is labelled a hallucination, inverting the standard notion. So the headline claim as written is unsupported; the supported claim is that LP-LM is faithful to its KB.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LP-LM, a question-answering system that combines a probabilistic context-free grammar (PCFG), Definite Clause Grammars (DCGs), and tabling in XSB Prolog to parse natural-language statements and questions, store statements as Prolog terms in a knowledge base (KB), and answer questions by unifying the parsed query term against KB facts. The authors claim that this architecture 'always produces answers that are reliable' and that LLMs 'hallucinate on even simple questions, unlike LP-LM.' They support these claims with four hand-selected example prompts comparing LP-LM with GPT-4o, GPT-4o mini, and Gemini, and with efficiency experiments on 12 synthetic grammars that they interpret as showing linear-time parsing for large grammars.","tokens_in":7161,"tokens_out":2862,"duration_ms":28553,"significance":"If the claims were properly scoped, the paper would provide a useful, transparent demonstration that a logic-programming-based retrieval mechanism can give machine-checkable grounding for answers to simple English questions. The strengths are real: the implementation is publicly available, the Prolog-based parsing and unification pipeline is described concretely, and the efficiency experiments include reproducible measurements. However, the headline claims are substantially overstated. The 'no hallucinations' guarantee is definitional once one assumes a truthful KB and a perfect hand-written grammar, and the paper's own Table 1 shows the system emitting a false statement as an answer. The LLM comparison also appears to mislabel correct, appropriately cautious LLM responses as hallucinations. The contribution is better characterized as a small, well-executed demonstration of KB-faithful retrieval than as a system that eliminates hallucination in any practically meaningful sense.","major_comments":[{"comment":"The abstract's claim that LP-LM 'always produces answers that are reliable' is contradicted by the paper's own example in Table 1, second row. After the false statement 'Fir trees can grow in human lungs' is inserted via add_kb, LP-LM answers 'What can grow in human lungs?' with exactly that false sentence. Since Section 1 defines hallucination as generating information that is 'factually incorrect,' this output is a hallucination under the paper's own definition. The no-hallucination property therefore holds only under an unstated assumption that the user-populated KB contains only true facts, and only for inputs the hand-written PCFG parses correctly. The authors must either revise the 'always reliable' and 'no hallucination' claims to state these conditions explicitly, or provide a mechanism that validates inserted statements against ground truth.","section":"Section 4, Table 1"},{"comment":"The comparison with LLMs is misleading and appears to invert the standard notion of hallucination. In the first three rows, GPT-4o, GPT-4o mini, and Gemini answer 'What causes temporary hearing loss?' with general causes such as loud noise, ear infections, and ototoxicity; these responses do not assert the user's specific premise 'Furosemide causes temporary hearing loss' and are at worst incomplete, not hallucinated. In the fourth row, GPT-4o mini explicitly says 'fir trees cannot grow in human lungs,' which is factually correct and appropriately contradicts the false premise; labeling this as a hallucination is not defensible. The authors need a transparent evaluation rubric that distinguishes unsupported claims, contextually appropriate corrections, and refusals, and they should report results over multiple runs and a broader set of queries before claiming that LLMs 'hallucinate on even simple questions.'","section":"Section 3, Table 1"},{"comment":"The linear-time claim is not established by the evidence presented. The experiments cover only 12 hand-constructed grammars with 3-5 input sentences of length 1-50 per grammar, with no statistical analysis or error bars, and the comparison is against NLTK's Viterbi parser described as a 'greedy heuristic' rather than a state-of-the-art baseline. The phrase 'runs in linear time in the size of input sentences' in the abstract is therefore too strong. The authors should either provide a formal complexity argument (for example, showing that with tabling and a fixed finite grammar the number of table entries is linear in sentence length) or weaken the claim to an observed empirical trend on the tested grammars.","section":"Section 4.3, Figures 2-5"}],"minor_comments":[{"comment":"The sentence 'The term run(X), where X is a variable, will be unified with a matching rule in the KB' should say 'matching fact' or 'matching term,' since the KB contains terms representing facts, not Prolog rules.","section":"Section 4.2"},{"comment":"The description of the x-axis in Figures 2-5 is vague: 'Higher numbered test cases represent sentences with longer lengths' would be clearer if each figure listed the actual sentence lengths or included a table of test-case parameters.","section":"Section 4.3"},{"comment":"The limitations paragraph acknowledges grammar coverage and the lack of reasoning capabilities, but it does not acknowledge the much more fundamental limitation that the system is only as reliable as the user-populated KB; this should be stated explicitly alongside the other limitations.","section":"Section 5"},{"comment":"The caption says 'The two Prolog versions are equivalent,' but the DCG rules and the hand-written Prolog clauses are not literally identical in all respects (e.g., argument order and the explicit difference lists); consider saying 'semantically equivalent' for clarity.","section":"Section 2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable short demonstration of a logic-programming-based QA Retrieval component, but the central claims need to be recalibrated before publication. The title and abstract promise 'no hallucinations' and 'always reliable,' yet the system's own example shows it will repeat false user-inserted statements. This is not merely a presentation issue; it changes the nature of the claimed contribution from a general hallucination-free QA system to a KB-faithful retrieval tool under strong assumptions. I would encourage the editors to request a revision where the authors honestly scope the guarantees, replace the anecdotal LLM comparison with a fair evaluation, and either prove or substantially soften the linear-time claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LP-LM is a legitimate, small integration of PCFG parsing and Prolog unification for KB-grounded question answering. The code is public, the mechanism is transparent, and for in-grammar questions against a trusted KB it does exactly what it says: answers are produced by unification with stored facts, so they are grounded by construction. That is real and worth acknowledging.\n\nWhat is new is modest: DCG parsing and Prolog unification are decades old, and the paper cites RAG and KALM as adjacent lines. The contribution is the concrete LP-LM system—add_kb/query_kb behavior, the grammar patterns, the XSB implementation—plus a small demonstration that this design yields exact retrieval on toy examples. The paper is clearly written and does not hide the main limitation (hand-written grammar, retrieval-only, no reasoning) in Section 5. Honest about scope in that respect.\n\nThe soft spots are real, and one is load-bearing for the advertised claim. The abstract says LP-LM 'always produces answers that are reliable' and Section 3 says LLMs hallucinate on even simple questions, unlike LP-LM. But the paper's own Table 1 includes the user inserting 'Fir trees can grow in human lungs' into the KB, and LP-LM then answers 'What can grow in human lungs?' with exactly that false sentence. Under any ordinary definition, that is an unreliable answer; under the standard definition, it is a hallucination from the KB's perspective. The only way the 'always reliable' claim works is with an unstated truthful-KB assumption and a grammar that parses the query correctly. The paper never states that assumption. Worse, in that same row GPT-4o's refusal to endorse the false premise is labeled a hallucination, which inverts the usual meaning. That framing is misleading, not just imprecise.\n\nThe linear-time claim is also weaker than advertised: it is an empirical observation on a few grammars per category, not a proven bound. The comparison with LLMs is four hand-picked single-run examples, no benchmark. These are real weaknesses, but they are the standard weaknesses of a short demo paper.\n\nBottom line: the architecture is sound for what it is, the code is public, and the paper is honest about the grammar limitation. What it is not is a demonstration of 'no hallucinations' in any general sense. This is exactly the kind of paper a serious editor should send to review with a request to fix the overclaims, add a truthful-KB assumption, and run a real benchmark. I would bring it to reading group as a cautionary example of how a grounded-retrieval guarantee gets oversold, and I would cite it for the Prolog-based grounding approach if I needed a concrete reference for that design pattern.","headline":"Solid little grounded-QA demo whose advertised 'no hallucination' guarantee only holds for a trusted KB; the paper's own false-fact example undercuts the headline claim.","tokens_in":7694,"tokens_out":2220,"would_cite":true,"duration_ms":19683,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LP-LM claims that parsing questions into Prolog terms and retrieving answers by unification against a knowledge base eliminates hallucination in question answering.","keywords":["logic programming","question answering","hallucination","definite clause grammar","probabilistic context-free grammar","knowledge base","Prolog","semantic parsing"],"falsifier":"Insert a false statement that the grammar accepts, such as 'Fir trees can grow in human lungs', then ask 'What can grow in human lungs?' and observe whether LP-LM answers 'fir trees'. If it does, the system's reliability is inherited entirely from the truthfulness of the knowledge base, and the unconditional no-hallucination claim fails in exactly the way the paper's comparison suggests LLMs fail.","tokens_in":6691,"feed_emoji":"✅","tokens_out":8209,"duration_ms":72788,"temperature":0.7,"pith_summary":"LP-LM is a question-answering system built on a simple design: every known fact and every question is converted into the same formal representation, so answering becomes a matching operation rather than a statistical guess. The paper claims this eliminates hallucination, because the answer is read off a knowledge base through Prolog unification and cannot be invented. The authors show that current LLMs give plausible but wrong answers on simple, context-dependent prompts, while LP-LM returns the stored fact. The payoff, if the claim holds, is a concrete demonstration that reliability in question answering does not require a larger model, only a representation that makes answers verifiable. The system is also efficient, with definite clause grammars and tabling giving linear-time parsing for grammars with many production rules.","feed_headline":"LP-LM answers questions from facts, not hallucinations","feed_subtitle":"It unifies each parsed question with stored facts, so every answer is grounded and auditable.","key_machinery":"The machinery is a probabilistic definite clause grammar pipeline running under tabling, with unification as the retrieval step. The PCFG supplies probabilities so that each sentence has a well-defined most probable parse; the DCG rules in Prolog make those productions executable, building both a constituency tree and a semantic term on the way. The semantic term always takes the verb's root form as its functor, which gives different surface sentences the same retrievable shape. Tabling memoizes repeated subgoals during the search for the best parse, which the paper credits for linear-time behaviour on grammars with many production rules. Unification then plays the role of query execution: a question term containing variables is matched against stored fact terms, and the resulting substitution is the answer.","core_discovery":"LP-LM's central claim is that hallucination can be avoided by construction rather than reduced by better statistics. A probabilistic context-free grammar assigns probabilities to English sentence structures; LP-LM parses an input with that grammar, selects the most probable constituency parse, and translates it into a Prolog term whose functor is the root form of the verb and whose arguments name the entities involved. Statements are inserted into a knowledge base as such terms, and questions are turned into terms with variables that are unified against the knowledge base: a wh-question gets its answer by substitution, and a yes/no question by checking whether the fully instantiated term is stored. Because the answer must be an exact match or a substitution instance of a stored fact, the system cannot produce a response unsupported by the knowledge base. The paper's experiments with well-known LLMs show that these models give plausible but wrong answers on the same simple prompts, while LP-LM returns the stored fact.","pith_inferences":["The 'always reliable' wording is conditional in a way the paper's framing obscures: LP-LM is only as trustworthy as the knowledge base it is given, because a false inserted statement is returned just as faithfully as a true one.","The LLM comparison is illustrative rather than a systematic benchmark; a natural next step is a large accuracy and coverage study that measures both how often LP-LM is correct and how many valid questions its grammar can even accept.","A hybrid architecture is an obvious extension: use an LLM to clean and normalise messy input into canonical statements, then let LP-LM perform the retrieval, which would widen the class of processable sentences while keeping the no-invention guarantee for the distilled facts.","The acceptance of false statements such as 'fir trees can grow in human lungs' suggests that real deployments need a separate curation or verification step, because the method itself cannot distinguish true from false entries."],"forward_implications":["For any question the grammar can parse and whose fact is in the knowledge base, LP-LM returns exactly the stored answer and cannot produce a conflicting invented one.","Because statements are inserted as explicit facts, the full set of possible answers is transparent, and every returned answer can be traced to the specific fact that produced it.","The linear-time parsing result means the no-hallucination behaviour does not come at the cost of scalability for grammars with many production rules.","Extending the grammar to cover more sentence patterns extends the reliable-answering guarantee to those patterns without changing the retrieval mechanism.","The comparison with LLMs implies that small, deterministic retrieval systems can outperform much larger generative models on simple relational question answering when the relevant facts are already available."],"supporting_citations":[{"why":"Defines hallucination in natural language generation and supplies the problem that LP-LM is designed to eliminate.","marker":"[9]"},{"why":"Introduces stochastic context-free grammars, the probability model LP-LM uses to select the most probable parse.","marker":"[11]"},{"why":"Presents XSB as an efficient deductive database engine, supporting the tabling-based implementation behind LP-LM's efficiency.","marker":"[12]"},{"why":"Documents the XSB Prolog system actually used to build LP-LM and run its DCG parsing, tabling, and unification.","marker":"[15]"},{"why":"Surveys retrieval-augmented generation, the main alternative approach LP-LM contrasts with for grounding answers in external knowledge.","marker":"[7]"}],"fun_headline_variants":["LP-LM: logic programming eliminates LLM hallucinations","No hallucinations: LP-LM answers from a Prolog knowledge base","LP-LM grounds answers in facts, not statistical guesses","Logic-based LLM: every answer is a verified fact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The no-hallucination guarantee assumes the knowledge base contains only true facts and the hand-written grammar correctly converts every question into the intended internal query; if a user inserts a false statement or the grammar mis-parses a question, LP-LM returns a wrong answer while still being described as reliable.","fun_headline_variants_meta":{"raw":{"variants":["LP-LM: logic programming eliminates LLM hallucinations","No hallucinations: LP-LM answers from a Prolog knowledge base","LP-LM grounds answers in facts, not statistical guesses","Logic-based LLM: every answer is a verified fact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2992,"prompt_tokens":890,"completion_tokens":2102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2034}},"tokens_in":506,"tokens_out":2102,"duration_ms":12874,"temperature":1.0,"reasoning_tokens":2034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:16:34.324382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Insert a false statement that the grammar accepts, such as 'Fir trees can grow in human lungs', then ask 'What can grow in human lungs?' and observe whether LP-LM answers 'fir trees'. If it does, the system's reliability is inherited entirely from the truthfulness of the knowledge base, and the unconditional no-hallucination claim fails in exactly the way the paper's comparison suggests LLMs fail.","supporting_citations":[{"cited_title":"Young (1990): The estimation of stochastic context-free grammars using the inside- outside algorithm","cited_arxiv_id":null,"evidence_quote":"Introduces stochastic context-free grammars, the probability model LP-LM uses to select the most probable parse."},{"cited_title":"Warren (1994): XSB as an Efficient Deductive Database Engine","cited_arxiv_id":null,"evidence_quote":"Presents XSB as an efficient deductive database engine, supporting the tabling-based implementation behind LP-LM's efficiency."},{"cited_title":"Warren, Konstantinos Sagonas, Juliana Freire, Prasad Rao, Baoqiu Cui, Ernie John- son, Luis de Castro, Rui F","cited_arxiv_id":null,"evidence_quote":"Documents the XSB Prolog system actually used to build LP-LM and run its DCG parsing, tabling, and unification."}],"review_version":1}