arxiv: 2601.15050 · v4 · submitted 2026-01-21 · 💻 cs.CL

Recognition: 2 theorem links

· Lean Theorem

Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore

Zhichao Yan , Yunxiao Zhao , Jiapu Wang , Jiaoyan Chen , Xiaoli Li , Ru Li , Jeff Z. Pan

Authors on Pith no claims yet

Pith reviewed 2026-05-16 12:23 UTC · model grok-4.3

classification 💻 cs.CL

keywords RAG evaluationglobal reasoningLogicScoreHorn rulesfactual accuracymulti-hop QALLM assessmentlogical integrity

0 comments

The pith

RAG systems achieve high factual accuracy but frequently fail to maintain global logical integrity in their responses.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current evaluation methods for Retrieval Augmented Generation systems focus narrowly on whether individual facts are correct, but this approach overlooks whether the overall reasoning holds together logically across a long answer. The paper introduces LogicScore, a new evaluation framework that uses Horn rules and backward verification to check three aspects of reasoning: whether the logic is complete, whether all premises are necessary, and whether the answer follows determinately. Experiments on multiple multi-hop question answering datasets show that top models like Gemini-3 Pro score over 92 percent on factual precision yet drop to around 35 percent on essentiality, a measure of non-redundant reasoning. This gap matters because it means models can produce answers that are factually supported but still contain logical gaps, redundant steps, or unclear connections, which undermines trust in AI-generated explanations. By shifting the focus to global reasoning quality, the work pushes for LLM development that values coherent deduction alongside accurate retrieval.

Core claim

The central discovery is that RAG models often excel at local factual accuracy but exhibit poor global reasoning integrity, as measured by LogicScore. Grounded in Horn rules, the evaluation uses backward verification to assess completeness of logical deduction, essentiality through non-redundancy, and determinateness of answer entailment. Across datasets like HotpotQA, MusiQue, and 2WikiMultiHopQA and models including GPT-5, Gemini-3 Pro, and LLaMA3, high precision scores contrast with low scores on the reasoning dimensions, revealing a capability gap that existing fact-checking methods miss.

What carries the argument

LogicScore, which applies backward verification over Horn rules to measure completeness, essentiality, and determinateness in reasoning chains.

If this is right

Evaluation benchmarks for RAG must incorporate global reasoning checks alongside factual metrics to prevent over-optimization for isolated facts.
LLM training and prompting strategies should target improvements in the three dimensions to reduce logical gaps and redundant premises in long-form outputs.
Task-specific fine-tuned models may exhibit different reasoning integrity profiles than general-purpose LLMs, requiring dimension-specific diagnostics.
The framework provides a structured way to diagnose specific failures such as ambiguous links or unaddressed gaps in multi-hop reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Future RAG architectures could integrate explicit logic enforcement modules that align generation steps with Horn-rule consistency during answer construction.
Applying LogicScore to domains beyond question answering, such as scientific summarization or policy analysis, would likely surface analogous reasoning shortfalls.
The observed gap suggests that scaling factual retrieval alone will not close logical deficiencies unless paired with targeted reasoning objectives.

Load-bearing premise

The assumption that Horn rules combined with backward verification on completeness, essentiality, and determinateness fully captures global reasoning integrity without missing other logical flaws.

What would settle it

A controlled human study in which answers scoring high on LogicScore receive significantly higher ratings for logical coherence, absence of gaps, and lack of redundancy than answers scoring low, with the difference persisting after controlling for factual accuracy.

Figures

Figures reproduced from arXiv: 2601.15050 by Jeff Z. Pan, Jiaoyan Chen, Jiapu Wang, Ru Li, Xiaoli Li, Yunxiao Zhao, Zhichao Yan.

**Figure 1.** Figure 1: Motivation of LOGICSCORE. Traditional methods (pink area) yield high Factual Quality by focusing on local evidence, overlooking reasoning flaws. Our framework (blue area) evaluates Logic Quality via Completeness, Conciseness, and Determinateness, exposing logical deficits in the long-form answer. in grounding model outputs in reliable evidence. Attributed Question Answering (AQA) [Bohnet et al., 2022] en… view at source ↗

**Figure 2.** Figure 2: Overview of the LOGICSCORE evaluation framework, consisting of three phases: (1) Answer Generation, where the model produces a Long-form Answer (LA) and a Short Answer (SA) based on the question and top-k documents; (2) Logic Transformation, which decomposes the LA into a set of atomic propositions (P) structured as Horn clauses; and (3) Logic Evaluation, which assesses the reasoning quality across three … view at source ↗

**Figure 4.** Figure 4: Impact of reasoning depth on logic quality metrics. The [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 3.** Figure 3: Case study. We observe three logic error types when prompting LLMs to generate attributed long-form answers: [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 5.** Figure 5: scaling Influence in multi-hops. destroyer class to the specific acronym requested, the narrative deviates into a generalization about “destroyer classes” before making an unjustified leap to “Navy SEALs”. This discontinuity fractures the logical path, preventing a coherent derivation of the “SEAL” definition. Similarly, DeepSeek-R1, despite employing chain-ofthought reinforcement learning, displays flaw… view at source ↗

**Figure 6.** Figure 6: Prompt for Long-form answer generation [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

**Figure 7.** Figure 7: Prompt for Logic Transformation [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

**Figure 8.** Figure 8: Prompt for triple extraction in Completeness evaluation [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

**Figure 9.** Figure 9: Prompt for entity detection in Completeness evaluation [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

**Figure 10.** Figure 10: Prompt for short answer generation in Determinateness evaluation [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

**Figure 11.** Figure 11: Screen shot of logic transformation evaluation system. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

**Figure 12.** Figure 12: Screen shot of logic quality evaluation system. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

read the original abstract

Current evaluation methods for Retrieval Augmented Generation (RAG) suffer from \textit{factual myopia}: they relentlessly emphasize factual accuracy yet neglect global logical integrity in long-form answer generation. This drives models to force unnatural connections, producing factually grounded yet logically incoherent responses with unaddressed gaps, ambiguous links, or redundant premises. To mitigate this, we present \textsc{LogicScore}, shifting from local, fact-by-fact assessment to rigorous global reasoning scrutiny. Grounded in Horn Rules, our approach integrates a backward verification mechanism to systematically evaluate three key reasoning dimensions: \textit{Completeness} (logically sound deduction), \textit{Essentiality} (non-redundancy), and \textit{Determinateness} (consistent answer entailment). Extensive experiments across three multi-hop QA datasets (HotpotQA, MusiQue, and 2WikiMultiHopQA) and over 20 LLMs (including GPT-5, Gemini-3-Pro, LLaMA3, and task-specific tuned models) reveal a critical capability gap: leading models often achieve high factual accuracy (e.g., 92.85\% precision for Gemini-3 Pro) but struggle with global reasoning quality (e.g., 35.11\% Essentiality for Gemini-3 Pro). Our work establishes a robust standard for logical evaluation, highlighting the need to prioritize reasoning coherence alongside factual grounding in LLM development.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

LogicScore flags a real factual-vs-reasoning gap in RAG outputs using Horn-rule backward checks, but the extraction step from text lacks the validation needed to trust the numbers.

read the letter

The paper's main move is to stop treating RAG evaluation as a fact checklist and instead score whether the whole answer hangs together logically. LogicScore converts generated text into Horn rules, then runs backward verification on three axes: completeness of the deduction, essentiality of the premises, and determinateness of the conclusion. The experiments on HotpotQA, MusiQue, and 2WikiMultiHopQA across more than twenty models, including GPT-5 and Gemini-3 Pro, show the expected split—Gemini-3 Pro hits 92.85% fact precision yet only 35.11% essentiality. That pattern is the clearest new signal here, and the breadth of models and datasets makes the gap look consistent rather than anecdotal. The framework itself is a clean step past local fact metrics, and the three dimensions line up with the problem the authors describe. The soft spot is exactly where the stress-test note points: everything rests on turning free-form answers into accurate Horn rules. If premise extraction drops implicit steps, mangles quantifiers, or forces non-Horn content into the format, the low essentiality scores could be artifacts. The abstract gives no error rates, no human agreement numbers on the extracted rules, and no comparison to other formalisms, so the mapping from text to score stays unanchored. A reader who works on RAG benchmarks or logical consistency checks would get value from the idea and the reported gap, even if the current numbers need more grounding. The paper deserves a serious referee because the underlying problem is real and the proposed fix is concrete; the revision would mainly need to document and test the extraction pipeline so the scores can be reproduced and trusted.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces LogicScore, a framework for evaluating global reasoning integrity in RAG systems beyond factual accuracy. Grounded in Horn rules and a backward verification process, it measures three dimensions—Completeness (sound deduction), Essentiality (non-redundancy), and Determinateness (consistent entailment)—across LLM-generated answers. Experiments on HotpotQA, MusiQue, and 2WikiMultiHopQA with over 20 models (including GPT-5, Gemini-3 Pro, LLaMA3) show high factual precision (e.g., 92.85% for Gemini-3 Pro) but substantially lower reasoning scores (e.g., 35.11% Essentiality for the same model), highlighting a gap in current evaluation practices.

Significance. If the Horn-rule extraction and backward verification prove reliable, LogicScore could provide a valuable new standard for assessing logical coherence in long-form RAG outputs, pushing LLM development toward responses that are not only factually grounded but also free of gaps, redundancies, and ambiguities. The empirical demonstration of the factual-vs-reasoning disconnect across multiple datasets and models supplies concrete evidence that could influence future benchmarks and training objectives.

major comments (3)

[§3] §3 (LogicScore definition): The backward verification mechanism depends on accurate extraction of Horn rules from free-form LLM text, yet no details are provided on handling implicit premises, quantifier scope, or non-strict Horn forms. This is load-bearing for the central claim, as systematic extraction errors could artifactually depress Essentiality scores rather than reflect genuine logical gaps.
[§5] §5 (Experiments): No human validation, inter-annotator agreement, or error analysis is reported for the extracted rules or the three dimension scores. Without this, the reported gap (high factual accuracy vs. low Essentiality/Determinateness) cannot be confidently attributed to reasoning deficiencies rather than formalization artifacts.
[§4.2] §4.2 (Evaluation dimensions): The mapping from backward verification to 'Essentiality' (non-redundancy) assumes all premises are explicitly derivable; if the rule formalizer omits contextually implied premises, the metric may penalize valid but concise reasoning, weakening the interpretation of the 35.11% score.

minor comments (2)

[Table 1] Table 1 and §5.1: The list of 20+ LLMs is incomplete in the main text; an appendix table enumerating all models, their sizes, and tuning status would improve reproducibility.
[§3.1] Notation in §3.1: The symbols for the three dimensions (C, E, D) are introduced without an explicit summary table; adding one would aid readers in tracking the formulas.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thoughtful and detailed review of our manuscript. The comments have helped us identify areas for improvement in the presentation and validation of LogicScore. Below, we provide point-by-point responses to the major comments and indicate the revisions we intend to make in the updated version.

read point-by-point responses

Referee: [§3] §3 (LogicScore definition): The backward verification mechanism depends on accurate extraction of Horn rules from free-form LLM text, yet no details are provided on handling implicit premises, quantifier scope, or non-strict Horn forms. This is load-bearing for the central claim, as systematic extraction errors could artifactually depress Essentiality scores rather than reflect genuine logical gaps.

Authors: We thank the referee for pointing out the need for greater transparency in the rule extraction process. The manuscript provides an overview of the backward verification but indeed lacks specifics on edge cases such as implicit premises and quantifier handling. In the revised manuscript, we will expand Section 3 with a new subsection detailing the Horn rule extraction pipeline. This will include: (1) a description of how implicit premises are inferred using the retrieved context and LLM prompting with specific instructions; (2) handling of quantifier scope by restricting to universal quantification in Horn clauses; and (3) conversion of non-strict forms to strict Horn rules via logical normalization. We will also include pseudocode and examples to illustrate the process. These additions will allow readers to better assess potential extraction errors and strengthen the reliability of the Essentiality scores. revision: yes
Referee: [§5] §5 (Experiments): No human validation, inter-annotator agreement, or error analysis is reported for the extracted rules or the three dimension scores. Without this, the reported gap (high factual accuracy vs. low Essentiality/Determinateness) cannot be confidently attributed to reasoning deficiencies rather than formalization artifacts.

Authors: We agree that the lack of human validation is a limitation in the current experimental setup. To address this, we will perform a human study on a randomly sampled subset of 300 instances (100 per dataset). Two independent annotators will evaluate the accuracy of extracted Horn rules and the validity of the three dimension scores, with inter-annotator agreement measured using Cohen's kappa. Additionally, we will include a detailed error analysis in the revised Section 5, categorizing discrepancies into extraction artifacts versus actual reasoning deficiencies. This will provide evidence that the observed gaps (e.g., high factual accuracy but low Essentiality) are primarily due to reasoning issues rather than formalization problems. revision: yes
Referee: [§4.2] §4.2 (Evaluation dimensions): The mapping from backward verification to 'Essentiality' (non-redundancy) assumes all premises are explicitly derivable; if the rule formalizer omits contextually implied premises, the metric may penalize valid but concise reasoning, weakening the interpretation of the 35.11% score.

Authors: This is a valid concern regarding the interpretation of Essentiality. Our current formulation intentionally focuses on explicit premises to quantify non-redundancy in a strict, verifiable manner, which aligns with the goal of detecting unnecessary statements in the generated answer. However, we recognize that this may undervalue concise reasoning that relies on implied premises. In the revision, we will clarify this assumption in Section 4.2 and introduce an optional 'context-aware' variant of Essentiality that incorporates implied premises from the retrieval context. We will also add a discussion of this limitation and re-analyze the 35.11% score under the new variant to provide a more nuanced view. revision: partial

Circularity Check

0 steps flagged

LogicScore framework defined independently of results; no circular reduction

full rationale

The paper introduces LogicScore as a new evaluation method grounded in Horn rules with an explicit backward verification procedure to measure completeness, essentiality, and determinateness. This definition precedes and is independent of the reported experiments on HotpotQA, MusiQue, and 2WikiMultiHopQA. No equations or steps in the abstract reduce the three dimensions to fitted parameters or self-citations; the reported scores (e.g., 35.11% Essentiality) are presented as outputs of applying the pre-defined method rather than inputs that define it. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review provides insufficient detail to enumerate free parameters or invented entities; the main assumption is the suitability of Horn rules for modeling RAG reasoning.

axioms (1)

domain assumption Horn rules can model the reasoning structure in RAG-generated answers for evaluation purposes
The approach is explicitly grounded in Horn Rules as stated in the abstract.

pith-pipeline@v0.9.0 · 5571 in / 1136 out tokens · 24368 ms · 2026-05-16T12:23:10.301784+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/LogicAsFunctionalEquation.lean SatisfiesLawsOfLogic unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Grounded in Horn Rules, our approach integrates a backward verification mechanism to systematically evaluate three key reasoning dimensions: Completeness, Conciseness, and Determinateness.
IndisputableMonolith/Foundation/ArithmeticFromLogic.lean LogicNat unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

we adopt Horn Rules, which are tractable, interpretable logical structures that formalize natural language reasoning into deterministic proof trees

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

[1]

**Read all provided documents in full.**

work page
[2]

**Identify relevant sentences** that help answer the question

work page
[3]

[DocumentID]<S#>

**Construct a logical reasoning passage** before presenting the final conclusion, using citation tokens in the form “[DocumentID]<S#>” immediately after the statement they support

work page
[4]

**Avoid including any information not supported by the provided documents.**

work page
[5]

**Don’t copy the original sentence, but paraphrase it based on the meaning.**

work page
[6]

**Use as many short sentences as possible while retaining the key information.**

work page
[7]

**The final conclusion should appear strictly after the reasoning section.** #Steps

work page
[8]

Parse the documents and break them into discrete facts based on ‘<S#>‘ markers

work page
[9]

Identify which sentences are relevant to answering the question

work page
[10]

Use citation tokens to indicate exact supporting sentences

Write a coherent reasoning chain that explains step-by-step how the evidence leads to the conclusion. Use citation tokens to indicate exact supporting sentences

work page
[11]

Always place every step between<STATEMENT>and</STATEMENT>, don’t copy the original sentence, but paraphrase it based on the meaning

work page
[12]

Present the final well-formed answer/conclusion **after** the reasoning section. #Output Format The response must be structured as follows in plain text: **Reasoning:** [A multi-sentence explanation in logical order, with citations like ‘[1]<S1>‘ and ‘[2]<S3>or [1]<S1><S2>‘ immediately after referenced facts.] **Answer:** [A short, concise direct answer t...

work page 1928
[13]

**Read and comprehend** the full natural reasoning process provided and question

work page
[14]

Assign each proposition a short label or variable (e.g., ‘P1‘, ‘P2‘)

**Break down** the text into minimal distinct logical statements or propositions. Assign each proposition a short label or variable (e.g., ‘P1‘, ‘P2‘)

work page
[15]

**Determine relationships** between propositions: - Use∧for conditions that must all be true (logical AND)

work page
[16]

**Reconstruct** the reasoning as a single logical expression combining∧, preserving original precedence and grouping with parentheses as needed

work page
[17]

**Double-check** that the reconstructed expression faithfully mirrors the intent of the natural reasoning pro cess

work page
[18]

propositions

**Ensure** the logical expression represents by P* rather than a statement. #Output Format Output should be in **JSON** with the following structure: { “propositions”:{ “P1”: “[first proposition in plain language]”, “P2”: “[second proposition in plain language]”, “...”: “...” }, “logical expression”: “P1∧P2 ...” } - Use UTF-8 characters for∧. - Ensure par...

work page
[19]

**Read and Understand** the given question carefully

work page
[20]

**Identify Potential Entities** by scanning for proper nouns, temporal expressions, locations, etc

work page
[21]

**Determine Entity Types** (e.g., Person, Organization, Location, Date, Time, Event, Product)

work page
[22]

**List the Entities** with their corresponding types in a structured format

work page
[23]

entities

Ensure the output only contains the extracted entities and their types, no additional text. #Output Format Output the result as a JSON object without code block formatting, containing: - “entities”: an array of objects, each with: - “text”: the exact entity text from the question - “type”: the entity type Example: Input: “Where did Barack Obama give his N...

work page 2009
[24]

**Understand the sentence**: Identify entities, relationships, and values explicitly or implicitly mentioned

work page
[25]

**Reasoning**: Break down how each relationship is identified, including disambiguation of terms

work page
[26]

(subject, predicate, object)

**Triple extraction**: Formulate each triple in the “(subject, predicate, object)” structure

work page
[27]

subject”: the entity being described. - “predicate

**Validate**: Ensure all triples are supported by the provided sentence and do not introduce facts not present. #Output Format Respond in JSON format, with an array of objects, each containing: - “subject”: the entity being described. - “predicate”: the relationship between subject and object. - “object”: the value or entity connected to the subject. Exam...

work page
[28]

Read and fully understand the natural question provided

work page
[29]

Break down the problem or question into smaller parts

work page
[30]

Produce detailed, step-by-step reasoning explaining thought process and relevant facts

work page
[31]

Ensure reasoning flows logically toward a conclusion

work page
[32]

State the final answer only after the reasoning is complete. #Output Format Provide the output in the following structure: - Reasoning: [Multi-sentence logical explanation of the path to answer, including any inference steps and key facts] - Answer: [Single sentence or short paragraph conclusion that directly addresses the natural question] #Examples Exam...

work page