Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that combining chain-of-thought prompting with variable entity type information produces the most accurate and readable natural-language explanations of knowledge-graph rules among the prompting strategies it tested, and…

desk verdict Useful first look at LLM verbalization of KG rules, but the headline comparison rests on fragile human evaluation and at least one bad table entry. read the letter →

arxiv 2507.23740 v1 pith:3XQGCFX4 submitted 2025-07-31 cs.CL cs.AIcs.LG

Nasim Shirvani-Mahdavi , Devin Wingfield , Amin Ghasemi , Chengkai Li This is my paper
classification cs.CLcs.AIcs.LG
keywords knowledgegraphslogicalrulesnaturallanguageexplanationlargemodelschain-of-thoughtpromptinghumanevaluationAMIEFreebase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can turn logical rules mined from knowledge graphs—abstract Horn clauses such as “if a person is the mother of a child, then the person is the parent of that child”—into natural-language explanations that ordinary users can understand. It tests zero-shot, few-shot, variable-entity-type, and chain-of-thought prompting strategies on rules extracted by AMIE from FB15k-237, FB-CVT-REV, and FB+CVT-REV. Human annotators rated the outputs on correctness, clarity, and hallucination, and the paper also tested LLM-as-a-judge as a scalable alternative to human scoring. The central finding is that combining chain-of-thought prompting with variable entity type information yields the most accurate and readable explanations, with Gemini 2.0 Flash performing best among the three models evaluated. If correct, this gives a concrete prompt recipe for explaining KG rules and a baseline for scaling evaluation through automated judges.

What carries the argument

The carrying mechanism is the logical rule itself—an AMIE-mined Horn clause with a body of atoms and a head atom, written $B_1 \land \dots \land B_n \Rightarrow H$—plus the prompt design that presents the rule to the LLM. The decisive added components are the list of entity types appended for each variable (derived from the near-functional mapping between edge types and the types of subjects and objects they connect) and a five-step chain-of-thought instruction that parses the rule, selects the contextually relevant type for each variable, interprets each atom with those types, synthesizes the overall implication, and finally writes a concise explanation in natural language.

What would settle it

A fresh annotation of a random sample of rules, stratified by atom count and relation type, that fails to show an advantage for chain-of-thought with variable types over the zero-shot baseline would falsify the central ranking; likewise, if a different aggregation rule such as the median of all three annotators reversed the order between strategies on the same 100 rules, the reported comparison would not be robust.

Watch

Extended reading notes

Core claim

The paper's claim is that large language models can generate faithful, clear natural-language explanations for logical rules mined from knowledge graphs, and that the specific combination of chain-of-thought prompting plus variable entity type hints outperforms the alternatives. The authors support this with a three-phase human evaluation: phase one compares zero-shot and few-shot prompting on FB15k-237 rules; phase two adds variable entity types for rules from the large-scale FB-CVT-REV and FB+CVT-REV datasets; phase three applies chain-of-thought prompting across GPT-3.5 Turbo, GPT-4o Mini, and Gemini 2.0 Flash. On correctness and clarity scales of 1 to 5, the variable-type prompt consistently outperforms the zero-shot baseline, and the CoT prompt further improves GPT-3.5 Turbo's scores. Gemini 2.0 Flash achieves the highest overall correctness and clarity, and hallucination and omission counts remain low across all settings, indicating that the remaining errors concentrate in rules with concatenated relations and mediator nodes.

Load-bearing premise

The findings depend on the sampled rules—the 100 highest-head-coverage rules per phase—being representative of the rules people actually need explained, and on the majority-vote aggregation of three annotators (with one dissenting annotator's scores discarded) being an unbiased measure of explanation quality.

Editorial extensions

If this is right

  • Knowledge-graph tools can adopt a ready-made prompt recipe—chain-of-thought plus variable entity types—to turn mined rules into readable text without per-rule templates.
  • The evaluation metrics and annotation protocol give a replicable standard for measuring explanation correctness, clarity, and hallucination in this task.
  • Since the same rule set produced better explanations under CoT prompting, the reported gains are attributable to the prompt structure rather than to model choice alone, though model choice also matters.
  • The moderate judge-annotator correlation suggests that LLM-as-a-judge can partially substitute for human evaluation, enabling larger-scale screening of generated explanations.
  • Hallucination counts stay low even as rule complexity rises, so the main remaining obstacle is explaining concatenated and mediator-node rules clearly rather than preventing fabrication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The prompt recipe likely transfers to other symbolic-output settings—proof traces, plan descriptions, or rule-based systems beyond AMIE—where the crucial ingredient is having the same kind of entity-type annotation available from the underlying data.
  • A natural next test the paper leaves implicit: use the highest-rated (rule, explanation) pairs as pseudo-ground truth, fine-tune smaller open models, and compare their output quality and cost against the best prompting strategy.
  • The self-family judge bias reported in the literature suggests that the moderate judge-annotator correlations could be improved by blending judges from different model families or calibrating judge scores against a small human-annotated set.
  • If the goal is deployment, the most informative follow-up experiment is measuring whether these explanations actually help a human worker verify or reject inferred facts, which would test the downstream value the paper motivates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes and evaluates LLM-based natural language explanations for logical rules mined by the AMIE algorithm from Freebase-derived knowledge graphs (FB15k-237, FB-CVT-REV, and FB+CVT-REV). It compares zero-shot, few-shot, variable-entity-type, and chain-of-thought prompting strategies using GPT-3.5 Turbo, GPT-4o Mini, and Gemini 2.0 Flash, with human annotation of correctness, clarity, missed/hallucinated entities and relations, rule logicalness, and GPT-2 perplexity. It also reports an exploratory LLM-as-a-judge analysis. The headline claim is that chain-of-thought prompting combined with variable type information yields the most accurate and readable explanations.

Significance. If the headline claim were supported, this would be a useful first study of LLM-based explanation of KG rules, with practical guidance on prompting. The paper's strengths include public release of scripts, prompts, rules, explanations, and annotated data; a concrete human evaluation protocol; and an honest exploratory treatment of LLM-as-a-judge with both cross-family judge settings. However, the comparative evidence for the headline claim is currently thin: the decisive differences are small aggregate means without uncertainty quantification, and at least one table entry appears internally inconsistent.

major comments (4)
  1. [Section 4, Tables 3 and 4; Section 5] The central comparative claim that CoT plus variable types is the most accurate and readable strategy rests on mean differences of 0.07 in correctness and clarity between the GPT-3.5 Phase 2 variable-type scores (4.21/4.19) and the Phase 3 CoT+type scores (4.28/4.26). The paper reports no paired significance test, confidence interval, effect size, or inter-annotator agreement, so the claim of 'significant' improvement in Section 3.1, Phase 2, is unsupported. Please report per-rule paired differences with a suitable test (e.g., Wilcoxon signed-rank), variance, and annotation reliability (e.g., Krippendorff's alpha or Cohen's kappa) for all phases.
  2. [Section 4, Phase 1] The aggregation rule drops the dissenting annotator's metric scores after majority-vote selection of the preferred explanation ('we only averaged the measures provided by annotators 1 and 2'). If annotators differ systematically in severity, this deletion can bias the chosen prompt's average and distort the zero-shot versus few-shot comparison in Table 2. Please analyze all three annotators' scores, for example with a mixed-effects model or by reporting per-annotator means, or justify the deletion explicitly.
  3. [Table 3, '2 atoms' row, variable-type prompt] The reported m_ent=0.31 and m_rel=0.41 contradict the text's statement that both explanation types have minimal missing entities and relations, and they are inconsistent with the neighboring zero-shot row (0.03/0.04) and the 3-atoms variable-type row (0.07/0.08). This suggests a data or rendering error; it must be corrected or explicitly explained, and the affected aggregates recomputed, because the inconsistency casts doubt on the reliability of the table's other entries.
  4. [Section 4, Phase 2 and Phase 3 sample selection] The evaluation uses only the 100 rules with the highest head coverage in each phase, yet Table 3 shows that rule complexity (2 vs. 3 atoms, concatenated relations) strongly affects scores. Without evidence that this subset is representative of the rules KG users need explained, the external validity of the comparative claim is limited. Please justify the selection or report results on a broader or stratified sample.
minor comments (4)
  1. [Abstract] The abstract contains a malformed URL with duplicated braces: 'https://github.com/idirlab/KGRule2NL}{https://github.com/idirlab/KGRule2NL' should be a single URL.
  2. [Section 4, Phase 3] Hallucination and missed-entity metrics are dropped in Phase 3 without reporting their values for the new models and CoT prompts; please provide these numbers or justify their omission with the Phase 3 data.
  3. [Section 3.1, Phase 2] The phrase 'there is almost a function that maps from the edge type to a type' is informal; please define the intended mapping precisely or replace it with a formal statement.
  4. [Tables 2 and 3] The abbreviations m_ent, m_rel, h_ent, h_rel, correct, and logical are not defined in the table captions; please add a definition line in each caption for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparative claim rests on independent human evaluation of LLM-generated explanations, not on inputs that are re-labeled as predictions.

full rationale

The paper's derivation chain is empirical rather than definitional. Logical rules are mined by AMIE from public benchmark datasets; the prompt variants (zero-shot, few-shot, variable entity types, and chain-of-thought) are distinct experimental inputs; and the headline claim that CoT plus variable type information yields the most accurate and readable explanations is judged by three human annotators scoring correctness, clarity, and hallucination, with results reported in Tables 2-4. No explained quantity is defined in terms of the predicted output, and no fitted parameter is renamed as a prediction. The LLM-as-a-judge analysis is explicitly exploratory and is correlated against human scores rather than replacing them, so it does not feed back into the main result and cannot make it circular. The paper cites the authors' own prior datasets and the 'almost a function' type-inference observation from [18], but those are externally checkable data properties whose use in prompt design is independently tested by human evaluation; thus the self-citations are not load-bearing in a circular way. The absence of significance testing and inter-annotator agreement statistics concerns evidence quality or correctness risk, not circularity. No step in the paper reduces to its own input by construction, so no circular step is identified.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper contributes an empirical evaluation rather than a derivation, so the ledger lists the hand-chosen experimental thresholds, the reliance on AMIE's rule quality metrics, and the dataset and type-inference assumptions from prior work. No new entities or fitted physical parameters appear.

free parameters (5)
  • AMIE minimum head coverage = 0.1
    Hand-chosen threshold for rule mining; affects which rules are included in the study and therefore the difficulty of the explanation task.
  • AMIE minimum standard confidence = 0.1
    Hand-chosen threshold for rule mining; affects rule quality and representativeness.
  • AMIE maximum number of atoms = 3
    Limits rule complexity; the paper notes rules with 3 atoms are harder to explain, so this choice bounds the evaluation scope.
  • Number of example pairs in few-shot/CoT prompts = 2
    Prompt design choice; could influence the explanation quality and the comparison between strategies.
  • Rule subset size per evaluation phase = 100
    The top 100 rules by head coverage are selected for annotation; this selection affects the distribution of rule complexity and the generalizability of the results.
assumptions (3)
  • domain assumption AMIE 3.5.1's mined rules with its confidence and head coverage metrics are meaningful logical rules for KG completion.
    The paper relies on AMIE's output as ground truth rules without independent verification; this is standard practice but an assumption about rule quality.
  • domain assumption The Freebase-derived datasets FB-CVT-REV and FB+CVT-REV correctly eliminate data leakage and support the 'almost a function' type inference for variable entities.
    The variable entity type lists come from the authors' prior work [18]; if those datasets or type mappings are flawed, the variable-type prompting results are affected.
  • domain assumption Human annotations of correctness and clarity on a 1-5 scale are valid measures of explanation quality as defined.
    The evaluation defines correctness as inclusion of all rule components in exact logical order; this definition rewards literal reproduction and may penalize valid paraphrases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs." pith.science (2026). https://pith.science/paper/3XQGCFX4

@misc{pith2026250723740,
  author       = {Pith},
  title        = {Pith review of: Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XQGCFX4}},
  note         = {Machine review of arXiv:2507.23740}
}
read the original abstract

Knowledge graphs (KGs) often contain sufficient information to support the inference of new facts. Identifying logical rules not only improves the completeness of a knowledge graph but also enables the detection of potential errors, reveals subtle data patterns, and enhances the overall capacity for reasoning and interpretation. However, the complexity of such rules, combined with the unique labeling conventions of each KG, can make them difficult for humans to understand. In this paper, we explore the potential of large language models to generate natural language explanations for logical rules. Specifically, we extract logical rules using the AMIE 3.5.1 rule discovery algorithm from the benchmark dataset FB15k-237 and two large-scale datasets, FB-CVT-REV and FB+CVT-REV. We examine various prompting strategies, including zero- and few-shot prompting, including variable entity types, and chain-of-thought reasoning. We conduct a comprehensive human evaluation of the generated explanations based on correctness, clarity, and hallucination, and also assess the use of large language models as automatic judges. Our results demonstrate promising performance in terms of explanation correctness and clarity, although several challenges remain for future research. All scripts and data used in this study are publicly available at https://github.com/idirlab/KGRule2NL}{https://github.com/idirlab/KGRule2NL.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Rule2Text generates and evaluates natural language explanations of knowledge graph rules, finding that chain-of-thought prompting with entity types works best and that fine-tuning Zephyr on LLM-built ground truth shar...

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv:2311.02765 (2023) Rule2Text: NL Explanation of Logical Rules in KGs 9

    Æsøy, K., Ozaki, A.: Rule Learning as Machine Translation using the Atomic Knowledge Bank. arXiv:2311.02765 (2023) Rule2Text: NL Explanation of Logical Rules in KGs 9

    arXiv 2023

  2. [2]

    In: IJCAI

    Betz, P., Galárraga, L., Ott, S., Meilicke, C., Suchanek, F., Stuckenschmidt, H.: PyClause-Simple and efficient rule handling for knowledge graphs. In: IJCAI. pp. 8610–8613 (2023)

    work page 2023

  3. [3]

    In: SIGMOD

    Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collabo- ratively created graph database for structuring human knowledge. In: SIGMOD. pp. 1247–1250 (2008)

    work page 2008

  4. [4]

    In: NeurIPS

    Bordes, A., Usunier, N., Garcia-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: NeurIPS. pp. 2787–2795 (2013)

    work page 2013

  5. [5]

    arXiv:2004.14579 (2020)

    Chen, Z., Chen, W., Zha, H., Zhou, X., Zhang, Y., Sundaresan, S., Wang, W.Y.: Logic2Text: High-fidelity natural language generation from logical forms. arXiv:2004.14579 (2020)

    arXiv 2020

  6. [6]

    arXiv preprint arXiv:2002.05867 (2020)

    Clark, P., Tafjord, O., Richardson, K.: Transformers as soft reasoners over lan- guage. arXiv preprint arXiv:2002.05867 (2020)

    arXiv 2020

  7. [7]

    VLDB 24(6), 707–730 (Dec 2015)

    Galárraga, L., Teflioudi, C., Hose, K., Suchanek, F.M.: Fast Rule Mining in Onto- logical Knowledge Bases with AMIE++. VLDB 24(6), 707–730 (Dec 2015)

    work page 2015

  8. [8]

    Galárraga, L.A., Teflioudi, C., Hose, K., Suchanek, F.: AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In: WWW. pp. 413–422 (2013)

    work page 2013

Show all 23 references
  1. [9]

    TNNLS33(2), 494–514 (2021)

    Ji, S., Pan, S., Cambria, E., Marttinen, P., Philip, S.Y.: A survey on kgs: Repre- sentation, acquisition, and applications. TNNLS33(2), 494–514 (2021)

    2021

  2. [10]

    prompt engineering vs

    Kermani, A., Perez-Rosas, V., Metsis, V.: A systematic evaluation of llm strategies for mental health text analysis: Fine-tuning vs. prompt engineering vs. rag. In: The 10th Workshop on CLPsych. p. 172 (2025)

    2025

  3. [11]

    arXiv preprint arXiv:2504.02794 (2025)

    Kiafar, B., Ravva, P.U., Joy, A.A., Daher, S., Barmaki, R.L.: Mena: Multi- modal epistemic network analysis for visualizing competencies and emotions. arXiv preprint arXiv:2504.02794 (2025)

    2025 arXiv

  4. [12]

    VLDB33(1), 131–161 (2024)

    Meilicke, C., Chekol, M.W., Betz, P., Fink, M., Stuckeschmidt, H.: Anytime bottom-up rule learning for large-scale KGC. VLDB33(1), 131–161 (2024)

    2024

  5. [13]

    VLDS884(6), 15–20 (2012)

    Nakashole, N., Sozio, M., Suchanek, F.M., Theobald, M.: Query-time reasoning in uncertain RDF KBs with soft and hard rules. VLDS884(6), 15–20 (2012)

    2012

  6. [14]

    NeurIPs37, 68772–68802 (2024)

    Panickssery, A., Bowman, S., Feng, S.: LLM evaluators recognize and favor their own generations. NeurIPs37, 68772–68802 (2024)

    2024

  7. [15]

    arXiv:2402.10400 (2024)

    Servantez, S., Barrow, J., Hammond, K., Jain, R.: Chain of Logic: Rule-Based Reasoning with Large Language Models. arXiv:2402.10400 (2024)

    2024 arXiv

  8. [16]

    In: EMNLP

    Shi,X.,Zhu,Z.,Zhang,Z.,Li,C.:Hallucinationmitigationinnaturallanguagegen- eration from large-scale open-domain knowledge graphs. In: EMNLP. pp. 12506– 12521 (2023)

    2023

  9. [17]

    arXiv:2504.08970 (2025)

    Shirvani-Mahdavi, N., Akrami, F., Li, C.: On Large-scale Evaluation of Embedding Models for Knowledge Graph Completion. arXiv:2504.08970 (2025)

    2025 arXiv

  10. [18]

    In: ISWC

    Shirvani-Mahdavi, N., Akrami, F., Saeef, M.S., Shi, X., Li, C.: Comprehensive analysis of Freebase and dataset creation for robust evaluation of knowledge graph link prediction models. In: ISWC. pp. 113–133. Springer (2023)

    2023

  11. [19]

    CACM 57(10), 78–85 (2014)

    Vrandečić, D., Krötzsch, M.: Wikidata: a free collaborative knowledge base. CACM 57(10), 78–85 (2014)

    2014

  12. [20]

    NeurIPS 35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 35, 24824–24837 (2022)

    2022

  13. [21]

    TASLP31, 1499–1510 (2023)

    Wu, X., Cai, Y., Lian, Z., Leung, H.f., Wang, T.: Generating natural language from logic expressions with structural representation. TASLP31, 1499–1510 (2023)

    2023

  14. [22]

    arXiv:2303.12023 (2023) 10 N

    Yang, Z., Du, X., Mao, R., Ni, J., Cambria, E.: Logical reasoning over natural language as knowledge representation: A survey. arXiv:2303.12023 (2023) 10 N. Shirvani-Mahdavi et al

    2023 arXiv

  15. [23]

    NeurIPS 36, 46595–46623 (2023)

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS 36, 46595–46623 (2023)

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.