{"id":"a024922e-b850-4ab1-bb14-b177f554f488","arxiv_id":"2507.23740","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLMs generate mostly correct and clear explanations of knowledge-graph logical rules, and combining chain-of-thought prompting with entity type hints improves quality.","lead":"Machine learning rules that power knowledge graphs are often written in cryptic notation, and this study asks whether large language models can translate them into plain English. Human raters found the resulting explanations mostly correct and clear, with combining step-by-step reasoning and entity type hints working best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparative result is not yet supported: the 100-rule human evaluation lacks significance testing and inter-annotator agreement, and an implausible Table 3 entry suggests the underlying scores may be unreliable.","rationale":"The reader's weakest assumption (representativeness of the 100 highest-head-coverage rules and majority-vote aggregation) points in the same direction as my concern: the human evaluation is the load-bearing evidence for the central claim. I would sharpen this: the claim is comparative, so the absence of paired significance testing and reliability metrics is not a formatting nicety but the decisive risk. The concrete inconsistency in Table 3 strengthens this. I do not think the paper should be rejected; the qualitative finding is plausible and the GitHub release makes it checkable. But the paper should remain conditional on the raw annotations being released and the pairwise comparison being statistically supported, which is exactly what the CONDITIONAL verdict captures. Hence UNCHANGED is appropriate.","tokens_in":8512,"tokens_out":9154,"duration_ms":85942,"concrete_test":"Download the public GitHub annotation files and, for GPT-3.5, pair the Phase 2 variable-type score and Phase 3 CoT+type score for each of the same 100 rules; run a paired Wilcoxon signed-rank test and a bootstrap 95% CI on the mean difference for correctness and clarity. Then recompute Table 3's “2 atoms” row from the raw per-rule, per-annotator scores; if p > 0.05 or the row does not reproduce, the headline comparative claim is not established by the current data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (CoT + variable entity types gives the most accurate and readable explanations) is a comparative claim built on aggregate Likert scores from 100 hand-picked rules (50 per dataset in Phases 2 and 3). The decisive comparison is between GPT-3.5 Phase 2 variable-type scores (correctness 4.21, clarity 4.19) and Phase 3 CoT+type scores (correctness 4.28, clarity 4.26). The differences are 0.07 on a 1–5 scale, yet the paper reports no confidence intervals, no paired significance test, no effect size, and no inter-annotator agreement. The Phase 1 protocol also discards the dissenting annotator's scores after majority-vote selection, which can bias prompt-level aggregates. The data reliability concern is concrete: in Table 3, the variable-type “2 atoms” row reports m_ent=0.31 and m_rel=0.41, while the zero-shot row on the same rules reports 0.03/0.04 and the three-atom variable-type row reports 0.07/0.08. The text states that explanations have minimal missed entities and relations and that variable types help, so this entry is inconsistent as printed and could reflect a data or rendering error. If these aggregate numbers are not reproducible, or if the 0.07 difference is within annotator noise, the paper's headline finding does not follow from its evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates LLM-based natural language explanations for logical rules mined by the AMIE algorithm from Freebase-derived knowledge graphs (FB15k-237, FB-CVT-REV, and FB+CVT-REV). It compares zero-shot, few-shot, variable-entity-type, and chain-of-thought prompting strategies using GPT-3.5 Turbo, GPT-4o Mini, and Gemini 2.0 Flash, with human annotation of correctness, clarity, missed/hallucinated entities and relations, rule logicalness, and GPT-2 perplexity. It also reports an exploratory LLM-as-a-judge analysis. The headline claim is that chain-of-thought prompting combined with variable type information yields the most accurate and readable explanations.","tokens_in":8762,"tokens_out":4017,"duration_ms":36048,"significance":"If the headline claim were supported, this would be a useful first study of LLM-based explanation of KG rules, with practical guidance on prompting. The paper's strengths include public release of scripts, prompts, rules, explanations, and annotated data; a concrete human evaluation protocol; and an honest exploratory treatment of LLM-as-a-judge with both cross-family judge settings. However, the comparative evidence for the headline claim is currently thin: the decisive differences are small aggregate means without uncertainty quantification, and at least one table entry appears internally inconsistent.","major_comments":[{"comment":"The central comparative claim that CoT plus variable types is the most accurate and readable strategy rests on mean differences of 0.07 in correctness and clarity between the GPT-3.5 Phase 2 variable-type scores (4.21/4.19) and the Phase 3 CoT+type scores (4.28/4.26). The paper reports no paired significance test, confidence interval, effect size, or inter-annotator agreement, so the claim of 'significant' improvement in Section 3.1, Phase 2, is unsupported. Please report per-rule paired differences with a suitable test (e.g., Wilcoxon signed-rank), variance, and annotation reliability (e.g., Krippendorff's alpha or Cohen's kappa) for all phases.","section":"Section 4, Tables 3 and 4; Section 5"},{"comment":"The aggregation rule drops the dissenting annotator's metric scores after majority-vote selection of the preferred explanation ('we only averaged the measures provided by annotators 1 and 2'). If annotators differ systematically in severity, this deletion can bias the chosen prompt's average and distort the zero-shot versus few-shot comparison in Table 2. Please analyze all three annotators' scores, for example with a mixed-effects model or by reporting per-annotator means, or justify the deletion explicitly.","section":"Section 4, Phase 1"},{"comment":"The reported m_ent=0.31 and m_rel=0.41 contradict the text's statement that both explanation types have minimal missing entities and relations, and they are inconsistent with the neighboring zero-shot row (0.03/0.04) and the 3-atoms variable-type row (0.07/0.08). This suggests a data or rendering error; it must be corrected or explicitly explained, and the affected aggregates recomputed, because the inconsistency casts doubt on the reliability of the table's other entries.","section":"Table 3, '2 atoms' row, variable-type prompt"},{"comment":"The evaluation uses only the 100 rules with the highest head coverage in each phase, yet Table 3 shows that rule complexity (2 vs. 3 atoms, concatenated relations) strongly affects scores. Without evidence that this subset is representative of the rules KG users need explained, the external validity of the comparative claim is limited. Please justify the selection or report results on a broader or stratified sample.","section":"Section 4, Phase 2 and Phase 3 sample selection"}],"minor_comments":[{"comment":"The abstract contains a malformed URL with duplicated braces: 'https://github.com/idirlab/KGRule2NL}{https://github.com/idirlab/KGRule2NL' should be a single URL.","section":"Abstract"},{"comment":"Hallucination and missed-entity metrics are dropped in Phase 3 without reporting their values for the new models and CoT prompts; please provide these numbers or justify their omission with the Phase 3 data.","section":"Section 4, Phase 3"},{"comment":"The phrase 'there is almost a function that maps from the edge type to a type' is informal; please define the intended mapping precisely or replace it with a formal statement.","section":"Section 3.1, Phase 2"},{"comment":"The abbreviations m_ent, m_rel, h_ent, h_rel, correct, and logical are not defined in the table captions; please add a definition line in each caption for readability.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the released artifacts are valuable. My main concern is statistical support for the headline comparative claim, and the anomalous Table 3 entry should be checked against the released data before revision. I did not find evidence of citation or authorship misconduct; the reliance on the authors' own earlier dataset papers [18] is understandable given the task."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rule2Text is the first paper I know of that systematically tests LLM prompting strategies for turning mined KG rules into natural language. The task is real: AMIE rules are opaque, and a working verbalizer would help people debug and trust KG inference. The paper does useful groundwork—three datasets, three models, zero/few-shot, variable-type hints, CoT, human evaluation, and an honest exploratory LLM-as-judge section. They also ship scripts and data, which is more than many papers do.\n\nThe soft spots are in the evaluation, and they are not trivial. Phase 1 picks the majority-preferred prompt per rule and then averages only the annotators who chose it, discarding the dissenting annotator's scores entirely. That biases prompt-level averages upward and makes the zero-shot vs few-shot comparison uninterpretable. In Phase 2 the text says variable types 'significantly improved' performance, but there are no significance tests, no confidence intervals, and no inter-annotator agreement reported. The decisive Phase 2 vs Phase 3 comparison for GPT-3.5 is a 0.07 difference in correctness (4.21 vs 4.28) and 0.07 in clarity (4.19 vs 4.26) on a 1–5 scale; without error bars that is within annotator noise. And Table 3 has a genuinely implausible entry: the variable-type '2 atoms' row reports 0.31 missed entities and 0.41 missed relations, while the zero-shot row on the same rules reports 0.03/0.04 and the three-atom variable-type row 0.07/0.08. That is likely a data or rendering error, but it undermines confidence in the reported aggregates as printed.\n\nNone of this kills the qualitative story—variable-type hints and CoT do seem to help, and the LLM-as-judge analysis is explicitly exploratory and benchmarked against humans. But the central comparative claim is currently supported more by intuition than by statistics.\n\nThis paper deserves a serious referee and a revision that fixes the data issue, reports agreement and uncertainty, and makes the prompt comparison apples-to-apples. I'd take it.","headline":"Useful first look at LLM verbalization of KG rules, but the headline comparison rests on fragile human evaluation and at least one bad table entry.","tokens_in":9359,"tokens_out":2958,"would_cite":true,"duration_ms":28780,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining chain-of-thought prompting with variable entity type information produces the most accurate and readable natural-language explanations of knowledge-graph rules among the prompting strategies it tested, and…","keywords":["knowledge graphs","logical rules","natural language explanation","large language models","chain-of-thought prompting","human evaluation","AMIE","Freebase"],"falsifier":"A fresh annotation of a random sample of rules, stratified by atom count and relation type, that fails to show an advantage for chain-of-thought with variable types over the zero-shot baseline would falsify the central ranking; likewise, if a different aggregation rule such as the median of all three annotators reversed the order between strategies on the same 100 rules, the reported comparison would not be robust.","tokens_in":8259,"feed_emoji":"🧠","tokens_out":3882,"duration_ms":41239,"temperature":0.7,"pith_summary":"The paper asks whether large language models can turn logical rules mined from knowledge graphs—abstract Horn clauses such as “if a person is the mother of a child, then the person is the parent of that child”—into natural-language explanations that ordinary users can understand. It tests zero-shot, few-shot, variable-entity-type, and chain-of-thought prompting strategies on rules extracted by AMIE from FB15k-237, FB-CVT-REV, and FB+CVT-REV. Human annotators rated the outputs on correctness, clarity, and hallucination, and the paper also tested LLM-as-a-judge as a scalable alternative to human scoring. The central finding is that combining chain-of-thought prompting with variable entity type information yields the most accurate and readable explanations, with Gemini 2.0 Flash performing best among the three models evaluated. If correct, this gives a concrete prompt recipe for explaining KG rules and a baseline for scaling evaluation through automated judges.","feed_headline":"Chain-of-thought with type hints best explains KG rules","feed_subtitle":"Adding variable entity types and CoT reasoning gives the best accuracy and clarity across three datasets in human evaluation.","key_machinery":"The carrying mechanism is the logical rule itself—an AMIE-mined Horn clause with a body of atoms and a head atom, written $B_1 \\land \\dots \\land B_n \\Rightarrow H$—plus the prompt design that presents the rule to the LLM. The decisive added components are the list of entity types appended for each variable (derived from the near-functional mapping between edge types and the types of subjects and objects they connect) and a five-step chain-of-thought instruction that parses the rule, selects the contextually relevant type for each variable, interprets each atom with those types, synthesizes the overall implication, and finally writes a concise explanation in natural language.","core_discovery":"The paper's claim is that large language models can generate faithful, clear natural-language explanations for logical rules mined from knowledge graphs, and that the specific combination of chain-of-thought prompting plus variable entity type hints outperforms the alternatives. The authors support this with a three-phase human evaluation: phase one compares zero-shot and few-shot prompting on FB15k-237 rules; phase two adds variable entity types for rules from the large-scale FB-CVT-REV and FB+CVT-REV datasets; phase three applies chain-of-thought prompting across GPT-3.5 Turbo, GPT-4o Mini, and Gemini 2.0 Flash. On correctness and clarity scales of 1 to 5, the variable-type prompt consistently outperforms the zero-shot baseline, and the CoT prompt further improves GPT-3.5 Turbo's scores. Gemini 2.0 Flash achieves the highest overall correctness and clarity, and hallucination and omission counts remain low across all settings, indicating that the remaining errors concentrate in rules with concatenated relations and mediator nodes.","pith_inferences":["The prompt recipe likely transfers to other symbolic-output settings—proof traces, plan descriptions, or rule-based systems beyond AMIE—where the crucial ingredient is having the same kind of entity-type annotation available from the underlying data.","A natural next test the paper leaves implicit: use the highest-rated (rule, explanation) pairs as pseudo-ground truth, fine-tune smaller open models, and compare their output quality and cost against the best prompting strategy.","The self-family judge bias reported in the literature suggests that the moderate judge-annotator correlations could be improved by blending judges from different model families or calibrating judge scores against a small human-annotated set.","If the goal is deployment, the most informative follow-up experiment is measuring whether these explanations actually help a human worker verify or reject inferred facts, which would test the downstream value the paper motivates."],"forward_implications":["Knowledge-graph tools can adopt a ready-made prompt recipe—chain-of-thought plus variable entity types—to turn mined rules into readable text without per-rule templates.","The evaluation metrics and annotation protocol give a replicable standard for measuring explanation correctness, clarity, and hallucination in this task.","Since the same rule set produced better explanations under CoT prompting, the reported gains are attributable to the prompt structure rather than to model choice alone, though model choice also matters.","The moderate judge-annotator correlation suggests that LLM-as-a-judge can partially substitute for human evaluation, enabling larger-scale screening of generated explanations.","Hallucination counts stay low even as rule complexity rises, so the main remaining obstacle is explaining concatenated and mediator-node rules clearly rather than preventing fabrication."],"supporting_citations":[{"why":"Supplies the AMIE association rule mining algorithm that extracts the logical rules to be explained.","marker":"[8]"},{"why":"Provides the FB-CVT-REV and FB+CVT-REV datasets and the near-functional edge-type to entity-type mapping used to build the variable type prompts.","marker":"[18]"},{"why":"Provides the FB15k-237 benchmark dataset used for the phase one experiments.","marker":"[4]"},{"why":"Supplies the chain-of-thought prompting method that the paper combines with variable entity types in the best-performing prompt.","marker":"[20]"},{"why":"Motivates and supplies the LLM-as-a-judge evaluation approach tested for scalable scoring of explanations.","marker":"[23]"},{"why":"Motivates the cross-model judge design by documenting that LLM judges tend to favor generations from their own model family.","marker":"[14]"}],"fun_headline_variants":["CoT plus type hints gives clearest KG rule explanations","Type-hinted CoT prompts best translate KG rules","Rule2Text: LLMs explain KG rules best with CoT","Entity types and CoT yield top KG rule clarity","Few-shot with types and CoT tops KG rule text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings depend on the sampled rules—the 100 highest-head-coverage rules per phase—being representative of the rules people actually need explained, and on the majority-vote aggregation of three annotators (with one dissenting annotator's scores discarded) being an unbiased measure of explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["CoT plus type hints gives clearest KG rule explanations","Type-hinted CoT prompts best translate KG rules","Rule2Text: LLMs explain KG rules best with CoT","Entity types and CoT yield top KG rule clarity","Few-shot with types and CoT tops KG rule text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1440,"prompt_tokens":992,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":608,"tokens_out":448,"duration_ms":4703,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:25:01.698524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A fresh annotation of a random sample of rules, stratified by atom count and relation type, that fails to show an advantage for chain-of-thought with variable types over the zero-shot baseline would falsify the central ranking; likewise, if a different aggregation rule such as the median of all three annotators reversed the order between strategies on the same 100 rules, the reported comparison would not be robust.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AMIE association rule mining algorithm that extracts the logical rules to be explained."},{"cited_title":"In: ISWC","cited_arxiv_id":null,"evidence_quote":"Provides the FB-CVT-REV and FB+CVT-REV datasets and the near-functional edge-type to entity-type mapping used to build the variable type prompts."},{"cited_title":"In: NeurIPS","cited_arxiv_id":null,"evidence_quote":"Provides the FB15k-237 benchmark dataset used for the phase one experiments."},{"cited_title":"NeurIPS 35, 24824–24837 (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting method that the paper combines with variable entity types in the best-performing prompt."},{"cited_title":"NeurIPS 36, 46595–46623 (2023)","cited_arxiv_id":null,"evidence_quote":"Motivates and supplies the LLM-as-a-judge evaluation approach tested for scalable scoring of explanations."},{"cited_title":"NeurIPs37, 68772–68802 (2024)","cited_arxiv_id":null,"evidence_quote":"Motivates the cross-model judge design by documenting that LLM judges tend to favor generations from their own model family."}],"review_version":1}