{"id":"a47f1c3c-7092-4003-9e98-dc14ca1c716b","arxiv_id":"2508.10971","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Rule2Text generates and evaluates natural language explanations of knowledge graph rules, finding that chain-of-thought prompting with entity types works best and that fine-tuning Zephyr on LLM-built ground truth sharply raises automatic metrics.","lead":"This paper introduces Rule2Text, a framework that uses large language models to turn formal knowledge graph rules into plain English explanations. It evaluates prompting strategies, builds an LLM judge to scale up evaluation, and fine-tunes an open-source model on the resulting data, reporting large gains on a biomedical benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains are measured only by BLEU/ROUGE/METEOR, not human evaluation; these lexical metrics are unvalidated for explanation quality, so the headline claim of improved fine-tuned quality is unsupported.","rationale":"The reader's weakest assumption concerned LLM-as-a-judge reliability. I agree that moderate human-judge agreement (alpha 0.59) is a real concern for the pseudo-ground-truth selection. However, I see a more immediately load-bearing issue: even if the training data were perfectly clean, Table 5's evaluation of fine-tuned Zephyr relies exclusively on BLEU/ROUGE/METEOR. These metrics are known to be poor proxies for human-judged quality in open-ended generation, and the paper does not validate them for this task (e.g., by correlating them with the human annotations already collected). The dramatic ROUGE improvement on ogbl-biokg (0.02 to 0.78) suggests the base model and fine-tuned model have almost no lexical overlap with the reference, which could be a sign of mode collapse or memorization rather than improved semantic correctness. Without human evaluation or metric validation, the paper's headline claim that fine-tuning improves explanation quality is not supported. This is a distinct concern from judge reliability, but both point to CONDITIONAL: the paper needs additional evidence (metric validation or human eval of the fine-tuned model) before the quantitative claims can be accepted. Since the reader already issued CONDITIONAL, I do not change the overall verdict.","tokens_in":12993,"tokens_out":6280,"duration_ms":68294,"concrete_test":"Using the 100 human-annotated examples (or a representative subset), compute BLEU/ROUGE/METEOR between each model-generated explanation and the corresponding human-approved reference, and correlate these scores with the human correctness ratings already collected in Section 5.2. If Spearman correlations are weak (<0.3), Table 5's metric gains cannot be interpreted as quality improvements. Additionally, conduct a human evaluation of fine-tuned vs base Zephyr on a 50-example sample from ogbl-biokg using the Section 5.2 rubric; if fine-tuned outputs are not rated significantly higher on correctness/clarity, the central fine-tuning claim should be retracted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the abstract — 'significant improvements in explanation quality after fine-tuning' — is supported only by Table 5, which reports BLEU, ROUGE, and METEOR on a 50-example test set. These are surface-level n-gram overlap metrics, and the paper provides no evidence that they correlate with the human correctness/clarity rubric used elsewhere. The test references are themselves Gemini-generated (with some human involvement), so high scores may reflect imitation of Gemini's style rather than genuine accuracy. The ogbl-biokg ROUGE jump from .02 to .78 is suspiciously large; a base model with .02 ROUGE shares almost no n-grams with the reference, and such a leap is more typical of template overfitting than of improved semantic correctness. Critically, no human evaluation was run on the fine-tuned Zephyr outputs, despite a human evaluation pipeline already being in place. Since the paper's headline claim rests on these unvalidated lexical metrics, it is not established. This is distinct from, though amplified by, the moderate LLM-judge agreement (Spearman 0.69, Krippendorff's alpha 0.59) used to select pseudo-ground truth for training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Rule2Text, an LLM-based framework for generating natural-language explanations of knowledge graph rules mined by AMIE. It compares prompting strategies (zero-shot, few-shot, variable-type augmentation, and Chain-of-Thought) across GPT-3.5 Turbo, GPT-4o mini, and Gemini 2.0 Flash; evaluates outputs by three human annotators on correctness and clarity; develops an LLM-as-a-judge protocol; constructs fine-tuning datasets for Freebase and ogbl-biokg; fine-tunes Zephyr-7B-beta; and adds a type-inference module for KGs lacking explicit types. The central claims are that CoT combined with variable-type information substantially improves explanation quality, that Gemini 2.0 Flash performs best, that the LLM judge agrees strongly with humans, and that fine-tuning significantly improves explanation quality, especially on biomedical rules.","tokens_in":13306,"tokens_out":5969,"duration_ms":62568,"significance":"If the claims hold, the paper would be the first comprehensive study of LLM explanation generation for KG rules, offering a practical recipe (CoT + variable types), a validated judge, public code/data, and a path to fine-tune small open models. The authors do ship public assets and use independent three-annotator human evaluation for the prompt-engineering phases, with the fine-tuning test set drawn from human-annotated examples. These are concrete strengths. However, the load-bearing evidence for the fine-tuning and judge claims is incomplete: the reported judge agreement is moderate, and the headline fine-tuning result rests on unvalidated lexical metrics with no human evaluation of the fine-tuned model. The stress-test concern about the fine-tuning evaluation therefore lands on reading the paper.","major_comments":[{"comment":"The abstract's headline claim — 'significant improvements in explanation quality after fine-tuning' — is supported only by BLEU, ROUGE, and METEOR on 50 test examples per dataset. The paper defines correctness and clarity as the evaluation criteria (Section 5.1) and uses human ratings for the prompt-engineering phases, but no human evaluation of the fine-tuned Zephyr outputs is reported, and no evidence is given that these lexical metrics track the human rubric. The ROUGE jump from 0.02 to 0.78 on ogbl-biokg is especially hard to interpret: a base model with 0.02 ROUGE shares essentially no n-grams with the reference, and a large gain could reflect imitation of the Gemini-derived reference style rather than improved semantic correctness. Please add human evaluation of the fine-tuned model, or at minimum validate the automatic metrics against human scores on the same test instances, and r","section":"Section 7, Table 5 (Zephyr Performance)"},{"comment":"The LLM-as-a-judge is load-bearing for the fine-tuning pipeline because it selects the 400 pseudo-ground-truth training examples (Section 6.2). Yet the reported agreement with humans is Spearman 0.69 and Krippendorff's alpha = 0.59 on 100 validation instances. That is moderate agreement, not 'strong' as claimed in the Abstract and Section 5.3. A judge with this level of consensus can systematically favor fluent but incorrect explanations; if so, the training labels are contaminated. Please report judge error patterns (e.g., a confusion matrix over score categories), examples where the judge disagrees with humans, and a human check on a sample of judge-selected training instances. Also report human-human inter-annotator agreement to calibrate what alpha=0.59 means in this task.","section":"Section 5.3 and Section 4.3"},{"comment":"The prompt-engineering conclusions (e.g., variable-type prompt improves correctness from 3.94 to 4.21 in Table 3; CoT improves Gemini to 4.67 in Table 4) are based on means over 100 rules with three annotators, with no standard deviations, significance tests, or inter-annotator reliability statistics. Without these, 'substantial improvements' and 'significant improvements' are not established; the differences are small relative to the granularity of a 1-5 scale and could be annotation noise. Please add per-condition distributions, paired tests across the 100 rules, and human-human agreement.","section":"Section 7, Tables 2-4"},{"comment":"The fine-tuning test set has only 50 instances per dataset and uses a single reference per rule. The references are Gemini-generated explanations that were human-annotated and possibly edited (Section 4.3). Fine-tuning on Gemini-generated pseudo-ground truth and then evaluating against Gemini-derived references creates a risk that high scores reflect lexical closeness to a particular style rather than correctness. Please report multi-reference evaluation, human evaluation, or both, and clarify exactly which of the 100 human-annotated examples fall into the validation split versus the test split.","section":"Section 6.2 and Table 5"}],"minor_comments":[{"comment":"'Strong agreement' is inconsistent with the reported Krippendorff's alpha of 0.59; suggest 'moderate agreement'.","section":"Abstract and Section 5.3"},{"comment":"The sentence 'the conversion process has resulted in a higher number of rules in these two datasets compared to those in FB+CVT-REV' is ambiguous; clarify which datasets are being compared.","section":"Section 6.1"},{"comment":"Calling judge-selected training instances 'ground-truth datasets' is misleading; use 'pseudo-ground truth' consistently.","section":"Section 6.2"},{"comment":"The 'triple evaluation' of each explanation is not described; specify whether repeated judge scores were averaged and how intra-model consistency was measured.","section":"Section 5.3"},{"comment":"Report inter-annotator agreement and confidence intervals; currently only mean scores are shown, which makes it impossible to judge the reliability of the reported differences.","section":"Section 7, Tables 2-4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the public release of code/data is a plus. The main risk is overclaiming from automatic metrics; the strongest claim (fine-tuning improves explanation quality) needs human evaluation or at least metric validation. I would also ask the editor to check that the relationship with the authors' short paper [26] and the preliminary self-enhancement-bias study is sufficiently delineated to avoid dual-publication concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely new task formulation, and the prompt-engineering results are the most solid part of the paper. The authors systematically compare zero-shot, few-shot, variable-type, and CoT prompting across Freebase and biomedical KGs, with human ratings from three blinded annotators. The finding that variable types plus CoT helps, with Gemini 2.0 Flash best, is credible given the numbers. The type inference module is a nice practical addition, and the code and datasets are public. Credit where due: this is reproducible, mostly well-described work.\n\nWhere it gets shaky is the fine-tuning claim. The abstract says 'significant improvements in explanation quality after fine-tuning,' but the only evidence is BLEU/ROUGE/METEOR on a 50-example test set. These n-gram overlap metrics are never validated against the correctness/clarity rubric used elsewhere, and no human evaluation was run on the fine-tuned Zephyr outputs even though the pipeline was in place. The ogbl-biokg ROUGE jump from .02 to .78 is exactly the kind of jump you'd expect if the model learned to copy Gemini's phrasing rather than to produce semantically correct explanations. That claim is not established by the data.\n\nSecond, the LLM-judge agreement is moderate, not strong: Spearman 0.69, Krippendorff's alpha 0.59. The abstract calls it 'strong agreement'; Section 5.3 says 'reasonably strong.' That is an overclaim. Since Gemini 2.0 Flash both generates candidate explanations and scores them to select pseudo-ground-truth for fine-tuning, there is a real circularity risk. The 100 human-annotated examples in the test set help, but the training labels are judge-selected.\n\nMinor: human evaluation on 100 rules per phase with no variance or significance tests, and no template baseline in the human evaluation. The prompt comparisons are still informative, though.\n\nOverall, this is a useful, honest systems paper for the KG-to-NL community. The prompt-engineering findings and released data are worth building on. But the fine-tuning result needs a proper human evaluation, or at least a validated automatic judge, before taking it at face value. I'd send it to peer review with a request for major revision: add human eval of fine-tuned outputs, report variability, and soften the agreement language. If the authors fix the evaluation, this could be a solid reference point for the task.","headline":"Genuinely new task and a credible prompt-engineering study, but the fine-tuning headline rests on unvalidated lexical metrics and a partially circular data pipeline.","tokens_in":13802,"tokens_out":2616,"would_cite":true,"duration_ms":26628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rule2Text shows that chain-of-thought prompting with variable entity types lets LLMs produce accurate, clear explanations of mined knowledge-graph rules, and that fine-tuning an open-source model on judge-selected examples substantially imp","keywords":["knowledge graph rules","natural language explanation","LLM-as-a-judge","chain-of-thought prompting","variable entity types","type inference","fine-tuning","explainability"],"falsifier":"Take the 100 rule-explanation pairs that humans scored in phases 2-3 and also obtain judge scores for them. Then adversarially edit a set of correct, human-approved explanations so that each contains exactly one error—one missing entity, one missing relation, or one hallucinated relation—and run the judge on the edited versions. If the judge's score fails to drop below the 'correct' threshold on a substantial share of edited explanations, the judge cannot support the pseudo-ground-truth selection, and the fine-tuning results built on it would need re-evaluation.","tokens_in":12881,"feed_emoji":"🤖","tokens_out":10163,"duration_ms":102052,"temperature":0.7,"pith_summary":"Knowledge-graph rule mining produces logical if-then rules, but the raw predicates (for example, /travel/accommodation/accommodation_type) are opaque to non-experts. Rule2Text asks whether large language models can translate these rules into plain English, and establishes three things: the best prompting recipe is chain-of-thought reasoning combined with explicit variable entity types; an LLM judge built on Gemini 2.0 Flash agrees with human annotators closely enough (Spearman 0.69) to scale evaluation and select pseudo-ground-truth; and fine-tuning the open-source Zephyr-7B on judge-selected examples sharply improves explanation quality, especially on the biomedical ogbl-biokg graph, where ROUGE rises from 0.02 to 0.78. If right, the framework gives a practical pipeline for making KG rules usable by non-experts in domains such as healthcare.","feed_headline":"Fine-tuning lifts KG-rule explanations from 0.02 to 0.78","feed_subtitle":"A validated LLM judge scores them, so small open models can learn to explain mined rules without costly human labeling.","key_machinery":"The load-bearing mechanism is a comparison-style chain-of-thought prompt: the model is asked, step by step, to check that every variable entity and every relation appearing in the rule also appears in the explanation, and to flag anything missing or hallucinated. This same verification prompt, combined with a scoring rubric and few-shot exemplars, is reused as the LLM-as-a-judge, which is what makes scalable evaluation and pseudo-ground-truth construction possible. Variable entity type information—the other half of the recipe—comes from the KG's type system when available, or from a type-inference module that shows the model three random instantiations of the rule.","core_discovery":"On the paper's own terms, the central discovery is that the obstacle to explaining KG rules is not the LLM's language ability but the model's ignorance of what the variables in a rule denote. Once the rule is given together with the entity types of its variables—either read from the KG's type system or inferred from a few rule instances—and the model is walked through a chain-of-thought prompt that verifies each variable and relation, explanations become substantially more correct and clear: on human evaluation, Gemini 2.0 Flash reaches 4.67/5 correctness and 4.70/5 clarity. The paper also establishes that an LLM judge can stand in for human annotation: using a similar verification-style pro","pith_inferences":["The paper does not test whether its judge is robust to adversarial corruptions; a direct stress-test would edit correct explanations to drop one relation or insert a hallucinated entity and check whether the judge's scores drop as humans' would.","The tennis-player subtype error points to an untested improvement: choosing rule instances that maximize entity-type diversity before type inference could eliminate over-specific guesses.","The authors only check self-enhancement bias for the proprietary models; a natural extension is to use the fine-tuned Zephyr as its own judge after a similar bias check, turning the whole pipeline into an open, self-sustaining loop."],"forward_implications":["The chain-of-thought plus variable-types prompt is a directly reusable recipe: any KG with readable relation labels and known or inferable entity types can get explanations without task-specific training.","The validated LLM judge can replace most human annotation effort; only low-scoring or judge/human-disagreement examples need expert review, making ground-truth dataset construction scale to thousands of rules.","Fine-tuning a 7B open model on 400 judge-selected pairs is enough to move explanation quality from near-zero content overlap to high overlap on a specialized biomedical KG, suggesting domain-specific fine-tuning is worthwhile for specialized graphs.","The type-inference module extends the recipe to KGs without explicit type information, at the cost of occasional over-specific type guesses when the sampled instances are narrow, such as inferring 'tennis player' instead of 'professional athlete'.","Because the framework treats rule mining as a pluggable component, the same explanation pipeline can be attached to other rule miners, not just AMIE."],"supporting_citations":[{"why":"Supplies the AMIE rule mining system and its implementation, which produce the Horn rules that the framework explains.","marker":"[3, 10]"},{"why":"Supplies the FB-CVT-REV and FB+CVT-REV datasets and the edge-type-to-entity-type mapping used to extract variable types.","marker":"[25]"},{"why":"Supplies ogbl-biokg, the biomedical knowledge graph used for the domain-specific evaluation and fine-tuning.","marker":"[12]"},{"why":"Preliminary short paper that reports the five-step chain-of-thought prompt and the self-enhancement bias check on which phase 3 and judge selection build.","marker":"[26]"},{"why":"Chain-of-thought prompting, the strategy that combined with variable types produces the best explanations.","marker":"[29]"},{"why":"Establishes the LLM-as-a-judge paradigm and the self-enhancement bias concern that motivates choosing Gemini 2.0 Flash as the judge.","marker":"[32]"},{"why":"Zephyr-7B-beta, the open-source model that is fine-tuned on the judge-selected ground-truth datasets.","marker":"[27]"}],"fun_headline_variants":["Entity types and chain-of-thought produce clear KG-rule explanations","LLM judge agrees with humans when scoring rule explanations","Fine-tuning makes small open models explain KG rules reliably","Type hints unlock the meaning of mined knowledge graph rules"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the Gemini 2.0 Flash judge's correctness scores are reliable enough to select pseudo-ground-truth explanations and to measure fine-tuning gains; the paper reports moderate agreement with humans (Spearman 0.69, Krippendorff's alpha 0.59), so if the judge systematically prefers fluent but wrong explanations, the 400 training labels are contaminated and the automatic-metric improvements would not reflect real accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Entity types and chain-of-thought produce clear KG-rule explanations","LLM judge agrees with humans when scoring rule explanations","Fine-tuning makes small open models explain KG rules reliably","Type hints unlock the meaning of mined knowledge graph rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1699,"prompt_tokens":840,"completion_tokens":859,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":793}},"tokens_in":584,"tokens_out":859,"duration_ms":9840,"temperature":1.0,"reasoning_tokens":793,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:13:30.668286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 100 rule-explanation pairs that humans scored in phases 2-3 and also obtain judge scores for them. Then adversarially edit a set of correct, human-approved explanations so that each contains exactly one error—one missing entity, one missing relation, or one hallucinated relation—and run the judge on the edited versions. If the judge's score fails to drop below the 'correct' threshold on a substantial share of edited explanations, the judge cannot support the pseudo-ground-truth selection, and the fine-tuning results built on it would need re-evaluation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FB-CVT-REV and FB+CVT-REV datasets and the edge-type-to-entity-type mapping used to extract variable types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Preliminary short paper that reports the five-step chain-of-thought prompt and the self-enhancement bias check on which phase 3 and judge selection build."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-a-judge paradigm and the self-enhancement bias concern that motivates choosing Gemini 2.0 Flash as the judge."}],"review_version":1}