{"id":"99e7d3e0-1f26-44a8-be1d-dbf420d6f3a3","arxiv_id":"2502.02896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Metalinguistic disagreements, where the dispute is over word meaning rather than facts, appear in LLM fact-checking against knowledge graphs, based on a 250-triple pilot study.","lead":"This paper asks whether disagreements between large language models and knowledge graphs are sometimes about the meaning of words, not about facts. It reports a small experiment suggesting such disagreements do occur, and proposes a human-annotated benchmark to measure them reliably.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MDR values in Table 1 depend entirely on a single unvalidated LLM-as-judge; without human annotation or inter-judge agreement, the claimed occurrence of metalinguistic disagreement remains a plausible but unsupported hypothesis.","rationale":"The paper's contribution is twofold: a hypothesis about metalinguistic disagreement between LLMs and KGs, and a proposal for a benchmark to detect it. The pilot experiment is the only evidence offered for the hypothesis. Every MDR value in Table 1 is the output of a single LLM judge applied to rationales from a first LLM. The authors transparently acknowledge the limitation in Section 3, but that does not remove the dependency: without human validation, the empirical rates could be inflated by the judge's forced-choice prompt or by systematic differences in how the source LLMs phrase negative rationales. The concrete test I propose is the one the paper itself outlines as future work: human annotation of a sample of the same rationales, with inter-annotator agreement computed against the LLM judge. If human agreement is poor, the occurrence claim is unsupported; if agreement is good, the pilot becomes credible. Either way, the benchmark proposal remains valuable, so the conditional verdict stands. My read therefore does not change the reader's verdict.","tokens_in":6721,"tokens_out":5315,"duration_ms":48723,"concrete_test":"Select a stratified random sample of 50 false-negative rationales from the experiment, spanning all nine source LLMs. Have at least three human annotators, blind to the LLM judge's labels and to the paper's hypothesis, classify each rationale as 'factual error', 'metalinguistic disagreement', or 'ambiguous', using a pre-registered rubric. Compute Fleiss' kappa among annotators and the agreement rate between the human majority and the gpt-4o-2024-05-13 judge. If human-judge agreement is below 0.6 or the kappa is below 0.4, the MDR values in Table 1 cannot be treated as reliable evidence for the occurrence claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that metalinguistic disagreements occur between LLMs and KGs—rests on the MDR numbers in Table 1. These numbers are produced by a second LLM (gpt-4o-2024-05-13) that classifies, in a forced-choice setting, whether the first LLM's rationale indicates a metalinguistic disagreement. The paper itself flags in Section 3 that LLM-as-a-judge 'may not be a reliable substitute for human judgment' and that detected disagreements could be 'artifacts of how different LLMs process and generate language.' The risk is concrete: the judge prompt explicitly asks for metalinguistic versus factual classification, which creates a strong prior toward labeling rationales that mention predicate meaning as metalinguistic; there is no baseline (e.g., reversed labels or open-ended classification), no inter-judge agreement, and no human gold standard. The two examples in Table 2 are illustrative, not systematically sampled, and cannot support a rate. Consequently, the existence claim 'metalinguistic disagreement does occur' is not established by the pilot; the experiment only shows that an LLM can be prompted to produce rationales that another LLM labels as metalinguistic. This is exactly the gap the proposed human-annotated benchmark would fill, which is why the paper is best read as a position paper with a pilot, not as a demonstration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses whether errors made by LLMs when fact-checking knowledge graph triples can be metalinguistic disagreements, rather than purely factual ones. It reports a pilot experiment on 250 triples sampled from T-REx, using nine LLMs as zero-shot chain-of-thought truth-value classifiers and a second LLM-as-a-judge (gpt-4o-2024-05-13) to classify whether the first LLM's rationale indicates a metalinguistic disagreement. Table 1 reports false-negative rates and metalinguistic disagreement rates (MDR) between 0.040 and 0.264. The paper then proposes requirements for a human-annotated benchmark that would address the pilot's limitations, and discusses implications for knowledge graph and ontology engineering.","tokens_in":7068,"tokens_out":3526,"duration_ms":30742,"significance":"The conceptual distinction drawn here is valuable: evaluation pipelines that treat every LLM-KG mismatch as a factual error may misattribute representational disagreements that would require different remedies, such as ontology engineering or prompt redesign. The manuscript is transparent about its limitations, explicitly listing the lack of human validation, possible conflation with other error types, and the small sample size, and it releases code and data on GitHub. If the occurrence of metalinguistic disagreement were established with validated annotations, the proposed benchmark would be a useful community resource. However, the empirical support in the current paper is a small pilot whose central numbers come from an unvalidated LLM-as-a-judge, so the existence claim remains a plausible hypothesis rather than a demonstrated result.","major_comments":[{"comment":"The MDR values, which are the main quantitative evidence for the paper's hypothesis, depend entirely on the second LLM-as-a-judge classifying the first LLM's rationale. The paper itself states in Section 3 that an LLM-as-a-judge 'may not be a reliable substitute for human judgment' and that detected disagreements could be artifacts of how different LLMs process language. Because the judge is asked a forced-choice question about whether a rationale indicates metalinguistic disagreement, there is no control condition to estimate label bias. I would need to see at least a human-annotated subset with inter-annotator agreement, a reversed-labels or open-ended judge baseline, or an agreement analysis across multiple judges before Table 1 can be treated as evidence that metalinguistic disagreements actually occur.","section":"Section 2, Table 1"},{"comment":"The experimental protocol is under-specified: the paper does not provide the prompts used for either the truth-value classifier or the LLM-as-a-judge, nor does it describe how the judge was calibrated or whether the judge was used with any prompt variations. In addition, one of the evaluated classifiers is gpt-4o-2024-05-13, which is the same model used as the judge; this creates a potential circularity in which a model judges rationales produced by itself. The paper should report the exact prompts, use a judge that is not one of the evaluated classifiers or at least measure judge-model agreement, and make all judge outputs available rather than only summaries.","section":"Section 2"},{"comment":"The two examples in Table 2 are illustrative and are not shown to be representative of the 250 sampled triples or of the MDR counts in Table 1. No information is given about how these examples were selected, how often the judge assigned each type of metalinguistic label, or whether the rationales were randomly drawn. A rate claim cannot be supported by anecdotal examples; the manuscript should provide a systematic breakdown of the judge's classifications, including examples of false positives and negatives relative to the intended metalinguistic category.","section":"Section 2, Table 2"},{"comment":"The paper proposes a benchmark but does not present a benchmark dataset, an annotation guideline, or a pilot human evaluation; it only lists requirements and a plan to extend the dataset through crowdsourcing. As written, the contribution is a position statement with a small pilot, not a citable benchmark. To make the central claim load-bearing, the authors need to either add a human-annotated validation study on a subset of the 250 triples or clearly reframe the paper as a proposal that makes no empirical existence claim. Without this, the paper does not establish that metalinguistic disagreement occurs between LLMs and KGs.","section":"Section 3"}],"minor_comments":[{"comment":"The sentence 'We argue that the above results suggest that that metalinguistic disagreement...' contains a duplicated 'that' and should be corrected.","section":"Section 3, first sentence"},{"comment":"The corresponding author's email address appears garbled in the header as '/envel⌢pe-⌢penb.p.allen@uva.nl'; this should be rendered as a standard email address.","section":"Author affiliation/header"},{"comment":"The sample size of 250 triples is small, and the reported rates have no confidence intervals or statistical significance tests; adding these would help calibrate how much weight to place on the observed differences across LLMs.","section":"Section 2"},{"comment":"In the second example, the predicate is referred to both as 'made from material' and 'made from material (P186)', while the Wikidata property P186 is usually labeled 'material used'; the paper should use a consistent predicate label.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and well-framed, and the proposed benchmark direction is sensible for the Semantic Web / KG engineering community. My concern is that the empirical pilot, as currently reported, cannot bear the weight of the existence claim in the abstract. I would support acceptance after the authors either add a human-validated subset with agreement metrics or substantially reframe the contribution as a proposal with illustrative examples only."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper introduces a genuinely new angle on LLM fact-checking against knowledge graphs—some errors that look factual may actually be disagreements about the meaning of predicates or terms—and it does so honestly, with a pilot the authors themselves clearly label as insufficient. The empirical support is thin, but the idea is worth taking seriously.\n\nWhat's new: applying Plunkett and Sundell's notion of metalinguistic disagreement to the specific setting of LLM-versus-KG evaluation, proposing a benchmark that would separate factual from metalinguistic disagreement, and reporting a first exploratory measurement (MDR across nine LLMs on 250 T-REx triples). The paper is well-written and transparent. It ships code and data on GitHub, and it explicitly lists its own limitations: no human validation of the labels, possible conflation with hallucination or context misinterpretation, and a small sample. That is real credit.\n\nThe soft spot is exactly where the reader and the stress-test note point. The MDR numbers in Table 1 depend entirely on a single unvalidated LLM-as-judge (gpt-4o) classifying whether the first LLM's rationale indicates metalinguistic disagreement. The judge prompt forces a metalinguistic-versus-factual classification, which creates a strong prior toward labeling any rationale that mentions predicate meaning as metalinguistic. There is no human gold standard, no inter-judge agreement, and no baseline like reversed labels or open-ended classification. So the measured rates don't establish that metalinguistic disagreement occurs; they show that an LLM can be prompted to produce rationales that another LLM labels that way. The two examples in Table 2 are illustrative, not a systematic sample. To the authors' credit, their own wording is careful—they say \"we hypothesize\" rather than \"we demonstrate\"—and the proposed benchmark with human annotation, negative examples, and inter-annotator agreement is a sensible next step. But the existence claim remains a hypothesis.\n\nWho this is for: anyone working on KG construction, LLM fact-checking, or error taxonomies in LLM evaluation. The paper is a useful conceptual contribution and a good prompt for thinking about remediation: factual errors might be fixed by KG refinement, while metalinguistic disagreements might require ontology engineering or intent-definition prompting. That distinction is the real value.\n\nRecommendation: a serious referee should engage with this, not desk-reject it. The idea is new, the benchmark design is concrete, and the authors are appropriately humble about what the pilot can support. The review should push them to validate the LLM-as-judge with human annotation and to report inter-judge agreement before any rate claims are taken as evidence. Send it to a venue with room for position papers or a workshop track; it doesn't have the empirical weight for a main conference acceptance as-is, but it deserves careful reading.","headline":"A thoughtful position paper that introduces a genuinely new error category for LLM-KG fact-checking, but the pilot evidence for that category rests on an unvalidated LLM-as-judge, so the existence claim is plausible, not established.","tokens_in":7503,"tokens_out":1816,"would_cite":true,"duration_ms":95245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Up to 26% of LLM 'fact errors' against knowledge graphs may be disputes about word meanings, not facts.","keywords":["metalinguistic disagreement","knowledge graphs","large language models","fact checking","LLM-as-a-judge","zero-shot chain-of-thought","ontology engineering","benchmark"],"falsifier":"A direct test would be to have several human annotators label the same 250 rationale texts as factual disagreement, metalinguistic disagreement, or agreement, and compare their labels with the LLM-as-a-judge labels; if human-judge agreement is at or near chance, or if a different judge model yields materially different metalinguistic disagreement rates, the reported rates would not support the hypothesis.","tokens_in":6478,"feed_emoji":"💬","tokens_out":6157,"duration_ms":49546,"temperature":0.7,"pith_summary":"The paper asks whether a reported error by a large language model checking facts against a knowledge graph is always a factual error, or sometimes a metalinguistic disagreement: a dispute about what the words or predicates mean. It hypothesizes that metalinguistic disagreement really does occur, and reports a pilot test on 250 Wikipedia-derived KG alignments in which a second LLM judged the first LLM's rationales. Across nine LLMs, between 4.0% and 26.4% of alignments, with a mean of 9.7%, were flagged as metalinguistic disagreements, and these formed a substantial share of false negatives. If the hypothesis holds, standard accuracy metrics conflate two error types that need different fixes: factual errors can be repaired with knowledge graph refinement or few-shot facts, while metalinguistic disagreements need ontology engineering or intensional definitions in prompts. The proposed human-annotated benchmark is designed to test this at scale.","feed_headline":"Up to 26% of LLM 'fact errors' may be word-meaning disputes","feed_subtitle":"A 250-triple pilot finds LLMs disagree with Wikidata about what predicates mean, not just about facts.","key_machinery":"The load-bearing mechanism is a two-stage LLM pipeline: a zero-shot chain-of-thought classifier assigns a truth value to a KG triple with its source text as context and produces a written rationale, and a second zero-shot chain-of-thought classifier acts as an LLM-as-a-judge, labeling each rationale as exhibiting metalinguistic disagreement or not. Metalinguistic disagreement, defined as disagreement about the meaning or use of a term or predicate rather than about facts, is the central object being detected. The pipeline defines the benchmark's core operation: separating false negatives caused by meaning disputes from those caused by factual disputes, and all reported metalinguistic disagreement rates flow through the judge's labels.","core_discovery":"The paper's central claim is that metalinguistic disagreement between LLMs and KGs is a real, detectable phenomenon in fact-checking tasks, not just a theoretical possibility. Using a zero-shot chain-of-thought classifier that assigns truth values to Wikidata triples aligned with Wikipedia abstracts, and a second LLM-as-a-judge that reads the classifier's rationale, the authors find that on the 250 sampled alignments the metalinguistic disagreement rate ranges from 0.040 to 0.264, with a mean of 0.097. Two illustrative cases show the disagreement is about predicates: one classifier rejects 'followed by' because 'replaced by' fits better, and another rejects 'made from material' because sugar is not a primary ingredient. The authors conclude that this motivates a benchmark distinguishing factual from metalinguistic disagreement, with human annotation, inter-annotator agreement metrics, multiple KG sources, and deliberately ambiguous, temporal, and gradable examples.","pith_inferences":["As an extension, if metalinguistic disagreement is driven by predicate semantics, then predicates with temporal or gradable content, such as 'followed by', should show systematically higher disagreement rates than well-defined relations; this can be tested by stratifying the sampled triples by predicate.","As an extension, the 9.7% mean rate rests on only 250 triples, so the true rate has a wide uncertainty; human labels would calibrate the judge model and shrink that uncertainty.","As an extension, the same two-stage design could audit retrieval-augmented pipelines, where retrieval context plus generator rationale may hide meaning-level mismatches between query terms and KG schema."],"forward_implications":["Fact-checking accuracy metrics for LLMs against KGs should not be read as purely factual; a measured error can be a semantic mismatch that will persist even if the correct facts are supplied.","Knowledge graph construction and refinement may need ontology engineering to align predicate meanings, rather than only fact correction, when disagreements are metalinguistic.","Prompts that supply intensional definitions of concepts and relations in natural language may reduce false negatives that are actually semantic rather than factual.","A human-annotated benchmark with inter-annotator agreement and multiple KG sources is needed before occurrence rates can be confidently estimated."],"supporting_citations":[{"why":"Supplies the aligned Wikipedia-abstract and Wikidata-triple corpus from which the 250 test alignments were sampled.","marker":"[22]"},{"why":"Supplies the zero-shot chain-of-thought prompting technique the first classifier uses to produce truth values and rationales.","marker":"[23]"},{"why":"Supplies the approach for prompting an LLM with a KG triple and Wikipedia abstract context to judge truth value.","marker":"[24]"},{"why":"Supplies the LLM-as-a-judge approach the second classifier uses to label rationales as metalinguistic disagreements.","marker":"[25]"},{"why":"Supplies the human-annotation and crowdsourcing design pattern the proposed benchmark plans to follow.","marker":"[6]"},{"why":"Supplies the definition of metalinguistic disagreement that motivates and operationalizes the detection task.","marker":"[14]"}],"fun_headline_variants":["LLM 'fact errors' often hide word-meaning disputes","One in ten LLM-KG mismatches is about meaning, not facts","Up to 26% of LLM fact-check 'errors' are semantic disputes","Metalinguistic clashes: the overlooked cause of LLM fact errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire measured disagreement rate rests on the assumption that a second LLM reliably recognizes when the first LLM's rationale expresses a disagreement about word meaning rather than a factual point or a confabulation.","fun_headline_variants_meta":{"raw":{"variants":["LLM 'fact errors' often hide word-meaning disputes","One in ten LLM-KG mismatches is about meaning, not facts","Up to 26% of LLM fact-check 'errors' are semantic disputes","Metalinguistic clashes: the overlooked cause of LLM fact errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1706,"prompt_tokens":910,"completion_tokens":796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":526,"tokens_out":796,"duration_ms":7999,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:40:28.245840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to have several human annotators label the same 250 rationale texts as factual disagreement, metalinguistic disagreement, or agreement, and compare their labels with the LLM-as-a-judge labels; if human-judge agreement is at or near chance, or if a different judge model yields materially different metalinguistic disagreement rates, the reported rates would not support the hypothesis.","supporting_citations":[{"cited_title":"Elsahar, P","cited_arxiv_id":null,"evidence_quote":"Supplies the aligned Wikipedia-abstract and Wikidata-triple corpus from which the 250 test alignments were sampled."},{"cited_title":"Kojima, S","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot chain-of-thought prompting technique the first classifier uses to produce truth values and rationales."},{"cited_title":"Evaluating Class Membership Relations in Knowledge Graphs using Large Language Models","cited_arxiv_id":"2404.17000","evidence_quote":"Supplies the approach for prompting an LLM with a KG triple and Wikipedia abstract context to judge truth value."},{"cited_title":"Mickus, E","cited_arxiv_id":null,"evidence_quote":"Supplies the human-annotation and crowdsourcing design pattern the proposed benchmark plans to follow."},{"cited_title":"Plunkett, T","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of metalinguistic disagreement that motivates and operationalizes the detection task."}],"review_version":1}