{"id":"22dfebe6-ee90-4386-9cdb-59276ff4aec2","arxiv_id":"2607.10212","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"KGCQual is a sentence-level metric that scores extracted knowledge-graph triples by noun completeness/resolution/connectivity and verb multiplicity/similarity against a POS-and-dependency ideal graph.","lead":"KGCQual scores how well text-to-triple extractors preserve nouns, verbs, and negation relative to a POS/dependency-derived ideal graph. It gives a model-agnostic, interpretable way to compare OpenIE and LLM extractors before triples enter a knowledge graph.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The headline Spearman correlation rests on a tiny, selectively aggregated set of classical systems and is not robust under the paper’s own per-dataset numbers.","rationale":"The Reader correctly flags the imperfect POS/dependency “ideal” graph as a central modelling assumption and rightly keeps the verdict CONDITIONAL. That assumption is real but is already acknowledged in §6 and is not the single most load-bearing threat to the strongest claim as written. The claim’s decisive empirical support is the downstream correlation; that support is statistically thin and selectively aggregated. Strengthening (or retracting) the correlation analysis is therefore the highest-leverage check. The rest of the paper—formulas, multi-system tables, ablation, code link—remains a useful engineering contribution once the correlation claim is tempered or made robust. Hence I leave the verdict CONDITIONAL but shift the primary concern from oracle fidelity to the fragility of the reported ρ.","tokens_in":13242,"tokens_out":613,"duration_ms":7522,"concrete_test":"Recompute Spearman ρ and p on the full set of systems that produced any triples (including all three LLMs and the omitted ClausIE/BenchIE case) both per-dataset and pooled, reporting exact n and bootstrap 95 % CIs. If the pooled |ρ| falls below ~0.6 or p exceeds 0.05 under any reasonable inclusion rule, the downstream-validation pillar of the strongest claim collapses.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim leans on “KGCQual scores correlate significantly with link prediction performance (ρ = −0.900, p = 0.037)” (abstract, §5.6, Table 6). That number is obtained only after (i) aggregating three datasets, (ii) restricting to five classical OpenIE systems, (iii) dropping ClausIE on BenchIE because it produced no valid triples, and (iv) using only TuckER (ComplEx/NodePiece give near-zero MRR). The paper’s own Table 11 shows the per-dataset picture is far weaker: WebNLG ρ = +0.400 (n = 5, p = 0.505), BenchIE ρ = +0.400 (n = 4, p = 0.600), and only TinyButMighty reaches ρ = −0.900. With n ≈ 5 the aggregated p-value is fragile; a single rank swap or the re-inclusion of the omitted ClausIE point can erase significance. Thus the claimed “direct evidence that KGCQual reflects downstream utility” is not yet secure.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes KGCQual, an intrinsic sentence-level metric for assessing the quality of triples produced by automated information-extraction systems before they are integrated into a knowledge graph. An “ideal” reference graph is constructed from POS tags and dependency parses; extracted graphs are then scored by a noun metric (completeness, resolution, connectivity, with a lightweight negation penalty) and a verb metric (predicate multiplicity and lexical/embedding similarity). The composite score M = α MN + (1−α) MV is evaluated on eight extractors (five classical OpenIE tools and three LLMs) across WebNLG, TinyButMighty and BenchIE, with gold-triple lower bounds, an ablation of noun/verb weights, case studies, and a downstream TuckER link-prediction correlation (aggregated Spearman ρ = −0.900, p = 0.037). Code is released.","tokens_in":13505,"tokens_out":767,"duration_ms":6658,"significance":"If the metric is shown to be a reliable, model-agnostic proxy for extraction fidelity, it would fill a genuine gap between post-hoc KG quality frameworks and the practical realities of pipeline-driven construction. The explicit formulas, public code, multi-system multi-dataset evaluation, ablation, and attempt at downstream validation are concrete strengths that make the work usable and falsifiable. The central claim that KGCQual “correlates significantly with link-prediction performance” is, however, currently overstated relative to the supporting evidence; once that claim is properly scoped, the paper still offers a useful diagnostic tool for the community.","major_comments":[{"comment":"§5.6 and Table 6 present the headline Spearman ρ = −0.900 (p = 0.037) as “direct evidence that KGCQual reflects downstream utility.” Table 11 shows that this figure is obtained only after aggregation across three datasets, restriction to five classical systems, omission of ClausIE on BenchIE, and exclusive use of TuckER. Per-dataset correlations are non-significant or even positive (WebNLG ρ = +0.400, p = 0.505; BenchIE ρ = +0.400, p = 0.600). With n ≈ 5 the aggregated p-value is fragile. The abstract and §5.6 must either (a) report the per-dataset numbers prominently and qualify the claim, or (b) supply a statistically more robust validation (larger n, inclusion of LLM extractors, sensitivity analysis). As written, the strongest claim is not yet secure.","section":null},{"comment":"§3.2–3.4 and Limitations §6 treat the POS+dependency “ideal” graph as a faithful proxy for intended entities, predicates and polarity. The same linguistic pipeline is used both to define the reference and (implicitly) by several of the classical extractors being ranked, creating a mild circularity risk that is never quantified. A small human-judged sample of ideal-graph fidelity, or an explicit comparison against gold triples beyond the aggregate lower bounds of Table 4, is needed to underwrite the claim that deviations from Gi measure true extraction quality rather than parser idiosyncrasies.","section":null},{"comment":"The free parameters α = 0.5 and λ = 0.3 are fixed without sensitivity analysis beyond the five discrete weightings of the ablation (Table 5 / Appendix Table 7). Because the final ranking of systems and the Likert thresholds of §4.5 depend on these choices, a short sensitivity sweep (or a clear statement that rankings are stable under reasonable ranges) is required for the metric to be reproducible by others.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real contribution here is a concrete, fully specified intrinsic score that ranks OpenIE and LLM extractors by how well their triples match a POS/dependency ideal of the source sentence. That fills a practical gap: most KG-quality work starts after triples already exist, and most OpenIE evals stay at gold-triple F1 or downstream tasks. They give explicit formulas for noun completeness/resolution/connectivity and verb multiplicity/similarity (with a light UD-negation penalty), ship code, run eight systems on three public sets, show gold lower bounds, do a clean noun/verb ablation, and include readable case studies. That package is useful for anyone building or comparing extraction pipelines.\n\nWhat works: the metric is model-agnostic, decomposable, and easy to interpret. ClausIE consistently looks best, MinIE worst, LLMs in the middle—matching the qualitative examples. The ablation correctly flags predicate fidelity as the harder problem. Gold triples sit well below the automated systems, so the scale is not completely unanchored.\n\nSoft spots, in proportion. The ideal graph is only as good as NLTK POS + dependency parsing; the authors themselves flag this in Limitations and note that deep semantic coherence is out of scope. Gold scores never reach zero, which is honest but means the oracle itself is noisy. The headline Spearman ρ = −0.900 (p = 0.037) is the weakest claim: it appears only after aggregating three datasets, restricting to five classical systems, dropping ClausIE on BenchIE, and using only TuckER. Their own Table 11 shows the per-dataset picture is mixed or non-significant. That does not kill the paper; it just means the “downstream utility” sentence needs tempering. Free parameters (α, λ, Likert cut-offs) are fixed without sensitivity analysis, but that is minor.\n\nThis is for Semantic Web and OpenIE practitioners who need a cheap, sentence-level diagnostic before they dump triples into a KG. It is not a theoretical breakthrough, but it is reproducible engineering that a serious editor should send to referees. I would cite the metric definition and the multi-system tables; I would not lean on the aggregated correlation until it is re-run with more systems and per-dataset reporting. Worth engaging.","headline":"Solid engineering metric for the extraction stage of KG construction; the headline correlation is oversold but the rest of the work is usable and referee-ready.","tokens_in":14140,"tokens_out":556,"would_cite":true,"duration_ms":5168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An intrinsic metric scores how faithfully extracted triples match the entities, predicates, and polarity of the source sentence, and those scores track link-prediction quality.","keywords":["knowledge graph construction","information extraction evaluation","triple quality metric","entity fidelity","predicate multiplicity","negation handling","link prediction correlation","OpenIE"],"falsifier":"Construct a controlled set of sentences whose POS/dependency ideal graph deliberately mismatches human-annotated gold triples on entity boundaries or negation, recompute KGCQual, and check whether the correlation with link-prediction MRR collapses.","tokens_in":14104,"feed_emoji":"📊","tokens_out":625,"duration_ms":7292,"temperature":0.7,"pith_summary":"Knowledge graphs built by automated extractors routinely lose entities, collapse noun phrases, drop predicate instances, or invert negation, yet most evaluation still relies on downstream tasks or costly manual checks. This paper introduces KGCQual, a sentence-level score that asks how close an extractor’s triples come to an “ideal” reference graph built from the same sentence by part-of-speech tagging and dependency parsing. The score splits into a noun component (completeness, resolution, connectivity, with a graded penalty for polarity mismatches) and a verb component (multiplicity and lexical/semantic similarity). Across classical OpenIE tools and LLM extractors on WebNLG, TinyButMighty and BenchIE, lower KGCQual scores flag the systems that better preserve structure; gold triples sit near the bottom of the scale; and the ranking correlates strongly with how well a link-prediction model later performs on the same graphs. The result is a transparent, extractor-agnostic diagnostic that can be run before triples ever enter a knowledge base.","feed_headline":"Metric ranks how well extractors keep sentence structure","feed_subtitle":"KGCQual scores track omissions, polarity errors and later link-prediction success","key_machinery":"KGCQual score M = α MN + (1−α) MV, where MN normalizes deviations in entity completeness, node resolution and connectivity (with Jaro–Winkler + UD-negation matching), and MV normalizes deviations in predicate multiplicity and lexical/embedding similarity; both are computed against a POS- and dependency-derived ideal graph.","core_discovery":"KGCQual is an intrinsic, model-agnostic metric that quantifies extraction fidelity by measuring how closely an automatically produced triple graph approximates an ideal reference graph of the source sentence’s noun phrases, predicate instances (including multiplicity) and negation polarity; the resulting score reliably ranks extractors and correlates significantly with downstream link-prediction performance on the same graphs.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["KGCQual ranks extractors by fidelity to ideal sentence graphs","Metric scores how well triples keep nouns predicates and negation","Intrinsic KG quality measure tracks omissions and polarity errors","KGCQual links extraction fidelity to link-prediction performance","Framework compares extractors via structural semantic graph match"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The claim rests on treating a graph built only from POS tags and dependency parses as a faithful enough stand-in for the sentence’s intended entities, relations and polarity.","fun_headline_variants_meta":{"raw":{"variants":["KGCQual ranks extractors by fidelity to ideal sentence graphs","Metric scores how well triples keep nouns predicates and negation","Intrinsic KG quality measure tracks omissions and polarity errors","KGCQual links extraction fidelity to link-prediction performance","Framework compares extractors via structural semantic graph match"]},"model":"grok-4.5","effort":"low","cost_usd":0.003874,"raw_usage":{"total_tokens":1223,"prompt_tokens":818,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":38740000,"prompt_tokens_details":{"text_tokens":818,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":345,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":818,"tokens_out":60,"duration_ms":3999,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T13:27:19.156593+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Construct a controlled set of sentences whose POS/dependency ideal graph deliberately mismatches human-annotated gold triples on entity boundaries or negation, recompute KGCQual, and check whether the correlation with link-prediction MRR collapses.","supporting_citations":[],"review_version":1}