KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

Aanchal Gupta; Manuj Mukherjee; Nipun Misra; Raghava Mutharaju; Vikranth Udandarao; Yogender Kumar

KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

T0 review · 3 major / 0 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An intrinsic metric scores how faithfully extracted triples match the entities, predicates, and polarity of the source sentence, and those scores track link-prediction quality.

desk verdict Solid engineering metric for the extraction stage of KG construction; the headline correlation is oversold but the rest of the work is usable and referee-ready. read the letter →

arxiv 2607.10212 v1 pith:PT25N3AE submitted 2026-07-11 cs.AI cs.CL

Nipun Misra , Vikranth Udandarao , Aanchal Gupta , Yogender Kumar , Manuj Mukherjee , Raghava Mutharaju This is my paper · ORCID

classification cs.AIcs.CL

keywords knowledgegraphconstructioninformationextractionevaluationtriplequalitymetricentityfidelitypredicatemultiplicitynegationhandlinglinkpredictioncorrelationOpenIE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge graphs built by automated extractors routinely lose entities, collapse noun phrases, drop predicate instances, or invert negation, yet most evaluation still relies on downstream tasks or costly manual checks. This paper introduces KGCQual, a sentence-level score that asks how close an extractor’s triples come to an “ideal” reference graph built from the same sentence by part-of-speech tagging and dependency parsing. The score splits into a noun component (completeness, resolution, connectivity, with a graded penalty for polarity mismatches) and a verb component (multiplicity and lexical/semantic similarity). Across classical OpenIE tools and LLM extractors on WebNLG, TinyButMighty and BenchIE, lower KGCQual scores flag the systems that better preserve structure; gold triples sit near the bottom of the scale; and the ranking correlates strongly with how well a link-prediction model later performs on the same graphs. The result is a transparent, extractor-agnostic diagnostic that can be run before triples ever enter a knowledge base.

What carries the argument

KGCQual score M = α MN + (1−α) MV, where MN normalizes deviations in entity completeness, node resolution and connectivity (with Jaro–Winkler + UD-negation matching), and MV normalizes deviations in predicate multiplicity and lexical/embedding similarity; both are computed against a POS- and dependency-derived ideal graph.

What would settle it

Construct a controlled set of sentences whose POS/dependency ideal graph deliberately mismatches human-annotated gold triples on entity boundaries or negation, recompute KGCQual, and check whether the correlation with link-prediction MRR collapses.

Watch

Extended reading notes

Core claim

KGCQual is an intrinsic, model-agnostic metric that quantifies extraction fidelity by measuring how closely an automatically produced triple graph approximates an ideal reference graph of the source sentence’s noun phrases, predicate instances (including multiplicity) and negation polarity; the resulting score reliably ranks extractors and correlates significantly with downstream link-prediction performance on the same graphs.

Load-bearing premise

The claim rests on treating a graph built only from POS tags and dependency parses as a faithful enough stand-in for the sentence’s intended entities, relations and polarity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real contribution here is a concrete, fully specified intrinsic score that ranks OpenIE and LLM extractors by how well their triples match a POS/dependency ideal of the source sentence. That fills a practical gap: most KG-quality work starts after triples already exist, and most OpenIE evals stay at gold-triple F1 or downstream tasks. They give explicit formulas for noun completeness/resolution/connectivity and verb multiplicity/similarity (with a light UD-negation penalty), ship code, run eight systems on three public sets, show gold lower bounds, do a clean noun/verb ablation, and include readable case studies. That package is useful for anyone building or comparing extraction pipelines.

What works: the metric is model-agnostic, decomposable, and easy to interpret. ClausIE consistently looks best, MinIE worst, LLMs in the middle—matching the qualitative examples. The ablation correctly flags predicate fidelity as the harder problem. Gold triples sit well below the automated systems, so the scale is not completely unanchored.

Soft spots, in proportion. The ideal graph is only as good as NLTK POS + dependency parsing; the authors themselves flag this in Limitations and note that deep semantic coherence is out of scope. Gold scores never reach zero, which is honest but means the oracle itself is noisy. The headline Spearman ρ = −0.900 (p = 0.037) is the weakest claim: it appears only after aggregating three datasets, restricting to five classical systems, dropping ClausIE on BenchIE, and using only TuckER. Their own Table 11 shows the per-dataset picture is mixed or non-significant. That does not kill the paper; it just means the “downstream utility” sentence needs tempering. Free parameters (α, λ, Likert cut-offs) are fixed without sensitivity analysis, but that is minor.

This is for Semantic Web and OpenIE practitioners who need a cheap, sentence-level diagnostic before they dump triples into a KG. It is not a theoretical breakthrough, but it is reproducible engineering that a serious editor should send to referees. I would cite the metric definition and the multi-system tables; I would not lean on the aggregated correlation until it is re-run with more systems and per-dataset reporting. Worth engaging.

Referee Report

3 major / 0 minor

Summary. The paper proposes KGCQual, an intrinsic sentence-level metric for assessing the quality of triples produced by automated information-extraction systems before they are integrated into a knowledge graph. An “ideal” reference graph is constructed from POS tags and dependency parses; extracted graphs are then scored by a noun metric (completeness, resolution, connectivity, with a lightweight negation penalty) and a verb metric (predicate multiplicity and lexical/embedding similarity). The composite score M = α MN + (1−α) MV is evaluated on eight extractors (five classical OpenIE tools and three LLMs) across WebNLG, TinyButMighty and BenchIE, with gold-triple lower bounds, an ablation of noun/verb weights, case studies, and a downstream TuckER link-prediction correlation (aggregated Spearman ρ = −0.900, p = 0.037). Code is released.

Significance. If the metric is shown to be a reliable, model-agnostic proxy for extraction fidelity, it would fill a genuine gap between post-hoc KG quality frameworks and the practical realities of pipeline-driven construction. The explicit formulas, public code, multi-system multi-dataset evaluation, ablation, and attempt at downstream validation are concrete strengths that make the work usable and falsifiable. The central claim that KGCQual “correlates significantly with link-prediction performance” is, however, currently overstated relative to the supporting evidence; once that claim is properly scoped, the paper still offers a useful diagnostic tool for the community.

major comments (3)

§5.6 and Table 6 present the headline Spearman ρ = −0.900 (p = 0.037) as “direct evidence that KGCQual reflects downstream utility.” Table 11 shows that this figure is obtained only after aggregation across three datasets, restriction to five classical systems, omission of ClausIE on BenchIE, and exclusive use of TuckER. Per-dataset correlations are non-significant or even positive (WebNLG ρ = +0.400, p = 0.505; BenchIE ρ = +0.400, p = 0.600). With n ≈ 5 the aggregated p-value is fragile. The abstract and §5.6 must either (a) report the per-dataset numbers prominently and qualify the claim, or (b) supply a statistically more robust validation (larger n, inclusion of LLM extractors, sensitivity analysis). As written, the strongest claim is not yet secure.
§3.2–3.4 and Limitations §6 treat the POS+dependency “ideal” graph as a faithful proxy for intended entities, predicates and polarity. The same linguistic pipeline is used both to define the reference and (implicitly) by several of the classical extractors being ranked, creating a mild circularity risk that is never quantified. A small human-judged sample of ideal-graph fidelity, or an explicit comparison against gold triples beyond the aggregate lower bounds of Table 4, is needed to underwrite the claim that deviations from Gi measure true extraction quality rather than parser idiosyncrasies.
The free parameters α = 0.5 and λ = 0.3 are fixed without sensitivity analysis beyond the five discrete weightings of the ablation (Table 5 / Appendix Table 7). Because the final ranking of systems and the Likert thresholds of §4.5 depend on these choices, a short sensitivity sweep (or a clear statement that rankings are stable under reasonable ranges) is required for the metric to be reproducible by others.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: KGCQual is defined from independent linguistic proxies and validated externally against gold triples and link-prediction MRR.

full rationale

The metric construction (ideal graph Gi from POS tags + UD dependency parses, noun metric MN via completeness/resolution/connectivity + negation-aware JW, verb metric MV via multiplicity + JW/embedding similarity, final M = 0.5 MN + 0.5 MV) is fully specified in §3 without reference to any extractor outputs, gold triples, or downstream embeddings. Evaluation then applies this fixed definition to external systems (MinIE, OllIE, Stanford OpenIE, ClausIE, Claude/Gemini/GPT) on public datasets (WebNLG, TinyButMighty, BenchIE), compares against human gold triples (Table 4), performs ablation on component weights (Table 5/7), and measures Spearman correlation to an independent TuckER link-prediction task (Table 6). No parameter is fitted to the correlation target; α=0.5 and λ=0.3 are stated constants. No self-citation supplies a uniqueness theorem or ansatz that forces the result. The correlation’s statistical fragility (small n, selective aggregation, model choice) is a robustness concern outside the circularity criteria. The derivation chain therefore contains no step that reduces a claimed prediction or first-principles claim to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on a small set of hand-chosen constants, standard NLP tooling treated as oracles, and the invented composite score itself. No deep physical or mathematical axioms are required; the load-bearing premises are engineering choices about what “ideal” means and how heavily to penalize polarity and multiplicity errors.

free parameters (3)

α (noun/verb weight) = 0.5
Fixed at 0.5 so entity and relation quality are equally important; no data-driven selection or sensitivity analysis beyond discrete ablation points.
λ (negation polarity penalty) = 0.3
Fixed at 0.3 to down-weight Jaro–Winkler when UD negation status mismatches; chosen by hand “to assign a reasonable penalty”.
Likert grade thresholds = 0.25 / 0.45
Grade 1 ≤ 0.25, intermediate 0.25–0.45, Grade 5 ≥ 0.45, anchored to observed gold-triple scores; post-hoc interpretive mapping, not derived.

assumptions (4)

domain assumption An ideal sentence-level graph can be approximated by treating all POS-derived noun phrases as distinct nodes and all dependency-derived verb instances (with multiplicity and UD negation) as edges forming one connected component.
Stated in §3.2 Ideal Graph Construction; the entire metric is defined as distance to this proxy.
domain assumption Jaro–Winkler similarity (and max with embedding cosine for verbs) plus a constant polarity penalty adequately measures lexical/semantic fidelity of extracted nouns and predicates.
§3.3–3.4; replaces exact string match with graded similarity.
domain assumption NLTK POS tagging and Universal Dependencies parses are sufficiently accurate oracles for constructing the ideal graph on the evaluation sentences.
Experimental protocol §4.2; errors in the oracle become false ground truth.
ad hoc to paper Normalization of deviation scores by 2N(Gi) and 2Vb(Gi) yields a comparable [0,1] fidelity measure across sentences of different length.
§3.3–3.4 formulas; the factor of 2 is chosen so the theoretical maximum of the unnormalized sum maps to 1.

invented entities (2)

KGCQual composite metric M = α MN + (1−α) MV independent evidence
purpose: Single scalar that ranks extractors by structural and lexical fidelity to a sentence-level ideal graph.
The paper’s primary contribution; defined in §3.5 and used for all rankings and the downstream correlation.
Ideal reference graph Gi
purpose: Unobserved target structure against which extracted graphs G1 are scored.
Constructed from POS and dependency cues (§3.2); not independently observed, only approximated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text." pith.science (2026). https://pith.science/paper/PT25N3AE

@misc{pith2026260710212,
  author       = {Pith},
  title        = {Pith review of: KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PT25N3AE}},
  note         = {Machine review of arXiv:2607.10212}
}

read the original abstract

Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.

Figures

Figures reproduced from arXiv: 2607.10212 by the authors.

**Figure 3.** Triple graph extracted by GPT based model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references

[1]

Angeli, G., Premkumar, M.J., Manning, C.: Stanford openie (version 4.x) system description. Tech. rep., Stanford NLP (2015), technical Report / System Demo

2015
[2]

In: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP-IJCNLP)

Balažević, I., Allen, C., Hospedales, T.M.: Tucker: Tensor factorization for knowl- edge graph completion. In: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP-IJCNLP). pp. 5185–5194 (2019)

2019
[3]

Computational Linguistics21(4) (1995) KGCQual 17

Brill, E.: Transformation-based error-driven learning and natural language pro- cessing: A case study in part-of-speech tagging. Computational Linguistics21(4) (1995) KGCQual 17

1995
[4]

ACL (2009)

Dagan, I., Dolan, B., Magnini, B.: Recognizing textual entailment. ACL (2009)

2009
[5]

In: Proceedings of WWW 2013

Del Corro, L., Gemulla, R.: Clausie: Clause-based open information extraction. In: Proceedings of WWW 2013. pp. 355–366 (2013)

2013
[6]

Semantic Web9(1), 77–129 (2018)

Färber, M., Bartscherer, F., Menne, C., Rettinger, A.: Linked data quality of db- pedia, freebase, opencyc, wikidata, and yago. Semantic Web9(1), 77–129 (2018)

2018
[7]

In: IJCAI (2011)

Gabriel, R., Soderland, S., Etzioni, O.: Open information extraction from the web. In: IJCAI (2011)

2011
[8]

In: Proceedings of the WebNLG Challenge (2017)

Gardent, C., Shimorina, A., Narayan, S., Perez-Beltrachini, L.: The webnlg chal- lenge: Generating text from rdf data. In: Proceedings of the WebNLG Challenge (2017)

2017

Show all 18 references

[9]

In: Proceedings of EMNLP

Gashteovski, K., Gemulla, R., Del Corro, L.: Minie: Minimizing facts in open in- formation extraction. In: Proceedings of EMNLP. pp. 2630–2640 (2017)

2017
[10]

In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) (2022)

Gashteovski, K., Yu, M., Kotnis, B., Lawrence, C., Niepert, M., Glavaš, G.: Benchie: A framework for multi-faceted fact-based open information extraction evaluation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) (2022)

2022
[11]

ACM Computing Surveys54(4), 1–37 (2021)

Hogan, A., Blomqvist, E., Cochez, M., D’Amato, C., de Melo, G., Gutiérrez, C., Kirrane, S., Neumaier, S., Polleres, A., Prince, V., Škoda, P., Lange, C., Zimmer- mann, A.: Knowledge graphs. ACM Computing Surveys54(4), 1–37 (2021)

2021
[12]

In: Proceedings of the Transformer Circuits Interpretability Workshop (TCIW) 2024 (2024)

Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Nichols, L.: On the biology of a large language model. In: Proceedings of the Transformer Circuits Interpretability Workshop (TCIW) 2024 (2024)

2024
[13]

In: Proceedings of ACL 2012 (2012)

Mausam, Schmitz, H.P., Etzioni, O.: Ollie: Learning open information extraction from question-answer pairs. In: Proceedings of ACL 2012 (2012)

2012
[14]

In: Proceedings of ACL (2024)

Patel, S., Gomez, L.: Comparison of llms in extracting synthesis conditions: Gemini excels in completeness. In: Proceedings of ACL (2024)

2024
[15]

Roy Choudhury, S., Dey, Z.: Gpt-3 powered information extraction for building robust knowledge bases (2024)

2024
[16]

In: Proceedings of the 20th Joint ACL–ISO Workshop on Interoperable Semantic Annotation @ LREC-COLING

Salman, M., Haller, A., Rodriguez Mendez, S.J., Naseem, U.: Tiny but mighty: A crowdsourced benchmark dataset for triple extraction from unstructured text. In: Proceedings of the 20th Joint ACL–ISO Workshop on Interoperable Semantic Annotation @ LREC-COLING. pp. 71–81. ELRA an...

2024
[17]

In: Proceedings of the IEEE International Conference on Data Engineering (ICDE) (2020)

Smith, A.B., Miller, P.R., Wilson, H.: Efficient knowledge graph accuracy evalua- tion. In: Proceedings of the IEEE International Conference on Data Engineering (ICDE) (2020)

2020
[18]

Semantic Web7(1), 63–93 (2016) A Ablation Study: Full Results Table 7 extends Table 5 with all five weighting scenarios

Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for linked data: A survey. Semantic Web7(1), 63–93 (2016) A Ablation Study: Full Results Table 7 extends Table 5 with all five weighting scenarios. Lower scores indicate higher extracti...

2016

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

Angeli, G., Premkumar, M.J., Manning, C.: Stanford openie (version 4.x) system description. Tech. rep., Stanford NLP (2015), technical Report / System Demo

2015

[2] [2]

In: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP-IJCNLP)

Balažević, I., Allen, C., Hospedales, T.M.: Tucker: Tensor factorization for knowl- edge graph completion. In: Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP-IJCNLP). pp. 5185–5194 (2019)

2019

[3] [3]

Computational Linguistics21(4) (1995) KGCQual 17

Brill, E.: Transformation-based error-driven learning and natural language pro- cessing: A case study in part-of-speech tagging. Computational Linguistics21(4) (1995) KGCQual 17

1995

[4] [4]

ACL (2009)

Dagan, I., Dolan, B., Magnini, B.: Recognizing textual entailment. ACL (2009)

2009

[5] [5]

In: Proceedings of WWW 2013

Del Corro, L., Gemulla, R.: Clausie: Clause-based open information extraction. In: Proceedings of WWW 2013. pp. 355–366 (2013)

2013

[6] [6]

Semantic Web9(1), 77–129 (2018)

Färber, M., Bartscherer, F., Menne, C., Rettinger, A.: Linked data quality of db- pedia, freebase, opencyc, wikidata, and yago. Semantic Web9(1), 77–129 (2018)

2018

[7] [7]

In: IJCAI (2011)

Gabriel, R., Soderland, S., Etzioni, O.: Open information extraction from the web. In: IJCAI (2011)

2011

[8] [8]

In: Proceedings of the WebNLG Challenge (2017)

Gardent, C., Shimorina, A., Narayan, S., Perez-Beltrachini, L.: The webnlg chal- lenge: Generating text from rdf data. In: Proceedings of the WebNLG Challenge (2017)

2017