Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Populating Web Scale Knowledge Graphs using Distantly Supervised Relation Extraction and Validation

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training a knowledge-base-completion validator on the noisy output of distantly supervised relation extraction and combining the scores roughly halves extraction error across three knowledge-graph benchmarks.

desk verdict A sensible pipeline idea whose headline AUC gains are invalid because the validator is trained and tested on the same extracted triples. read the letter →

arxiv 1908.08104 v2 pith:Z5PDQFNA submitted 2019-08-21 cs.CL

Sarthak Dash , Michael R. Glass , Alfio Gliozzo , Mustafa Canim This is my paper · ORCID
classification cs.CL
keywords distantsupervisionrelationextractionknowledgebasevalidationcompletionconfidencere-estimationProjEsoftmaxgraphpopulationweb-scalecorpora
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that relation extraction can be made much more accurate by adding a second, global validation step: a knowledge-base-completion model trained on the noisy extracted triples themselves. If true, knowledge graphs could be extended from web text without hand-labeled data, domain-specific rules, or language adaptation. On three benchmarks the combined pipeline lifts precision-recall AUC from 0.499 to 0.630, from 0.294 to 0.785, and from 0.872 to 0.966, which the authors describe as roughly 50% error reduction and up to 100% relative improvement. The mechanism is a soft-label loss that lets a KBC network learn from confidence-scored extractions rather than clean one-hot facts.

What carries the argument

The central object is a modified knowledge-base-completion loss. Equation (2) replaces the one-hot target vector used to train a ProjE softmax network with a confidence vector whose entries are the IE scores of extracted quads. The network therefore learns to project entity-relation pairs onto object distributions using the global structure of the induced, noisy graph. A final logistic-regression layer aggregates the IE score, the KBC score from the curated graph, and the KBC score from the extracted graph, using raw, logit, binned, relation-indicator, and pairwise quadratic features.

What would settle it

Split QIE before training KBVIE: train on one subset, compute precision-recall AUC on the other subset that the IE system produced but the validator never saw. If the gap between the IE-only curve and the full pipeline shrinks substantially, the reported gains are largely memorization rather than validation. Also compare predictions on triples whose entities appear in the original knowledge graph with those whose entities appear only in QIE.

Watch

Extended reading notes

Core claim

The paper argues that relation extraction and knowledge-base validation are two sides of the same operation and should be coupled: distantly supervised IE produces candidate triples with confidence scores, then a KBC-style network trained on those very candidates re-scores each triple using the global structure of the induced graph. The authors report that on NYT-FB this raises precision-recall AUC from 0.499 to 0.630, on CC-DBP from 0.294 to 0.785, and on NELL from 0.872 to 0.966, with the largest gains concentrated on triples whose entities are well connected. They also find that the validator trained on noisy IE output outperforms one trained on the original curated KG, and that the full system beats a probabilistic-logic validator on NELL without needing ontology constraints.

Load-bearing premise

The evaluation assumes that a KBC model trained on the extracted triples and then used to score those same triples gives an honest accuracy estimate; no held-out split of the extraction output is described, so if the model memorizes training triples, the AUC gains would not reflect generalization to newly extracted facts.

Editorial extensions

If this is right

  • A knowledge graph can be extended from unlabeled text without hand-labeled relation examples or hand-written rules, because the validator learns from the IE system's own confidence-scored output.
  • The same pipeline should transfer across languages and domains with no adaptation cost, since entity recognition is dictionary matching and nothing else depends on language-specific analytics.
  • Validation gains concentrate on well-connected entities, so the system becomes more reliable precisely where a knowledge graph is already rich.
  • The proposed validation can score triples containing entities not present in the original knowledge graph, which standard KBC models cannot do.
  • On the NELL benchmark the combination reaches 0.966 AUC and outperforms a probabilistic-soft-logic alternative that requires ontology constraints and validated triples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gain survives a held-out test, a natural extension is to run the validator generatively and propose triples the IE system never extracted; the authors list this as future work, but the learned scorer already has that capability.
  • Because precision is judged by membership in the original knowledge base, which is a positive-unlabeled setup, the true precision at fixed recall is probably higher than the reported curves; gains on facts already in DBpedia or Freebase may overstate or understate performance on genuinely new facts.
  • The confidence-vector loss is architecture-agnostic: any KBC network that accepts object distributions could be trained on noisy extraction output, so the validation idea is not tied to ProjE specifically.
  • The cleanest use of the final scores is to rank candidate facts for human review, since nothing in the method produces calibrated probabilities and the reported AUC measures ranking, not calibration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage pipeline for populating knowledge graphs: a distantly supervised deep relation extraction (RE) system that produces noisy triples with confidence scores, followed by a knowledge base validation (KBV) component based on a ProjE-style embedding model modified to train on noisy confidence labels (KBVIE). The confidence scores from RE, KBV, and KBVIE are combined by a logistic regression layer. The system is evaluated on NYT-FB, CC-DBP, and NELL, with the central claim that the validation step substantially boosts RE accuracy, reporting AUC gains such as 0.499 to 0.630 on NYT-FB and 0.294 to 0.785 on CC-DBP.

Significance. If the reported gains were valid, the paper would make a useful practical contribution: it would show that a KBC model trained on noisy IE output can re-rank extractions using global graph structure, and that the approach scales to web-scale corpora. The confidence-weighted loss in Eq. (2) is a sensible adaptation of ProjE to noisy supervision, and the CC-DBP experiment addresses an important scalability question. However, the evaluation protocol does not support the central claim: the validator is trained and tested on the same extracted triples, and for NYT-FB and CC-DBP the KBV model also has the ground-truth answer key in its training set. The headline numbers are therefore not a valid measure of generalization to newly extracted triples.

major comments (4)
  1. [Section 4.2 / Eq. (2)] KBVIE is trained on QIE and then evaluated on the same QIE without any described held-out split. Equation (2) optimizes the network to predict the confidence vector s defined on QIE; Section 4.2 states "In all cases, KBVIE is trained on the output of the IE systems. Then we apply both systems to validate QIE." The precision-recall curves in Figures 4-6 and the AUC gains in Table 2 (e.g., 0.294 to 0.785 on CC-DBP) therefore compare the model's scores on triples whose confidence labels were used during training. This memorization channel invalidates the central claim that validation boosts RE accuracy for newly extracted triples.
  2. [Section 4.2, Table 2] For NYT-FB and CC-DBP, the KBV model's training set is described as the intersection of QIE with KBtrain. Since the ground-truth label for a triple in these benchmarks is presence in Freebase or DBpedia, every positive test triple in QIE is also in KBtrain and hence in KBV's training data. The AUC gains attributed to KBV (0.609 and 0.636) can thus reflect memorization of the answer key rather than validation of unseen triples.
  3. [Section 3.3] The confidence re-estimation layer is said to be "typically trained on a validation set separate from the training set, but it can also use the training set itself." The paper does not report which option was used for each benchmark, nor whether this validation set overlaps QIE. If the final logistic regression is trained on the same triples whose labels define the precision-recall curve, then the combined scores in Table 2 are not an unbiased estimate of generalization.
  4. [General evaluation methodology] No error bars, variance estimates, or significance tests are reported for any of the AUC values in Table 2. Given the magnitude of the CC-DBP jump (0.294 to 0.785) and the absence of a held-out split, it is impossible to determine how much of the gain is due to the method rather than to training on the test set. A valid evaluation would require reporting AUC on a held-out portion of QIE or on newly extracted triples not used to train KBVIE or the re-estimation layer.
minor comments (5)
  1. [Abstract / Section 4.2] The abstract says "relative improvement of up to 100%," but Section 4.2 reports a relative improvement of 167% on CC-DBP; these statements should be reconciled.
  2. [Section 4.2] The text contains a typo: "derived by the riples validated by humans" should read "triples."
  3. [Table 1] The column header "|KB KB|" appears duplicated; it should likely be "|KB|".
  4. [Section 3.2] The sentence "The network is trained for each triplet in the training data" uses "triplet" where "triple" is the standard term used elsewhere in the paper.
  5. [Section 3.2] Hyperparameters for the ProjE variant (embedding dimension, number of epochs, learning rate, negative sample count) are not reported; these details are needed for reproducibility of the KBVIE and KBV results.

Circularity Check

1 steps flagged · score 8.0 of 10

The reported AUC gains are not demonstrated because KBVIE is trained on QIE and then used to score the same QIE, with no held-out split; the validation scores are fitted to the very confidence scores they purport to re-assess.

  1. fitted input called prediction [Section 3.2, Eqs. (1)-(2); Section 4.2]
    "This network considers a set of quads QIE as the probabilistic knowledge graph for training and learns a function KBVIE (⟨e1,r,e 2⟩) that returns a confidence score s for the triple at hand. ... In all cases, KBVIE is trained on the output of the IE systems. Then we apply both systems to validate QIE , the output of the IE system, generating two additional confidence scores."

    Equation (2) defines the training loss as the cross-entropy between the confidence vector s, built from the quads in QIE via Eq. (1), and the network's prediction. Thus KBVIE's parameters are fit to reproduce the IE confidence scores of the triples in QIE. The evaluation then applies KBVIE to that same QIE and reports precision/recall over those triples. A score produced by a model trained on a triple's own confidence signal is not an independent validation of that triple; the 'prediction' is a re-scoring of training data. The reported improvement of KBVIE over IE could therefore reflect memorization of the input scores rather than generalization to newly extracted triples.

full rationale

The paper's central load-bearing claim, that combining KBC validation with distantly supervised RE boosts RE accuracy, rests on an evaluation protocol in which the validation model is trained and tested on the same extracted triple set. Section 3.2 states that KBVIE treats QIE as the training knowledge graph and Eq. (2) uses the confidence scores from QIE as targets; Section 4.2 then applies the trained KBVIE to validate QIE. No held-out split of QIE is described for KBVIE, so the PR curves and AUC values in Table 2 measure performance on training triples. For NYT-FB and CC-DBP, KBV has additional leakage because its training set is the intersection of QIE with KBtrain, meaning positive test triples appear in its training data. The final logistic regression is also permitted to train on the training set itself (Section 3.3). While the system is otherwise self-contained and uses standard benchmarks, the core comparison of IE versus IE+KBVIE is not an independent test of the validation step; it is a fitted-input-called-prediction setup. This warrants a high circularity score, though not the maximum, because some components (e.g., NELL manually validated data and the RE baseline itself) are externally grounded.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system introduces one new model component (KBVIE) but pulls all core algorithms from cited prior work. The main load-bearing assumptions are the distant supervision labeling rule, the local closed world assumption for KBC, positive-unlabeled evaluation, and the unstated validity of scoring the training set.

free parameters (3)
  • IE confidence threshold for quad filtering
    Section 3.1 says a confidence threshold is chosen below which quads are discarded; the value is not reported, yet it determines QIE size and the PR curve's operational range.
  • Minimum entity triple count for KBC embeddings = 3
    Section 3.2 states embeddings are trained only when an entity appears in at least three triples; this cutoff affects which triples get KBVIE and KBV scores.
  • KBC hyperparameters (embedding dimension, epochs, learning rate, negative sample count)
    ProjE settings are cited to prior work, but the actual configuration used for KBVIE and KBV is unspecified; these choices influence validation scores.
assumptions (4)
  • domain assumption Distant supervision assumption: if two entities co-occur in a sentence and are related in the input KG, the sentence expresses that relation.
    Section 3.1 uses KG triples to label co-occurring entity pairs as positive examples; this is the standard noisy-label assumption of distant supervision.
  • domain assumption Local closed world assumption for KBC training: for a given subject and relation, only the observed objects are positive; corrupted objects are negatives.
    Section 3.2 says KBC exploits a Local Closed World Assumption; negatives are produced by random object corruption.
  • domain assumption For NYT-FB and CC-DBP, absence of a triple in Freebase or DBpedia is treated as negative.
    Section 4.1 calls this positive-unlabeled evaluation and notes precision is underestimated; the evaluation relies on this assumption.
  • domain assumption Exact string matching of entity mentions is sufficient to identify entities in text.
    Section 3 says entity detection and linking is implemented by simple string match with respect to KB entities; no disambiguation is performed, which can create false entity links.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Populating Web Scale Knowledge Graphs using Distantly Supervised Relation Extraction and Validation." pith.science (2026). https://pith.science/paper/Z5PDQFNA

@misc{pith2026190808104,
  author       = {Pith},
  title        = {Pith review of: Populating Web Scale Knowledge Graphs using Distantly Supervised Relation Extraction and Validation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z5PDQFNA}},
  note         = {Machine review of arXiv:1908.08104}
}
read the original abstract

In this paper, we propose a fully automated system to extend knowledge graphs using external information from web-scale corpora. The designed system leverages a deep learning based technology for relation extraction that can be trained by a distantly supervised approach. In addition to that, the system uses a deep learning approach for knowledge base completion by utilizing the global structure information of the induced KG to further refine the confidence of the newly discovered relations. The designed system does not require any effort for adaptation to new languages and domains as it does not use any hand-labeled data, NLP analytics and inference rules. Our experiments, performed on a popular academic benchmark demonstrate that the suggested system boosts the performance of relation extraction by a wide margin, reporting error reductions of 50%, resulting in relative improvement of up to 100%. Also, a web-scale experiment conducted to extend DBPedia with knowledge from Common Crawl shows that our system is not only scalable but also does not require any adaptation cost, while yielding substantial accuracy gain.

Figures

Figures reproduced from arXiv: 1908.08104 by the authors.

Figure 1
Figure 1. Link Prediction and Knowledge Base Validation example. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline for our RE solution. 2 Related Work Deep learning has been widely explored for the task of information extraction. Both CNN-based [Zeng et al., 2014] and LSTM-based [Xu et al., 2015] models have been trained successfully for RE. Recently, cross sentence approaches have been explored by building paths connecting the two identified arguments through related entities [Zeng et al., 2016]. The context aggregatio… view at source ↗
Figure 3
Figure 3. Base ProjE softmax architecture for KBC. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Precision / Recall curves for CC-DBP 3 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 Precision Recall (of extracted triples) IE IE+KBC IE+KBCIE IE+KBC+KBCIE [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [4]

    Deep convolutional ranking for multilabel image annotation

    Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4894,

    arXiv

  2. [7]

    Holographic embeddings of knowledge graphs

    Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. Holographic embeddings of knowledge graphs. In AAAI, pages 1955–1961,

    work page 1955

  3. [8]

    Relation extraction with matrix factorization and universal schemas

    Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74–84,

    work page 2013

  4. [9]

    Multi-instance Multi-label Learning for Relation Extraction

    Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. Multi-instance Multi-label Learning for Relation Extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 455–465. Association for Computational Linguistics,

    work page 2012

  5. [12]

    Incorporating relation paths in neural relation extraction

    Wenyuan Zeng, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Incorporating relation paths in neural relation extraction. arXiv preprint arXiv:1609.07479,

    arXiv

  6. [2010]

    Convolutional 2d knowledge graph embeddings

    Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. arXiv preprint arXiv:1707.01476,

    arXiv

  7. [2011]

    Learning to refine an automatically extracted knowledge base using markov logic

    Shangpu Jiang, Daniel Lowd, and Dejing Dou. Learning to refine an automatically extracted knowledge base using markov logic. In Proceedings of the 2012 IEEE 12th International Conference on Data Mining , ICDM ’12, pages 912–917, Washington, DC, USA,

    work page 2012

  8. [2012]

    Classifying relations via long short term memory networks along shortest dependency paths

    Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1785–1794,

    work page 2015

Show all 12 references
  1. [2013]

    Kbgan: Adversarial learning for knowledge graph embeddings

    Liwei Cai and William Yang Wang. Kbgan: Adversarial learning for knowledge graph embeddings. arXiv preprint arXiv:1711.04071,

    arXiv

  2. [2015]

    Relation classification via convolutional deep neural network

    Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344,

    2014

  3. [2016]

    A novel embedding model for knowledge base completion based on convolutional neural network

    Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based on convolutional neural network. arXiv preprint arXiv:1712.02121,

    arXiv

  4. [2018]

    Inducing implicit relations from text using distantly supervised deep nets

    Michael Glass, Alfio Gliozzo, Oktie Hassanzadeh, Nandana Mihindukulasooriya, and Gaetano Rossiello. Inducing implicit relations from text using distantly supervised deep nets. In ISWC 2018,

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.