{"id":"b7ee3e27-8dd9-464f-861d-95e90cb419fa","arxiv_id":"1908.08104","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A relation-extraction pipeline trains a knowledge-base completion model on its own noisy output to re-rank candidate facts, claiming large accuracy gains that the evaluation setup does not support.","lead":"This paper describes an automated pipeline that finds new facts in large text collections and then re-scores them using a knowledge-graph model, with the goal of extending knowledge graphs like DBpedia. The authors report large accuracy gains, but the evaluation lets the re-scoring model train on the same facts it is asked to judge, so the gains are not yet convincing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported validation gains are not demonstrated because KBVIE is trained on QIE and then evaluated on the same QIE with no held-out split; the Table 2 AUCs can reflect fitting the test set.","rationale":"I read the paper as a systems paper whose central claim is empirical: combining distantly supervised RE with a KBC validator trained on IE output improves RE accuracy. That claim requires that the validator's scores be evaluated on triples it was not trained on. The paper does not describe such a split; instead, the stated protocol trains and evaluates KBVIE on the same QIE. This is the same load-bearing concern the reader identified. The paper's components are reasonable and the connectivity analysis in Section 4.3 is suggestive, but the headline numbers in Table 2 are not trustworthy under this protocol. I do not see a separate concern that would change the verdict; the evaluation flaw is sufficient to reject the paper's central empirical claim. This is an experimental-design issue that could be fixed by rerunning with held-out QIE splits, but as presented the evidence for the main claim is absent.","tokens_in":9994,"tokens_out":7624,"duration_ms":77347,"concrete_test":"Rerun the NYT-FB experiment with a held-out split of QIE: train KBVIE on, say, 80% of QIE stratified by relation and by KB-positive/negative status, then score the remaining 20% to compute IE+KBVIE and ALL AUC. If the improvement over IE shrinks substantially (or the ALL AUC drops from 0.630 toward 0.499), the Table 2 gains are an artifact of training on the test set. A stronger variant uses entity-disjoint splits so each test triple has at least one entity absent from KBVIE's training triples, forcing the KBC model to rely on embeddings learned from other triples; the same check should be applied to the CC-DBP and NELL experiments.","verdict_should_be":"REJECT","load_bearing_attack":"The central systems claim—that KBC-based validation boosts RE accuracy (Table 2: NYT-FB 0.499→0.630, CC-DBP 0.294→0.785, NELL 0.872→0.966)—is not supported by the evaluation protocol, because the validator is trained and tested on the same extracted triples. Section 3.2 states that KBVIE 'considers a set of quads QIE as the probabilistic knowledge graph for training'; Section 4.2 states 'In all cases, KBVIE is trained on the output of the IE systems. Then we apply both systems to validate QIE.' No held-out split of QIE is described. The loss in Eq. (2) trains the model to output the observed confidence vector s for each (e1,r,e2) in QIE; scoring the PR curve over the same QIE means the model's scores for test triples are computed from parameters fit on those triples and their confidence scores. For NYT-FB and CC-DBP, KBV has an additional form of leakage: its training set is the intersection of QIE with KBtrain, so positive test triples are in its training data. Section 3.3 also permits the final logistic regression to be trained 'on the training set itself.' Consequently, the reported AUC gains and the 'error reductions of 50%' do not establish that the validation step improves accuracy on newly extracted triples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for populating knowledge graphs: a distantly supervised deep relation extraction (RE) system that produces noisy triples with confidence scores, followed by a knowledge base validation (KBV) component based on a ProjE-style embedding model modified to train on noisy confidence labels (KBVIE). The confidence scores from RE, KBV, and KBVIE are combined by a logistic regression layer. The system is evaluated on NYT-FB, CC-DBP, and NELL, with the central claim that the validation step substantially boosts RE accuracy, reporting AUC gains such as 0.499 to 0.630 on NYT-FB and 0.294 to 0.785 on CC-DBP.","tokens_in":10354,"tokens_out":2962,"duration_ms":27952,"significance":"If the reported gains were valid, the paper would make a useful practical contribution: it would show that a KBC model trained on noisy IE output can re-rank extractions using global graph structure, and that the approach scales to web-scale corpora. The confidence-weighted loss in Eq. (2) is a sensible adaptation of ProjE to noisy supervision, and the CC-DBP experiment addresses an important scalability question. However, the evaluation protocol does not support the central claim: the validator is trained and tested on the same extracted triples, and for NYT-FB and CC-DBP the KBV model also has the ground-truth answer key in its training set. The headline numbers are therefore not a valid measure of generalization to newly extracted triples.","major_comments":[{"comment":"KBVIE is trained on QIE and then evaluated on the same QIE without any described held-out split. Equation (2) optimizes the network to predict the confidence vector s defined on QIE; Section 4.2 states \"In all cases, KBVIE is trained on the output of the IE systems. Then we apply both systems to validate QIE.\" The precision-recall curves in Figures 4-6 and the AUC gains in Table 2 (e.g., 0.294 to 0.785 on CC-DBP) therefore compare the model's scores on triples whose confidence labels were used during training. This memorization channel invalidates the central claim that validation boosts RE accuracy for newly extracted triples.","section":"Section 4.2 / Eq. (2)"},{"comment":"For NYT-FB and CC-DBP, the KBV model's training set is described as the intersection of QIE with KBtrain. Since the ground-truth label for a triple in these benchmarks is presence in Freebase or DBpedia, every positive test triple in QIE is also in KBtrain and hence in KBV's training data. The AUC gains attributed to KBV (0.609 and 0.636) can thus reflect memorization of the answer key rather than validation of unseen triples.","section":"Section 4.2, Table 2"},{"comment":"The confidence re-estimation layer is said to be \"typically trained on a validation set separate from the training set, but it can also use the training set itself.\" The paper does not report which option was used for each benchmark, nor whether this validation set overlaps QIE. If the final logistic regression is trained on the same triples whose labels define the precision-recall curve, then the combined scores in Table 2 are not an unbiased estimate of generalization.","section":"Section 3.3"},{"comment":"No error bars, variance estimates, or significance tests are reported for any of the AUC values in Table 2. Given the magnitude of the CC-DBP jump (0.294 to 0.785) and the absence of a held-out split, it is impossible to determine how much of the gain is due to the method rather than to training on the test set. A valid evaluation would require reporting AUC on a held-out portion of QIE or on newly extracted triples not used to train KBVIE or the re-estimation layer.","section":"General evaluation methodology"}],"minor_comments":[{"comment":"The abstract says \"relative improvement of up to 100%,\" but Section 4.2 reports a relative improvement of 167% on CC-DBP; these statements should be reconciled.","section":"Abstract / Section 4.2"},{"comment":"The text contains a typo: \"derived by the riples validated by humans\" should read \"triples.\"","section":"Section 4.2"},{"comment":"The column header \"|KB KB|\" appears duplicated; it should likely be \"|KB|\".","section":"Table 1"},{"comment":"The sentence \"The network is trained for each triplet in the training data\" uses \"triplet\" where \"triple\" is the standard term used elsewhere in the paper.","section":"Section 3.2"},{"comment":"Hyperparameters for the ProjE variant (embedding dimension, number of epochs, learning rate, negative sample count) are not reported; these details are needed for reproducibility of the KBVIE and KBV results.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The paper's central evaluation is invalid as reported because the validation models are trained and tested on the same QIE triples, with an additional leakage path for KBV on NYT-FB and CC-DBP. The reported AUC improvements are not credible evidence for the paper's main claim without a held-out evaluation. This is a load-bearing issue that cannot be fixed by local revisions; a new evaluation protocol would be required."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nYou should know before reading: the central claim is plausible but the evidence in this paper doesn't support it. The authors report that adding a KBC-based validator (KBVIE) to a distantly supervised RE system raises AUC from 0.499 to 0.630 on NYT-FB, 0.294 to 0.785 on CC-DBP, and 0.872 to 0.966 on NELL. But in every benchmark, KBVIE is trained on the same extracted triple set QIE that it is then evaluated against. Section 3.2 says the network 'considers a set of quads QIE as the probabilistic knowledge graph for training'; Section 4.2 says they apply the system to validate the same QIE. No held-out split of QIE is described. The loss in Eq. (2) is a cross-entropy between the model's prediction and the observed confidence vector for each triple in QIE, so the model is effectively fitting the labels it is later scored on. For NYT-FB and CC-DBP, KBV leaks as well: its training set is the intersection of QIE with KBtrain, which includes the positive test triples. The AUC gains are therefore not evidence that validation improves accuracy on newly extracted triples.\n\nThat said, the paper has real merit. The combination is new: applying ProjE-style KBC with a confidence-weighted soft-label loss to re-rank distantly supervised extractions, plus the three-way feature fusion (IE, KBV, KBVIE) and per-relation bias features. Training a KBC on the noisy output of an IE system is a natural idea and worth testing. The CC-DBP web-scale experiment is a useful test of scalability, and the NELL comparison to PSL is a fair point of reference. The writing is clear and the components are sensible.\n\nThe soft spots beyond the circularity are minor in comparison: no error bars, no stated hyperparameters for the KBC model, and the logistic regression 'can also use the training set itself' (Sec. 3.3), which suggests a casual attitude toward validation. The connectivity analysis in Table 3 does not repair the central issue because it uses the same flawed AUC.\n\nThis is a load-bearing flaw, not a quibble. The idea might work—I'd guess it does, at least for well-connected entities—but the experiments as presented don't show it. A proper evaluation would hold out triples from QIE, or better, validate on a completely new set of extractions. The authors could fix this and rerun; the contribution is worth taking seriously.\n\nWho is this for? Anyone working on KG population, distantly supervised RE, or KBC-based cleaning. I wouldn't cite it in its current form, but I would bring it to a reading group as a case study in evaluation leakage. It deserves peer review—not a desk rejection—but a reviewer should demand a clean held-out evaluation before publication.","headline":"A sensible pipeline idea whose headline AUC gains are invalid because the validator is trained and tested on the same extracted triples.","tokens_in":10869,"tokens_out":3140,"would_cite":false,"duration_ms":27902,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a knowledge-base-completion validator on the noisy output of distantly supervised relation extraction and combining the scores roughly halves extraction error across three knowledge-graph benchmarks.","keywords":["distant supervision","relation extraction","knowledge base validation","knowledge base completion","confidence re-estimation","ProjE softmax","knowledge graph population","web-scale corpora"],"falsifier":"Split QIE before training KBVIE: train on one subset, compute precision-recall AUC on the other subset that the IE system produced but the validator never saw. If the gap between the IE-only curve and the full pipeline shrinks substantially, the reported gains are largely memorization rather than validation. Also compare predictions on triples whose entities appear in the original knowledge graph with those whose entities appear only in QIE.","tokens_in":9830,"feed_emoji":"🕸️","tokens_out":4909,"duration_ms":46044,"temperature":0.7,"pith_summary":"The paper tries to show that relation extraction can be made much more accurate by adding a second, global validation step: a knowledge-base-completion model trained on the noisy extracted triples themselves. If true, knowledge graphs could be extended from web text without hand-labeled data, domain-specific rules, or language adaptation. On three benchmarks the combined pipeline lifts precision-recall AUC from 0.499 to 0.630, from 0.294 to 0.785, and from 0.872 to 0.966, which the authors describe as roughly 50% error reduction and up to 100% relative improvement. The mechanism is a soft-label loss that lets a KBC network learn from confidence-scored extractions rather than clean one-hot facts.","feed_headline":"Adding a KG validator halves relation-extraction error","feed_subtitle":"A confidence-trained completion model lifts web-scale DBpedia AUC from 0.294 to 0.785 without labeled data.","key_machinery":"The central object is a modified knowledge-base-completion loss. Equation (2) replaces the one-hot target vector used to train a ProjE softmax network with a confidence vector whose entries are the IE scores of extracted quads. The network therefore learns to project entity-relation pairs onto object distributions using the global structure of the induced, noisy graph. A final logistic-regression layer aggregates the IE score, the KBC score from the curated graph, and the KBC score from the extracted graph, using raw, logit, binned, relation-indicator, and pairwise quadratic features.","core_discovery":"The paper argues that relation extraction and knowledge-base validation are two sides of the same operation and should be coupled: distantly supervised IE produces candidate triples with confidence scores, then a KBC-style network trained on those very candidates re-scores each triple using the global structure of the induced graph. The authors report that on NYT-FB this raises precision-recall AUC from 0.499 to 0.630, on CC-DBP from 0.294 to 0.785, and on NELL from 0.872 to 0.966, with the largest gains concentrated on triples whose entities are well connected. They also find that the validator trained on noisy IE output outperforms one trained on the original curated KG, and that the full system beats a probabilistic-logic validator on NELL without needing ontology constraints.","pith_inferences":["If the gain survives a held-out test, a natural extension is to run the validator generatively and propose triples the IE system never extracted; the authors list this as future work, but the learned scorer already has that capability.","Because precision is judged by membership in the original knowledge base, which is a positive-unlabeled setup, the true precision at fixed recall is probably higher than the reported curves; gains on facts already in DBpedia or Freebase may overstate or understate performance on genuinely new facts.","The confidence-vector loss is architecture-agnostic: any KBC network that accepts object distributions could be trained on noisy extraction output, so the validation idea is not tied to ProjE specifically.","The cleanest use of the final scores is to rank candidate facts for human review, since nothing in the method produces calibrated probabilities and the reported AUC measures ranking, not calibration."],"forward_implications":["A knowledge graph can be extended from unlabeled text without hand-labeled relation examples or hand-written rules, because the validator learns from the IE system's own confidence-scored output.","The same pipeline should transfer across languages and domains with no adaptation cost, since entity recognition is dictionary matching and nothing else depends on language-specific analytics.","Validation gains concentrate on well-connected entities, so the system becomes more reliable precisely where a knowledge graph is already rich.","The proposed validation can score triples containing entities not present in the original knowledge graph, which standard KBC models cannot do.","On the NELL benchmark the combination reaches 0.966 AUC and outperforms a probabilistic-soft-logic alternative that requires ontology constraints and validated triples."],"supporting_citations":[{"why":"Supplies the distantly supervised deep relation-extraction system that generates the candidate quads QIE.","marker":"Glass et al. [2018]"},{"why":"Provides the PCNN+ATT model and the NYT-FB state-of-the-art baseline that the combined pipeline improves on.","marker":"Lin et al. [2016]"},{"why":"Contributes the ProjE softmax KBC architecture that is adapted into KBV and KBVIE.","marker":"Shi and Weninger [2017]"},{"why":"Provides the confidence-label cross-entropy idea behind the modified loss in Equation (2).","marker":"Gong et al. [2013]"},{"why":"Introduced the NYT-FB distant-supervision benchmark used for comparison.","marker":"Riedel et al. [2010]"},{"why":"Introduced and released the CC-DBP web-scale knowledge-base-population benchmark.","marker":"Glass and Gliozzo [2018]"},{"why":"Supplies the manually validated NELL triples used as ground truth for the NELL evaluation.","marker":"Jiang et al. [2012]"},{"why":"Provides the probabilistic-soft-logic knowledge-graph-identification baseline whose 90.4 AUC the NELL experiment beats.","marker":"Pujara et al. [2013]"}],"fun_headline_variants":["Distantly supervised IE + KG validation cuts errors by 50%","Coupling relation extraction with KG completion halves errors","No labeled data: web-scale KG population with 50% error cut","Self-trained KG validator improves relation extraction by 100% relative","From 0.294 to 0.785 AUC: coupled IE and completion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a KBC model trained on the extracted triples and then used to score those same triples gives an honest accuracy estimate; no held-out split of the extraction output is described, so if the model memorizes training triples, the AUC gains would not reflect generalization to newly extracted facts.","fun_headline_variants_meta":{"raw":{"variants":["Distantly supervised IE + KG validation cuts errors by 50%","Coupling relation extraction with KG completion halves errors","No labeled data: web-scale KG population with 50% error cut","Self-trained KG validator improves relation extraction by 100% relative","From 0.294 to 0.785 AUC: coupled IE and completion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3434,"prompt_tokens":880,"completion_tokens":2554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2464}},"tokens_in":496,"tokens_out":2554,"duration_ms":17213,"temperature":1.0,"reasoning_tokens":2464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:49:13.769000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split QIE before training KBVIE: train on one subset, compute precision-recall AUC on the other subset that the IE system produced but the validator never saw. If the gap between the IE-only curve and the full pipeline shrinks substantially, the reported gains are largely memorization rather than validation. Also compare predictions on triples whose entities appear in the original knowledge graph with those whose entities appear only in QIE.","supporting_citations":[{"cited_title":"Inducing implicit relations from text using distantly supervised deep nets","cited_arxiv_id":null,"evidence_quote":"Supplies the distantly supervised deep relation-extraction system that generates the candidate quads QIE."},{"cited_title":"Inducing implicit relations from text using distantly supervised deep nets","cited_arxiv_id":null,"evidence_quote":"Introduced and released the CC-DBP web-scale knowledge-base-population benchmark."},{"cited_title":"Learning to refine an automatically extracted knowledge base using markov logic","cited_arxiv_id":null,"evidence_quote":"Supplies the manually validated NELL triples used as ground truth for the NELL evaluation."}],"review_version":1}