REVIEW 4 major objections 6 minor 16 references
ICDM 2019 Knowledge Graph Contest: Team UWA
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that a straightforward rule-based NLP pipeline can extract usable knowledge-graph triples from domain-specific text without domain-annotated training data.
desk verdict A clear contest write-up whose 'simple and effective' claim has no quantitative support; the pipeline is described well, but the evaluation is two cherry-picked sentences. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a pair of hand-written algorithms: chunking of noun phrases and verb phrases, and the triple-mapping procedure. Chunking tags noun phrases as ENTITY chunks and verb phrases as VERB chunks, so that multi-word units such as 'Ford Motor Company' and 'was founded by' survive as single items. The mapping procedure then selects relations from verbs, prepositions, and postpositions, collects all ENTITY chunks to the left as heads and all ENTITY chunks to the right as tails, and emits all combinations as triples. A second pass builds a graph and adds location-style triples along shortest paths between named entities using 'in', 'at', and 'on'. Coreference resolution is the third supporting component: it rewrites phrases such as 'The company' back to 'Ford Motor Company', which lets triples from different sentences share a node and makes the graph connected.
What would settle it
Take a held-out set of automotive or public-security documents, have annotators mark every gold triple, run the described pipeline, and measure precision and recall. The claim fails if most domain entities—such as 'suspension' and 'magnetic dampers'—are missing from the triples, or if most extracted head-relation-tail combinations are judged incorrect. A minimal probe is the sentence about re-tuning the suspension and magnetic dampers: the pipeline should output triples containing those phrases as entities, not just the car-maker and model names.
Extended reading notes
Core claim
The paper's central claim is that a deliberately simple pipeline can deliver a refined set of triples where its alternatives cannot. The pipeline first cleans the raw text, then runs standard tokenisation, part-of-speech tagging, named-entity recognition, and dependency parsing. Noun phrases are chunked into entity units, verb phrases (including verb-plus-particle, verb-plus-adposition, and verb-plus-verb sequences) are chunked into relation units, and a neural coreference resolver replaces pronouns and definite descriptions with their antecedents. The triple-mapping component treats every verb, preposition, or postposition as a relation candidate, takes all entity chunks to its left as heads and all entity chunks to its right as tails, and emits every head-relation-tail combination. A second, document-level pass constructs a graph from these triples and adds prepositional triples such as (headquarters, in, Dearborn) for named-entity pairs connected by paths through 'in', 'at', or 'on'. Filtering removes triples with stop-word heads and strips articles from phrase boundaries. The paper demonstrates the output on two sentences and shows that coreference resolution lets triples from separate sentences share a single node, producing a connected graph.
Load-bearing premise
The whole approach depends on off-the-shelf language tools and hand-written chunking and mapping rules working well on the target domain's sentences; the paper only demonstrates the rules on a few example sentences and gives no measurement of how often they succeed.
Editorial extensions
If this is right
- Low-resource relation extraction can be tackled by composing off-the-shelf language tools with a small set of hand-written chunking and mapping rules, rather than training an end-to-end network.
- Noun-phrase chunking recovers domain entities that standard named-entity recognition misses, so the graph can include parts, products, and properties rather than only persons, locations, and organisations.
- Coreference resolution before triple mapping lets one entity accumulate triples across an entire document, yielding a graph whose node sizes reflect how central each entity is.
- Adding prepositional triples through graph paths creates location links that a sentence-local extractor would miss, such as connecting a headquarters entity to the city where it is located.
Reading between the lines
- A natural extension would be to derive the chunking patterns from a small seed set of domain triples instead of writing them by hand, preserving the transparency of the pipeline while reducing manual rule-writing.
- The graph-based preposition rule could be made relation-aware, adding 'in' triples only when the tail is a location-type entity; this would probably cut spurious links while keeping the connectivity benefit.
- Because the pipeline is modular, each stage can be diagnosed separately; reporting which stage drops each gold triple would turn the pipeline into a practical debugging tool for knowledge-base construction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short contest report describes Team UWA's triple extraction system for the ICDM 2019 Knowledge Graph Contest. The pipeline consists of text cleaning, tokenisation, POS tagging, NER, dependency parsing, noun/verb phrase chunking, coreference resolution, triple mapping, triple filtering, and article removal, followed by a visualisation stage that attaches degree/betweenness, SemEval relation labels, and named-entity types to each triple. The authors motivate the design by arguing that off-the-shelf NER misses domain-specific entities and that OpenIE and fixed-relation knowledge graph systems are unsuitable for the contest task. They claim in the abstract and conclusion that the resulting system is 'a simple and effective solution' to knowledge graph construction from domain-specific text. The evaluation section, however, contains only two worked example sentences about Ford Motor Company, a table of the extracted triples with centrality values, and a qualitative discussion of coreference resolution; no precision, recall, F1, baseline comparison, contest score, or error analysis is reported anywhere in the manuscript.
Significance. If the claimed effectiveness were actually demonstrated, the contribution would be a lightweight, readily reproducible alternative to end-to-end deep learning for triple extraction in low-resource domains. The paper is clearly written and the pipeline design is sensible, and the visualisation component with degree/betweenness and SemEval relation types is a useful add-on. However, the central claim is not backed by any quantitative evaluation. The only numbers in Table III are graph-theoretic centrality values, which say nothing about extraction quality. Because the entire contribution rests on the assertion that the pipeline is effective, and because the submitted evidence consists of two hand-picked general-English sentences, the significance of the result as presented cannot be assessed. The manuscript would need a real evaluation on the contest data or another domain-specific corpus, with appropriate metrics and at least one baseline, before the claimed contribution could be accepted.
major comments (4)
- [Section II-A and II-C] The central claim that the system is 'a simple and effective solution' is unsupported by the evidence. Section II-A evaluates exactly two sentences, reports no gold-standard comparison, no precision/recall/F1, no baseline, and no contest score. Table III's degree and betweenness values measure graph structure, not extraction correctness. The conclusion in Section II-C restates the effectiveness claim without adding any new evidence. This is a load-bearing evidence gap: if the system produced mostly spurious triples on longer documents, the claim would fail, and nothing in the manuscript rules that out.
- [Section I-B, Algorithm 2] Algorithm 2 generates triples by taking all entities on the left of a relation and all entities on the right, and forming the Cartesian product (heads x tails). In a sentence with several entities on each side, this produces O(H x T) triples, most of which are likely to be spurious. The paper criticises OpenIE for producing too many low-quality triples but does not measure whether its own pairwise rule suffers from the same problem. Since the manuscript claims a 'refined set of high-quality triples' is needed for the contest task, this combinatorial behaviour needs to be evaluated on realistic inputs.
- [Section I-A and Section II-A] The paper motivates the approach by arguing that domain-specific text (automotive engineering, public security) is poorly handled by off-the-shelf NER, giving the example where 'suspension' and 'magnetic dampers' are missed. Yet the only evaluation example is general English text about Ford Motor Company, and none of the extracted triples in Table III involve the domain entities the system is designed to capture. No automotive or public-security text is processed or shown. The domain-specific claim is therefore not tested at all, despite being central to the abstract and conclusion.
- [Section II-B] The coreference resolution evaluation is purely qualitative. The authors add a third sentence and observe that the visualisation shows Ford Motor Company as the shared entity, with no quantitative measure of coreference accuracy or its effect on downstream triples. Coreference errors can propagate to incorrect triples across sentences, so this component needs at least a simple accuracy figure or error examples to support the effectiveness claim.
minor comments (6)
- [Title] The title reads 'Team UW A' with an unintended space; it should be 'Team UWA'.
- [Section I-A] The phrase 'it is near-impossible to find the right pivot language that allows us to take advantages of existing high resource NER tools' is unclear in the context of NER and should be rephrased.
- [Section I-B] The reference for spaCy, reference [4], points to a blog post titled 'Introducing spaCy' rather than the library's official documentation or a stable citation; this should be corrected.
- [Table III] Several extracted triples are semantically questionable, e.g., 'Ford Motor Company in June 16, 1903' and 'Ford Motor Company is American multinational automaker' (missing article). The SemEval labels also look arbitrary in places, such as 'Cause-Effect' for 'American multinational automaker has main headquarters'. The paper should at least acknowledge and discuss such errors, since the table is the only evidence of output quality.
- [Section I-B] The description of the contest task is minimal; the paper never states what the evaluation criteria for the ICDM 2019 Knowledge Graph Contest were, which makes it difficult to judge whether the reported system would be competitive. Adding the contest's official evaluation protocol or a link would help.
- [Section I-C] The paper says the source code of the visualisation system is available on GitHub, but it does not state whether the triple extraction pipeline itself is released. Clarifying the availability of the extraction code would improve reproducibility.
Circularity Check
No circularity: the pipeline's outputs are not used as inputs, and the lack of quantitative evaluation is an evidence gap, not circular reasoning.
full rationale
The paper describes a rule-based triple extraction pipeline and visualisation system. Its central claim of being a 'simple and effective solution' is not supported by independent quantitative evaluation: Section II-A evaluates only two hand-picked sentences, with no gold triples, precision/recall/F1, baselines, or contest scores. This is a genuine evidentiary weakness, but it is not circularity. No parameter is fitted to data and then renamed as a prediction; no equation defines the output in terms of the input; no load-bearing claim rests on a self-citation chain; no uniqueness theorem is imported from the authors' prior work; and no known result is merely renamed. The triple-mapping algorithm (Algorithm 2) deterministically constructs triples from chunked phrases, and the degree/betweenness and SemEval relation labels are post-hoc annotations rather than evidence for extraction quality. The only self-referential element is that the authors illustrate their own system with their own examples, which is not a derivation that reduces to its own inputs. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption SpaCy's dependency parser, POS tagger, and NER produce sufficiently accurate outputs on the contest's domain-specific text.
- domain assumption NeuralCoref correctly resolves coreference chains (e.g., 'The company' to 'Ford Motor Company') and the pronoun-ignoring heuristic preserves meaning.
- ad hoc to paper The hand-coded chunking patterns (e.g., NP + 'of' + NP, verb + particle) and the relation selection rule 'relations <- verbs + prepositions + postpositions' generalize beyond the example sentences.
Cite this review
Pith. "Pith review of ICDM 2019 Knowledge Graph Contest: Team UWA." pith.science (2026). https://pith.science/paper/ATJALODI
@misc{pith2026190901807,
author = {Pith},
title = {Pith review of: ICDM 2019 Knowledge Graph Contest: Team UWA},
year = {2026},
howpublished = {\url{https://pith.science/paper/ATJALODI}},
note = {Machine review of arXiv:1909.01807}
}
read the original abstract
We present an overview of our triple extraction system for the ICDM 2019 Knowledge Graph Contest. Our system uses a pipeline-based approach to extract a set of triples from a given document. It offers a simple and effective solution to the challenge of knowledge graph construction from domain-specific text. It also provides the facility to visualise useful information about each triple such as the degree, betweenness, structured relation type(s), and named entity types.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction to the conll-2003 shared task: Language-independent named entity recognition,
E. F. T. K. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” CoNLL- 2003, 2003
work page 2003
-
[2]
Ontonotes release 5.0 ldc2013t19,
R. Weischedel, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini et al. , “Ontonotes release 5.0 ldc2013t19,” Linguistic Data Consortium, Philadelphia, PA, 2013
work page 2013
-
[3]
S. Bird, E. Klein, and E. Loper, Natural language processing with Python: analyzing text with the natural language toolkit . ” O’Reilly Media, Inc.”, 2009
work page 2009
-
[4]
M. Honnibal, “Spacy,” 2017. [Online]. Available: https://explosion.ai/ blog/introducing-spacy
work page 2017
-
[5]
Incorporating non-local information into information extraction systems by gibbs sampling,
J. R. Finkel, T. Grenager, and C. Manning, “Incorporating non-local information into information extraction systems by gibbs sampling,” in Proceedings of the 43rd annual meeting on association for computa- tional linguistics . Association for Computational Linguistics, 2005, pp. 363–370
work page 2005
-
[6]
Allennlp: A deep semantic natural language processing platform,
M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. F. Liu, M. Peters, M. Schmitz, and L. S. Zettlemoyer, “Allennlp: A deep semantic natural language processing platform,” 2017
work page 2017
-
[7]
Deep contextualized word representations,
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” arXiv preprint arXiv:1802.05365, 2018
arXiv 2018
-
[8]
A survey on open information extraction,
C. Niklaus, M. Cetto, A. Freitas, and S. Handschuh, “A survey on open information extraction,” arXiv preprint arXiv:1806.05599 , 2018
arXiv 2018
Show all 16 references
-
[9]
Neural open information extraction,
L. Cui, F. Wei, and M. Zhou, “Neural open information extraction,” arXiv preprint arXiv:1805.04270 , 2018
2018 arXiv
-
[10]
T2kg: An end-to-end system for creating knowledge graph from unstructured text,
N. Kertkeidkachorn and R. Ichise, “T2kg: An end-to-end system for creating knowledge graph from unstructured text,” in Workshops at the Thirty-First AAAI Conference on Artificial Intelligence , 2017
2017
-
[11]
Graph theory. 2005,
R. Diestel, “Graph theory. 2005,” Grad. Texts in Math , vol. 101, 2005
2005
-
[12]
Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nomi- nals,
I. Hendrickx, S. N. Kim, Z. Kozareva, P. Nakov, D. ´O S´eaghdha, S. Pad´o, M. Pennacchiotti, L. Romano, and S. Szpakowicz, “Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nomi- nals,” in Proceedings of the Workshop on Semantic Evaluations: ...
2010
-
[13]
Attention- based bidirectional long short-term memory networks for relation classi- fication,
P. Zhou, W. Shi, J. Tian, Z. Qi, B. Li, H. Hao, and B. Xu, “Attention- based bidirectional long short-term memory networks for relation classi- fication,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016, ...
2016
-
[14]
Learning multilingual named entity recognition from wikipedia,
J. Nothman, N. Ringland, W. Radford, T. Murphy, and J. R. Curran, “Learning multilingual named entity recognition from wikipedia,” Arti- ficial Intelligence, vol. 194, pp. 151–175, 2013
2013
-
[15]
Spice: Semantic propositional image caption evaluation,
P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in European Conference on Computer Vision. Springer, 2016, pp. 382–398
2016
-
[16]
Tensorflow: A system for large- scale machine learning,
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large- scale machine learning,” in 12th {USENIX} Symposium on Operating Systems Design and Implementation ( {OSDI} 16), 2016, pp. 265–283
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.