Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources

Daniel Specht Menezes; Pedro Savarese; Ruy Luiz Milidi\'u

REVIEW 4 major objections 5 minor 24 references

Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes a fully automatic method to build a massive NER training corpus from public Wikipedia and DBpedia data, generating the Portuguese SESAME dataset, and shows that adding it to a human-annotated corpus improves F1 from…

desk verdict Useful new Portuguese NER silver corpus and a plausible +1.5 F1 gain on gold data, but the gain is not yet rigorously established because the key table lacks variance estimates and controls. read the letter →

arxiv 1908.05758 v1 pith:MNCP5NJH submitted 2019-08-13 cs.CL cs.AI

Daniel Specht Menezes , Pedro Savarese , Ruy Luiz Milidiú This is my paper · ORCID

classification cs.CLcs.AI

keywords namedentityrecognitiondistantsupervisionsilverstandardcorpusWikipediaDBpediaPortugueseBiLSTM-CRFweaklysupervisedlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large silver-standard corpus for named entity recognition can be built without human annotation by deriving entity names from DBpedia and matching them against raw Wikipedia text, then filling gaps with an auxiliary NER predictor. The resulting Portuguese dataset, SESAME, contains about 3.65 million sentences and 87.8 million tokens. In the paper's baseline, training a BiLSTM-CRF on SESAME alone reaches 67.49 F1, below gold data, but combining SESAME with the human-annotated HAREM2 corpus raises first HAREM F1 from 72.72 to 74.22. That 1.5-point gain is the evidence that automatically generated data helps build better NER predictors.

What carries the argument

The central mechanism is exact string matching of entity names against raw text: each article is parsed from wikitext, cleaned of lists, tables, files, and uninformative sections, and each character segment that exactly equals one of the entity's names from DBpedia is tagged with the entity class. An auxiliary NER predictor (Polyglot) adds mentions not found in DBpedia, and any predicted mention conflicting with a matched one is dropped. This produces a silver corpus, meaning machine-generated labels that are noisy rather than human gold. The rest of the pipeline, including BIO (begin-inside-outside) encoding, sentence concatenation for cross-sentence mentions, and word-character boundary splitting, makes the labels directly consumable by sequence taggers.

What would settle it

Have human annotators inspect a random sample of 1,000 'Anot' SESAME mentions and classify each as a true entity mention, a false positive, or a class mismatch; if false positives plus mismatches exceed roughly one in five and removing such sentences eliminates the 1.5-point F1 boost on first HAREM, the central claim is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that exact string overlap between known DBpedia entity names and raw Wikipedia article text is sufficient signal to generate reliable word-level NER labels at scale, and that such labels, though noisy, carry useful training signal. The method identifies entity mentions by matching character segments exactly against names extracted from DBpedia for person, organization, and location classes; every matched mention is tagged 'Anot', while the auxiliary NER system Polyglot adds 'Pred' mentions, which are discarded on conflict. Sentences containing at least one matched entity are kept, tokenized in BIO format, and assembled into SESAME. The paper demonstrates the corpus's utility by training a BiLSTM-CRF plus CRF model on HAREM2 alone, SESAME alone, and their union, reporting that the union outperforms HAREM2 alone by 1.5 F1 points under the first HAREM evaluation with partial-match scoring.

Load-bearing premise

The method assumes that any text segment that exactly matches a known name of a DBpedia entity is a mention of that entity, without checking context, word sense, or whether the link would actually point there.

Editorial extensions

If this is right

Under the paper's argument, automatically generated silver corpora can be used as training supplements without manual labeling effort, lowering the annotation bottleneck for NER.
The 1.5-point F1 gain on first HAREM indicates that adding large weakly labeled data to a smaller gold corpus can produce better predictors than gold data alone.
SESAME alone is not a substitute for human annotation: its 67.49 F1 underperforms HAREM2's 72.72, so the value is complementary rather than replacement.
Because the method relies only on Wikipedia and DBpedia, it can be applied to any language with these sources, making the same distant-supervision pipeline available for languages with less human-annotated data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One implication the authors leave implicit: the 1.5-point gain may reflect corpus size and label density more than annotation trustworthiness, since the only other change between HAREM2 and HAREM2+SESAME is more training data; ablating on a matched-size subset would isolate scale from label quality.
Because matching is context-free, entity names that double as common nouns or other proper names should create systematic, classifiable errors; an error analysis by name ambiguity would test this prediction.
The same pipeline should transfer to any language with DBpedia coverage; if the gain is robust across languages, it would argue that distant supervision from exact name matching is a generally useful pretraining signal rather than a Portuguese-specific artifact.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real deliverable here is SESAME, an 87-million-token Portuguese NER dataset built from Wikipedia and DBpedia with an auxiliary NER system (Polyglot) used to densify annotations. That resource is genuinely useful for a mid-resource language, and the paper is honest about the method's limits—Section IV.G says outright that context is ignored. The prior art (Nothman et al., Weber and Vieira) is cited, and the main new elements are scale, the Polyglot augmentation, and the resulting corpus. Credit where due: the final evaluation uses human-annotated HAREM data, so the result is not circular, and the authors do not oversell SESAME as a replacement for gold data.

The soft spot is exactly where the stress-test puts it. Table VIII is the sole support for the abstract's claim that the dataset "helps building better NER predictors." It reports one F1 per condition, no standard deviation, no number of runs, no seeds. For a BiLSTM-CRF with dropout and SGD, 1.5 F1 points on a test set of a few thousand sentences could easily be within run-to-run variance. The "balanced re-sample" of SESAME used in the baseline is never specified, so we do not know how the 1.2M sentences were selected. And because the silver labels come from exact string matching without context, plus about 24% Polyglot predictions, the observed boost could reflect raw data volume or label density rather than label correctness. A label-perturbation control—e.g., shuffling or corrupting silver labels and showing the gain disappears—would separate those explanations. A comparison against the closest prior silver corpus (e.g., WikiNER) would also help establish that the method, not just the scale, is what matters.

None of this is fatal. The corpus is real, the methodology is clearly described, and the direction of the result is consistent with what distant supervision generally does. But the central empirical claim is supported only by point estimates, and that is a fixable but real gap. A serious referee should ask for error bars, a fully specified balancing procedure, and a control that isolates label quality from corpus size.

Who is this for? Anyone working on Portuguese NER, and anyone building silver corpora for low- and mid-resource languages. The paper deserves peer review, not desk rejection—it just needs a stronger empirical section before acceptance.

Recommendation: engage with it, but require the missing statistical controls.

Referee Report

4 major / 5 minor

Summary. The paper proposes a method to automatically generate a large silver-standard named entity recognition (NER) corpus for Portuguese, called SESAME, by combining DBpedia structured data with Wikipedia raw text. Entity mentions are detected by exact string matching against DBpedia entity names, supplemented by the Polyglot NER system for entities not in DBpedia. The authors report corpus statistics and evaluate a BiLSTM-CRF baseline trained on SESAME, on the human-annotated Second HAREM corpus, and on their union, showing that adding SESAME to HAREM2 improves First HAREM F1 from 72.72 to 74.22. The central claim is that the automatically generated corpus helps build better NER predictors.

Significance. If the empirical result is established, the contribution is practically useful: SESAME is a large, publicly available NER dataset for a less-resourced language, and the pipeline is language-agnostic. The evaluation is conducted on external human-annotated HAREM data, so the result is not circular with respect to the training signal. However, the empirical support for the central claim is currently fragile because the key comparison in Table VIII rests on single point estimates without variance or significance testing, and the experimental setup leaves unspecified important details about how the silver corpus was sampled and combined with the gold data. The paper would be strengthened by adding controls that isolate the contribution of label content from corpus size and label density.

major comments (4)

[V.C, Table VIII] The central claim of the paper rests entirely on the F1 difference between HAREM2 (72.72) and SESAME + HAREM2 (74.22) in Table VIII. These are single point estimates with no reported standard deviation, number of runs, or random seeds. For a BiLSTM-CRF with dropout and SGD, run-to-run variance can easily be on the order of 1.5 F1 points, especially given the small gold test set. The authors should report mean and standard deviation over at least five seeds and run a significance test (e.g., paired bootstrap or approximate randomization) on the difference.
[V.C] The description of the training data is underspecified: the baseline uses a "balanced re-sample of SESAME with a total of 1,216,976 sentences," but the balancing criterion is never defined. It is also unclear how SESAME and HAREM2 are combined in the union condition: is it a concatenation of the two corpora, and if so, in what proportion? Without this information, the observed F1 boost could be attributed to the sheer volume of extra data or to a shift in label distribution rather than to the correctness of the silver labels. The authors should specify the sampling procedure, report the class distribution of the re-sample and the union, and ideally add a control that trains on a similarly sized silver corpus (e.g., WikiNER) or on SESAME with label-permuted annotations.
[IV.C and IV.G] The mention-detection strategy assumes that exact string overlap between DBpedia names and raw Wikipedia text is a reliable mention signal, and Section IV.G explicitly states that the method does not account for context. This is a correctness risk for the silver labels: homographs such as "Copacabana" may be tagged LOC even when they do not refer to the place, and 24.4% of all labels come from the Polyglot predictor (Table VI), adding further noise. Because the paper's conclusion is that SESAME "helps building better NER predictors," the authors should quantify the label noise, for example by manually evaluating a random sample of Anot and Pred mentions or by comparing SESAME's labels against a gold corpus at mention level. This would show whether the observed improvement is driven by trustworthy annotation rather than by label density alone.
[V.B and V.C] The evaluation metric is the modified First HAREM F1, which accepts partial matches and multiple correct answers. While this metric is standard in the Portuguese NER literature, it is more permissive than the exact-match entity F1 commonly used in NER. Since the paper's claim is about building "better NER predictors," the authors should additionally report the standard exact-match F1 (or at least precision/recall on exact matches) for Table VIII, to confirm that the 1.5-point gain is not an artifact of the permissive scoring.

minor comments (5)

[IV.A] The text reads "unbulled list" and "unbulleted list" interchangeably; the former appears to be a typo for "unbulleted.".
[IV.H] The sentence "sentences with only parser extraction entities do not take advantage of the human discernment" is unclear: "parser extraction" likely refers to Polyglot-detected entities, but the phrase is not defined. Rephrase to "sentences containing only Polyglot-detected entities."
[V.A] The relationship between the "first HAREM," "miniHAREM," and "HAREM2" corpora is confusing. The text says the gold validation/test split uses the first HAREM and miniHAREM, while Table VIII trains on "HAREM2." Clarify that HAREM2 is the Second HAREM corpus and describe precisely which portions are used for training, validation, and test.
[References] Several reference URLs contain the misspelling "acessed" (for "accessed"); please correct throughout.
[IV.E] The diagram for case (2), describing entities that consist of only subsets of a word, is difficult to follow because the character tokens c0...c5 are not clearly mapped to the word segments. A more explicit before-and-after example would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Table VIII's F1 gain is measured on external human-annotated HAREM test data, and no prediction reduces to the dataset-generation inputs.

full rationale

The paper's central claim is that adding SESAME to HAREM2 raises F1 from 72.72 to 74.22 (Table VIII), with the test set drawn from the first HAREM corpus, a human-annotated benchmark that is external to the SESAME construction pipeline. SESAME labels are produced by exact string matching against DBpedia entity names (Section IV.C) plus Polyglot detections (Section IV.D); neither source is defined in terms of the HAREM gold labels, and the reported gain is therefore not forced by construction. The hyperparameters were selected using a 20% validation split of first HAREM (Section V.A), with the final comparison reported on the test portion, so the result is not a fitted parameter renamed as a prediction. The only self-citation is [19], cited with [18] and [20] to support the use of the First HAREM F1 metric; the metric itself is introduced by external reference [17], so this citation is not load-bearing. Section IV.G explicitly states the context-insensitive tagging limitation, which is a quality concern for the silver labels, not a circular reduction: lower label quality would only weaken the measured F1 gain, and cannot by itself produce the reported improvement. No equation in the paper reduces to its own input, and no uniqueness claim is imported from the authors' prior work. Consequently there is no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The dataset generation has no fitted parameters of its own; the free parameters concern the experimental baseline. The central empirical claim rests on assumptions about the reliability of DBpedia labels, exact string matching, and Polyglot predictions, plus the choice of the First HAREM evaluation protocol.

free parameters (2)

Balanced re-sample size of SESAME used for baseline training = 1,216,976 sentences
The baseline was not trained on all 3,650,909 sentences; the paper states a balanced re-sample was used but does not specify the balancing procedure, so this number is a hand-chosen experimental setting that affects the reported F1 scores.
Grid-searched hyperparameters of the BiLSTM-CRF baseline (Table VII) = 100-dim GloVe, 150 hidden units, learning rate 0.005
Selected by grid search on validation data, these values are fitted to the task and influence the measured boost.

assumptions (5)

domain assumption Exact string matching of DBpedia entity names in raw Wikipedia text is a valid detection rule for NER mentions.
The entire corpus construction in Section IV.C tags text segments that exactly match a known entity name, without using context or disambiguation.
domain assumption The DBpedia ontology classes person, organization, and location are correct, sufficient, and propagated correctly from English to Portuguese.
Section III.A relies on DBpedia's extraction and interlanguage projection to decide which article subjects are entities and what class they have.
domain assumption Polyglot's automatically predicted labels are accurate enough to add more correct than incorrect entity mentions.
Section IV.D uses Polyglot to add 24.39% of the labels that are absent from DBpedia; no accuracy audit of those extra labels is provided.
standard math The modified First HAREM F1 metric is an appropriate evaluation for comparing NER predictors.
Section V.B adopts the evaluation proposed in [17] and widely used in Portuguese NER literature; this is a standard external benchmark choice.
domain assumption NLTK Punkt tokenization and the conflict resolution rules preserve entity spans.
Section IV.E assumes the tokenizer and the concatenation and character-splitting rules maintain the BIO boundaries set by exact matching and Polyglot.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources." pith.science (2026). https://pith.science/paper/MNCP5NJH

@misc{pith2026190805758,
  author       = {Pith},
  title        = {Pith review of: Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNCP5NJH}},
  note         = {Machine review of arXiv:1908.05758}
}

read the original abstract

With the recent progress in machine learning, boosted by techniques such as deep learning, many tasks can be successfully solved once a large enough dataset is available for training. Nonetheless, human-annotated datasets are often expensive to produce, especially when labels are fine-grained, as is the case of Named Entity Recognition (NER), a task that operates with labels on a word-level. In this paper, we propose a method to automatically generate labeled datasets for NER from public data sources by exploiting links and structured data from DBpedia and Wikipedia. Due to the massive size of these data sources, the resulting dataset -- SESAME Available at https://sesame-pt.github.io -- is composed of millions of labeled sentences. We detail the method to generate the dataset, report relevant statistics, and design a baseline using a neural network, showing that our dataset helps building better NER predictors.

Figures

Figures reproduced from arXiv: 1908.05758 by the authors.

**Figure 2.** Example of an interlink (blue) in a Wikipedia article. Interlinks point [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Diagram of the database representing the links between entities and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Number of occurrences for different sentence lengths (number of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

[1]

Efﬁcient estimation of word representations in vector space,

T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efﬁcient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013

arXiv 2013
[2]

Learning multilingual named entity recognition from wikipedia,

J. Nothman, N. Ringland, W. Radford, T. Murphy, and J. R. Curran, “Learning multilingual named entity recognition from wikipedia,” Arti- ﬁcial Intelligence, 2013

work page 2013
[3]

Building a corpus for named entity recognition using portuguese wikipedia and dbpedia,

C. Weber and R. Vieira, “Building a corpus for named entity recognition using portuguese wikipedia and dbpedia,” 2014

work page 2014
[4]

Dbpedia: A nucleus for a web of open data,

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. G. Ives, “Dbpedia: A nucleus for a web of open data,” in ISWC/ASWC, 2007

work page 2007
[5]

Wikipedia,

“Wikipedia,” https://pt.wikipedia.org/, acessed: 11/8/2018

work page 2018
[6]

Named entity corpus construction using wikipedia and dbpedia ontology

Y . Hahm, J. Park, K. Lim, Y . Kim, D. Hwang, and K.-S. Choi, “Named entity corpus construction using wikipedia and dbpedia ontology.” in LREC, 2014

work page 2014
[7]

Infocaixa,

“Infocaixa,” https://pt.wikipedia.org/wiki/Ajuda:Infocaixa, acessed: 11/8/2018

work page 2018
[8]

Dbpedia datasets,

“Dbpedia datasets,” https://wiki.dbpedia.org/services-resources/datasets/ dbpedia-datasets#h434-10, acessed: 21/7/2018

work page 2018

Show all 24 references

[9]

The semantic web,

T. Berners-Lee, J. Hendler, and O. Lassila, “The semantic web,” Scien- tiﬁc american, 2001

2001
[10]

Dbpedia datasets,

“Dbpedia datasets,” http://dbpedia.org/ontology/, acessed: 21/7/2018

2018
[11]

Dbpedia datasets,

“Dbpedia datasets,” https://dbpedia.org/sparql, acessed: 21/7/2018

2018
[12]

mwparserfromhell,

“mwparserfromhell,” https://mwparserfromhell.readthedocs.io/en/latest/, acessed: 11/8/2018

2018
[13]

Polyglot-ner: Massive multilingual named entity recognition,

R. Al-Rfou, V . Kulkarni, B. Perozzi, and S. Skiena, “Polyglot-ner: Massive multilingual named entity recognition,” in SDM, 2015

2015
[14]

Nltk: the natural language toolkit,

S. Bird and E. Loper, “Nltk: the natural language toolkit,” in ACL, 2004

2004
[15]

Unsupervised multilingual sentence boundary detection,

T. Kiss and J. Strunk, “Unsupervised multilingual sentence boundary detection,” Comput. Linguist., 2006

2006
[16]

Caminhos percorridos no mapa da portuguesiﬁcac ¸ ˜ao: A linguateca em perspectiva,

D. Santos, “Caminhos percorridos no mapa da portuguesiﬁcac ¸ ˜ao: A linguateca em perspectiva,” Linguam´atica, 2009

2009
[17]

HAREM: An advanced NER evaluation contest for Portuguese,

D. Santos, N. Seco, N. Cardoso, and R. Vilela, “HAREM: An advanced NER evaluation contest for Portuguese,” in LREC, 2006

2006
[18]

Boosting named entity recognition with neural character embeddings,

C. N. d. Santos and V . Guimaraes, “Boosting named entity recognition with neural character embeddings,” arXiv preprint arXiv:1505.05008 , 2015

2015 arXiv
[19]

Entropy guided transformation learning,

C. N. dos Santos and R. L. Milidi ´u, “Entropy guided transformation learning,” in Foundations of Computational Intelligence , 2009

2009
[20]

O cortex e a sua participac ¸ao no harem,

C. N. Aranha, “O cortex e a sua participac ¸ao no harem,”Reconhecimento de entidades mencionadas em portugu ˆes: Documentac ¸ao e actas do HAREM, a primeira avaliac ¸ao conjunta na ´area. Linguateca, 2007

2007
[21]

Glove: Global vectors for word representation,

J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in EMNLP, 2014

2014
[22]

Learning character-level representations for part-of-speech tagging,

C. D. Santos and B. Zadrozny, “Learning character-level representations for part-of-speech tagging,” in ICML, 2014

2014
[23]

Framewise phoneme classiﬁcation with bidirectional lstm and other neural network architectures,

A. Graves and J. Schmidhuber, “Framewise phoneme classiﬁcation with bidirectional lstm and other neural network architectures,” Neural Networks, 2005

2005
[24]

Second harem: Advancing the state of the art of named entity recognition in portuguese

C. Freitas, C. Mota, D. Santos, H. G. Oliveira, and P. Carvalho, “Second harem: Advancing the state of the art of named entity recognition in portuguese.” in LREC, 2010

2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Efﬁcient estimation of word representations in vector space,

T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efﬁcient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013

arXiv 2013

[2] [2]

Learning multilingual named entity recognition from wikipedia,

J. Nothman, N. Ringland, W. Radford, T. Murphy, and J. R. Curran, “Learning multilingual named entity recognition from wikipedia,” Arti- ﬁcial Intelligence, 2013

work page 2013

[3] [3]

Building a corpus for named entity recognition using portuguese wikipedia and dbpedia,

C. Weber and R. Vieira, “Building a corpus for named entity recognition using portuguese wikipedia and dbpedia,” 2014

work page 2014

[4] [4]

Dbpedia: A nucleus for a web of open data,

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. G. Ives, “Dbpedia: A nucleus for a web of open data,” in ISWC/ASWC, 2007

work page 2007

[5] [5]

Wikipedia,

“Wikipedia,” https://pt.wikipedia.org/, acessed: 11/8/2018

work page 2018

[6] [6]

Named entity corpus construction using wikipedia and dbpedia ontology

Y . Hahm, J. Park, K. Lim, Y . Kim, D. Hwang, and K.-S. Choi, “Named entity corpus construction using wikipedia and dbpedia ontology.” in LREC, 2014

work page 2014

[7] [7]

Infocaixa,

“Infocaixa,” https://pt.wikipedia.org/wiki/Ajuda:Infocaixa, acessed: 11/8/2018

work page 2018

[8] [8]

Dbpedia datasets,

“Dbpedia datasets,” https://wiki.dbpedia.org/services-resources/datasets/ dbpedia-datasets#h434-10, acessed: 21/7/2018

work page 2018