{"id":"90e1d27a-1619-4297-a906-f90bfb7d247c","arxiv_id":"1908.05758","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Automatically generated silver-standard NER corpus SESAME from Portuguese Wikipedia and DBpedia, with a BiLSTM-CRF baseline showing a 1.5 F1 improvement when combined with hand-annotated HAREM2.","lead":"The authors describe a fully automatic recipe to build a massive Portuguese dataset for teaching computers to spot people, places, and organizations in text, using Wikipedia and DBpedia instead of human annotators. Their test shows that adding this dataset to a smaller hand-labeled corpus improves recognition accuracy by about 1.5 F1 points.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table VIII is the sole support for \"SESAME helps build better predictors,\" but it reports one run without error bars or a control that isolates label quality from corpus size/balancing; the 1.5 F1 gain is therefore not yet established.","rationale":"The reader's weakest assumption, that exact string overlap is a reliable mention signal, is a real risk, but I think the more load-bearing issue is empirical: the paper's only evidence for the central claim is a single comparison with no uncertainty quantification and no baseline that controls for data quantity. Even a perfectly labeled corpus would appear helpful if the only test is adding millions of sentences; the experiment as reported cannot separate label quality from corpus size. The label-shuffle control would settle this by destroying the link between surface forms and labels while keeping size, distribution, and model fixed. Because the reader already returned CONDITIONAL and explicitly requested a fully specified balancing procedure and stronger evidence, I do not think the verdict needs to change; my concern reinforces the conditions rather than overturning the paper. If the authors supply the control or error bars and the boost survives, the claim should be accepted; if the shuffled-label control reproduces the boost, the claim should be weakened.","tokens_in":8016,"tokens_out":7637,"duration_ms":84898,"concrete_test":"Label-shuffle control: take the exact SESAME+HAREM2 training setup with the balancing procedure fixed and documented, then shuffle the entity labels of whole mentions among mention spans in SESAME, preserving per-class counts and B/I structure. Train the same BiLSTM-CRF with the same hyperparameters for at least five random seeds, and compare the resulting first-HAREM F1 distribution against the reported 74.22 from the unchanged corpus. If the shuffled-label control reaches the same F1 within the same 95% confidence interval, then the observed boost is not evidence for the semantic reliability of SESAME's annotations, and the central claim should be weakened to 'more data helps' rather than 'silver labels help.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Abstract's 'showing that our dataset helps building better NER predictors,' operationalized only in Table VIII: F1 = 72.72 (HAREM2) vs 74.22 (SESAME + HAREM2). The paper reports no standard deviation, no number of runs, and no seed; it also never specifies how the 'balanced re-sample' of SESAME was combined with HAREM2 in the union condition (Section V.C). Because the test set is only a few thousand gold sentences, a 1.5-point F1 difference may be within run-to-run variance for a BiLSTM-CRF with dropout and SGD. The alternative reading is that the boost comes from the extra data volume or from label density, rather than from the correctness of the silver labels. Section IV.C's exact-name matching, combined with Section IV.G's explicit statement that mentions are tagged without regard to context, makes this alternative plausible: many 'Anot' mentions may be false positives, and the 24.4% 'Pred' mentions add further noise. Thus, the paper's central claim needs either confidence intervals or a label-perturbation control to show that the semantic content of SESAME labels, not just their presence, drives the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to automatically generate a large silver-standard named entity recognition (NER) corpus for Portuguese, called SESAME, by combining DBpedia structured data with Wikipedia raw text. Entity mentions are detected by exact string matching against DBpedia entity names, supplemented by the Polyglot NER system for entities not in DBpedia. The authors report corpus statistics and evaluate a BiLSTM-CRF baseline trained on SESAME, on the human-annotated Second HAREM corpus, and on their union, showing that adding SESAME to HAREM2 improves First HAREM F1 from 72.72 to 74.22. The central claim is that the automatically generated corpus helps build better NER predictors.","tokens_in":8243,"tokens_out":3957,"duration_ms":40565,"significance":"If the empirical result is established, the contribution is practically useful: SESAME is a large, publicly available NER dataset for a less-resourced language, and the pipeline is language-agnostic. The evaluation is conducted on external human-annotated HAREM data, so the result is not circular with respect to the training signal. However, the empirical support for the central claim is currently fragile because the key comparison in Table VIII rests on single point estimates without variance or significance testing, and the experimental setup leaves unspecified important details about how the silver corpus was sampled and combined with the gold data. The paper would be strengthened by adding controls that isolate the contribution of label content from corpus size and label density.","major_comments":[{"comment":"The central claim of the paper rests entirely on the F1 difference between HAREM2 (72.72) and SESAME + HAREM2 (74.22) in Table VIII. These are single point estimates with no reported standard deviation, number of runs, or random seeds. For a BiLSTM-CRF with dropout and SGD, run-to-run variance can easily be on the order of 1.5 F1 points, especially given the small gold test set. The authors should report mean and standard deviation over at least five seeds and run a significance test (e.g., paired bootstrap or approximate randomization) on the difference.","section":"V.C, Table VIII"},{"comment":"The description of the training data is underspecified: the baseline uses a \"balanced re-sample of SESAME with a total of 1,216,976 sentences,\" but the balancing criterion is never defined. It is also unclear how SESAME and HAREM2 are combined in the union condition: is it a concatenation of the two corpora, and if so, in what proportion? Without this information, the observed F1 boost could be attributed to the sheer volume of extra data or to a shift in label distribution rather than to the correctness of the silver labels. The authors should specify the sampling procedure, report the class distribution of the re-sample and the union, and ideally add a control that trains on a similarly sized silver corpus (e.g., WikiNER) or on SESAME with label-permuted annotations.","section":"V.C"},{"comment":"The mention-detection strategy assumes that exact string overlap between DBpedia names and raw Wikipedia text is a reliable mention signal, and Section IV.G explicitly states that the method does not account for context. This is a correctness risk for the silver labels: homographs such as \"Copacabana\" may be tagged LOC even when they do not refer to the place, and 24.4% of all labels come from the Polyglot predictor (Table VI), adding further noise. Because the paper's conclusion is that SESAME \"helps building better NER predictors,\" the authors should quantify the label noise, for example by manually evaluating a random sample of Anot and Pred mentions or by comparing SESAME's labels against a gold corpus at mention level. This would show whether the observed improvement is driven by trustworthy annotation rather than by label density alone.","section":"IV.C and IV.G"},{"comment":"The evaluation metric is the modified First HAREM F1, which accepts partial matches and multiple correct answers. While this metric is standard in the Portuguese NER literature, it is more permissive than the exact-match entity F1 commonly used in NER. Since the paper's claim is about building \"better NER predictors,\" the authors should additionally report the standard exact-match F1 (or at least precision/recall on exact matches) for Table VIII, to confirm that the 1.5-point gain is not an artifact of the permissive scoring.","section":"V.B and V.C"}],"minor_comments":[{"comment":"The text reads \"unbulled list\" and \"unbulleted list\" interchangeably; the former appears to be a typo for \"unbulleted.\".","section":"IV.A"},{"comment":"The sentence \"sentences with only parser extraction entities do not take advantage of the human discernment\" is unclear: \"parser extraction\" likely refers to Polyglot-detected entities, but the phrase is not defined. Rephrase to \"sentences containing only Polyglot-detected entities.\"","section":"IV.H"},{"comment":"The relationship between the \"first HAREM,\" \"miniHAREM,\" and \"HAREM2\" corpora is confusing. The text says the gold validation/test split uses the first HAREM and miniHAREM, while Table VIII trains on \"HAREM2.\" Clarify that HAREM2 is the Second HAREM corpus and describe precisely which portions are used for training, validation, and test.","section":"V.A"},{"comment":"Several reference URLs contain the misspelling \"acessed\" (for \"accessed\"); please correct throughout.","section":"References"},{"comment":"The diagram for case (2), describing entities that consist of only subsets of a word, is difficult to follow because the character tokens c0...c5 are not clearly mapped to the word segments. A more explicit before-and-after example would improve readability.","section":"IV.E"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is a large, publicly available silver-standard NER dataset and a reproducible pipeline. The evaluation design is reasonable in principle (external gold test set), but the missing variance information and underspecified training-set construction leave the central empirical claim insufficiently supported. With added experiments and clearer reporting, the paper could become a solid dataset-resource paper for the NER community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real deliverable here is SESAME, an 87-million-token Portuguese NER dataset built from Wikipedia and DBpedia with an auxiliary NER system (Polyglot) used to densify annotations. That resource is genuinely useful for a mid-resource language, and the paper is honest about the method's limits—Section IV.G says outright that context is ignored. The prior art (Nothman et al., Weber and Vieira) is cited, and the main new elements are scale, the Polyglot augmentation, and the resulting corpus. Credit where due: the final evaluation uses human-annotated HAREM data, so the result is not circular, and the authors do not oversell SESAME as a replacement for gold data.\n\nThe soft spot is exactly where the stress-test puts it. Table VIII is the sole support for the abstract's claim that the dataset \"helps building better NER predictors.\" It reports one F1 per condition, no standard deviation, no number of runs, no seeds. For a BiLSTM-CRF with dropout and SGD, 1.5 F1 points on a test set of a few thousand sentences could easily be within run-to-run variance. The \"balanced re-sample\" of SESAME used in the baseline is never specified, so we do not know how the 1.2M sentences were selected. And because the silver labels come from exact string matching without context, plus about 24% Polyglot predictions, the observed boost could reflect raw data volume or label density rather than label correctness. A label-perturbation control—e.g., shuffling or corrupting silver labels and showing the gain disappears—would separate those explanations. A comparison against the closest prior silver corpus (e.g., WikiNER) would also help establish that the method, not just the scale, is what matters.\n\nNone of this is fatal. The corpus is real, the methodology is clearly described, and the direction of the result is consistent with what distant supervision generally does. But the central empirical claim is supported only by point estimates, and that is a fixable but real gap. A serious referee should ask for error bars, a fully specified balancing procedure, and a control that isolates label quality from corpus size.\n\nWho is this for? Anyone working on Portuguese NER, and anyone building silver corpora for low- and mid-resource languages. The paper deserves peer review, not desk rejection—it just needs a stronger empirical section before acceptance.\n\nRecommendation: engage with it, but require the missing statistical controls.","headline":"Useful new Portuguese NER silver corpus and a plausible +1.5 F1 gain on gold data, but the gain is not yet rigorously established because the key table lacks variance estimates and controls.","tokens_in":8820,"tokens_out":1352,"would_cite":true,"duration_ms":15229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a fully automatic method to build a massive NER training corpus from public Wikipedia and DBpedia data, generating the Portuguese SESAME dataset, and shows that adding it to a human-annotated corpus improves F1 from…","keywords":["named entity recognition","distant supervision","silver standard corpus","Wikipedia","DBpedia","Portuguese","BiLSTM-CRF","weakly supervised learning"],"falsifier":"Have human annotators inspect a random sample of 1,000 'Anot' SESAME mentions and classify each as a true entity mention, a false positive, or a class mismatch; if false positives plus mismatches exceed roughly one in five and removing such sentences eliminates the 1.5-point F1 boost on first HAREM, the central claim is unsupported.","tokens_in":7785,"feed_emoji":"🏷️","tokens_out":7227,"duration_ms":69301,"temperature":0.7,"pith_summary":"The paper claims that a large silver-standard corpus for named entity recognition can be built without human annotation by deriving entity names from DBpedia and matching them against raw Wikipedia text, then filling gaps with an auxiliary NER predictor. The resulting Portuguese dataset, SESAME, contains about 3.65 million sentences and 87.8 million tokens. In the paper's baseline, training a BiLSTM-CRF on SESAME alone reaches 67.49 F1, below gold data, but combining SESAME with the human-annotated HAREM2 corpus raises first HAREM F1 from 72.72 to 74.22. That 1.5-point gain is the evidence that automatically generated data helps build better NER predictors.","feed_headline":"Millions of auto-labeled sentences boost Portuguese NER by 1.5 F1","feed_subtitle":"Using only public data, the method builds a 3.65M-sentence corpus that improves Portuguese NER when added to gold data.","key_machinery":"The central mechanism is exact string matching of entity names against raw text: each article is parsed from wikitext, cleaned of lists, tables, files, and uninformative sections, and each character segment that exactly equals one of the entity's names from DBpedia is tagged with the entity class. An auxiliary NER predictor (Polyglot) adds mentions not found in DBpedia, and any predicted mention conflicting with a matched one is dropped. This produces a silver corpus, meaning machine-generated labels that are noisy rather than human gold. The rest of the pipeline, including BIO (begin-inside-outside) encoding, sentence concatenation for cross-sentence mentions, and word-character boundary splitting, makes the labels directly consumable by sequence taggers.","core_discovery":"The central claim is that exact string overlap between known DBpedia entity names and raw Wikipedia article text is sufficient signal to generate reliable word-level NER labels at scale, and that such labels, though noisy, carry useful training signal. The method identifies entity mentions by matching character segments exactly against names extracted from DBpedia for person, organization, and location classes; every matched mention is tagged 'Anot', while the auxiliary NER system Polyglot adds 'Pred' mentions, which are discarded on conflict. Sentences containing at least one matched entity are kept, tokenized in BIO format, and assembled into SESAME. The paper demonstrates the corpus's utility by training a BiLSTM-CRF plus CRF model on HAREM2 alone, SESAME alone, and their union, reporting that the union outperforms HAREM2 alone by 1.5 F1 points under the first HAREM evaluation with partial-match scoring.","pith_inferences":["One implication the authors leave implicit: the 1.5-point gain may reflect corpus size and label density more than annotation trustworthiness, since the only other change between HAREM2 and HAREM2+SESAME is more training data; ablating on a matched-size subset would isolate scale from label quality.","Because matching is context-free, entity names that double as common nouns or other proper names should create systematic, classifiable errors; an error analysis by name ambiguity would test this prediction.","The same pipeline should transfer to any language with DBpedia coverage; if the gain is robust across languages, it would argue that distant supervision from exact name matching is a generally useful pretraining signal rather than a Portuguese-specific artifact."],"forward_implications":["Under the paper's argument, automatically generated silver corpora can be used as training supplements without manual labeling effort, lowering the annotation bottleneck for NER.","The 1.5-point F1 gain on first HAREM indicates that adding large weakly labeled data to a smaller gold corpus can produce better predictors than gold data alone.","SESAME alone is not a substitute for human annotation: its 67.49 F1 underperforms HAREM2's 72.72, so the value is complementary rather than replacement.","Because the method relies only on Wikipedia and DBpedia, it can be applied to any language with these sources, making the same distant-supervision pipeline available for languages with less human-annotated data."],"supporting_citations":[{"why":"An earlier automatic method that builds multilingual NER corpora from Wikipedia; it establishes the distant-learning approach this paper extends.","marker":"[2]"},{"why":"A prior Portuguese corpus built from Wikipedia and DBpedia; it is the direct baseline this method differentiates from by adding an auxiliary predictor.","marker":"[3]"},{"why":"The DBpedia source that supplies structured entity names, classes, and links to Wikipedia articles for the matching step.","marker":"[4]"},{"why":"The auxiliary NER predictor used to detect entity mentions missing from DBpedia, increasing tagged mentions by about a third.","marker":"[13]"},{"why":"The modified first HAREM evaluation metric used to score model output, including partial-match and multi-tag handling.","marker":"[17]"},{"why":"The manually annotated HAREM2 corpus used as gold training data in the main comparison with and without SESAME.","marker":"[24]"},{"why":"Pretrained word embeddings used by the baseline neural network, one of the model components whose hyperparameters were selected by grid search.","marker":"[21]"}],"fun_headline_variants":["DBpedia+Wikipedia auto-tag 3.65M sentences, NER gains 1.5 F1","3.65M auto-labeled sentences from free data lift Portuguese NER by 1.5 F1","No manual tagging: DBpedia+Wikipedia build 3.65M-sentence NER set","Free data sources yield 3.65M NER training sentences, adding 1.5 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that any text segment that exactly matches a known name of a DBpedia entity is a mention of that entity, without checking context, word sense, or whether the link would actually point there.","fun_headline_variants_meta":{"raw":{"variants":["DBpedia+Wikipedia auto-tag 3.65M sentences, NER gains 1.5 F1","3.65M auto-labeled sentences from free data lift Portuguese NER by 1.5 F1","No manual tagging: DBpedia+Wikipedia build 3.65M-sentence NER set","Free data sources yield 3.65M NER training sentences, adding 1.5 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00119,"raw_usage":{"total_tokens":5068,"prompt_tokens":885,"completion_tokens":4183,"prompt_tokens_details":{"cached_tokens":768},"prompt_cache_hit_tokens":768,"prompt_cache_miss_tokens":117,"completion_tokens_details":{"reasoning_tokens":4074}},"tokens_in":117,"tokens_out":4183,"duration_ms":297585,"temperature":1.0,"reasoning_tokens":4074,"cache_read_input_tokens":768,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:41:09.764306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators inspect a random sample of 1,000 'Anot' SESAME mentions and classify each as a true entity mention, a false positive, or a class mismatch; if false positives plus mismatches exceed roughly one in five and removing such sentences eliminates the 1.5-point F1 boost on first HAREM, the central claim is unsupported.","supporting_citations":[{"cited_title":"Learning multilingual named entity recognition from wikipedia,","cited_arxiv_id":null,"evidence_quote":"An earlier automatic method that builds multilingual NER corpora from Wikipedia; it establishes the distant-learning approach this paper extends."},{"cited_title":"Building a corpus for named entity recognition using portuguese wikipedia and dbpedia,","cited_arxiv_id":null,"evidence_quote":"A prior Portuguese corpus built from Wikipedia and DBpedia; it is the direct baseline this method differentiates from by adding an auxiliary predictor."},{"cited_title":"Dbpedia: A nucleus for a web of open data,","cited_arxiv_id":null,"evidence_quote":"The DBpedia source that supplies structured entity names, classes, and links to Wikipedia articles for the matching step."},{"cited_title":"Polyglot-ner: Massive multilingual named entity recognition,","cited_arxiv_id":null,"evidence_quote":"The auxiliary NER predictor used to detect entity mentions missing from DBpedia, increasing tagged mentions by about a third."},{"cited_title":"HAREM: An advanced NER evaluation contest for Portuguese,","cited_arxiv_id":null,"evidence_quote":"The modified first HAREM evaluation metric used to score model output, including partial-match and multi-tag handling."},{"cited_title":"Second harem: Advancing the state of the art of named entity recognition in portuguese","cited_arxiv_id":null,"evidence_quote":"The manually annotated HAREM2 corpus used as gold training data in the main comparison with and without SESAME."}],"review_version":1}