Pith. sign in

REVIEW 3 major objections 4 minor 20 references

A Finnish News Corpus for Named Entity Recognition

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents a manually annotated Finnish news corpus of 953 articles (193,742 tokens) with six named-entity classes and nested annotation, and reports that a rule-based recognizer outperforms two neural baselines on both in-domain…

desk verdict A solid, honestly documented Finnish NER resource whose baseline comparison is slightly circular but probably not wrong; the corpus is the contribution and it is worth publishing. read the letter →

arxiv 1908.04212 v1 pith:GLY2JMAJ submitted 2019-08-12 cs.CL

Teemu Ruokolainen , Pekka Kauppinen , Miikka Silfverberg , Krister Lindén This is my paper · ORCID
classification cs.CL
keywords Finnishnamedentityrecognitionnestedannotationgold-standardcorpusDigitodayrule-basedNERneuralbaselinesnewsarticlessixclasses
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents a resource for Finnish named entity recognition: a manually annotated corpus of 953 technology-news articles (193,742 word tokens) drawn from Digitoday, tagged with six classes: organization, location, person, product, event, and date, and including nested annotation. The practical goal is to give Finnish NER a reusable gold standard for training and evaluation, comparable in function to existing English and German resources. The paper also runs three baseline systems on the corpus: the rule-based FINER tool and two neural architectures. On an in-domain Digitoday test set FINER reaches F1 85.20; on an out-of-domain Wikipedia test set it reaches 79.91, ahead of the neural systems in both cases. A sympathetic reader takes the main claim to be that this corpus is a usable benchmark and that rule-based knowledge transfers across domains more reliably than the data-driven systems tested here.

What carries the argument

The central object is the corpus itself: 953 articles, 193,742 word tokens, six entity classes, and a nested annotation scheme in which a shorter entity inside a longer one is also tagged, for example a location inside an organization. The annotation was produced in two stages with discussion and refinement between annotators, and its consistency was checked once by an external annotator, with inter-annotator agreement of kappa 0.79 for top-level entities and 0.76 including nested ones. The experimental argument is carried by FINER, a rule-based recognizer that combines morphological analysis, gazetteers, and pattern-matching rules, while the two neural baselines use pretrained word embeddings and, in one case, morphological tags. The corpus is released in a three-column BIO format that keeps top-level and nested labels in separate columns.

What would settle it

Re-annotate a fresh sample of the same Digitoday articles with two independent annotators using the published guidelines; if agreement falls well below the reported kappa values, or if product-class and span errors cluster systematically, the gold-standard claim weakens. Alternatively, retrain the neural baselines without morphological features or gazetteer-influenced embeddings and check whether the reported F1 ordering on the Wikipedia set persists.

Watch

Extended reading notes

Core claim

The central claim is that the corpus is a valid, usable gold standard for Finnish NER, and that on this corpus the rule-based FINER system is the strongest baseline. FINER obtains F1 85.20 on all entities in the Digitoday test set and 79.91 on the Wikipedia test set; the differences over the two neural systems are reported as statistically significant. On Wikipedia the gap is large: the neural models fall to F1 60.64 and 52.52. The paper's error analysis adds that FINER's recall errors are mostly complete misses, whereas the neural systems more often find the right span but assign the wrong class.

Load-bearing premise

The corpus's value rests on the manual annotations being correct and consistent enough to serve as ground truth; that assumption is supported by a single agreement check with one external annotator, which gave kappa 0.79 top-level and 0.76 with nested entities, and the product class already showed notable disagreement.

Editorial extensions

If this is right

  • Future Finnish NER systems can be compared directly against FINER's F1 scores of 85.20 in-domain and 79.91 out-of-domain on the released corpus.
  • The nested annotation supports research on overlapping named entities rather than forcing a flat tag set.
  • The large performance drop of the neural systems on Wikipedia indicates that data-driven Finnish NER needs more training data or explicit domain adaptation before it generalizes.
  • The event class has too few instances in the training set to support reliable learning or evaluation, so future work may reasonably drop it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because one neural baseline receives morphological tags produced with resources that overlap FINER's gazetteers, the reported rule-versus-neural gap may not be a pure comparison of knowledge-based and learned systems; isolating the shared resources would test this.
  • The Wikipedia test set is strongly skewed toward biographies, so the out-of-domain gap may reflect a genre shift to person-heavy text as much as a general domain shift; a balanced out-of-domain sample would separate the two.
  • A concrete next experiment suggested by the paper's own reasoning is to use FINER to auto-annotate a large unlabeled corpus, manually correct a sample, train a neural model on the mix, and measure whether its Wikipedia F1 approaches FINER's 79.91.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Digitoday, a manually annotated Finnish news corpus of 953 articles (193,742 tokens) with six named entity classes and nested annotations, and reports baseline experiments comparing a rule-based system (FINER) with two neural architectures (GÜNGÖR-NN and SOHRAB-NN) on in-domain (Digitoday 2015) and out-of-domain (Finnish Wikipedia) test sets. The corpus is released through FIN-CLARIN, and the annotation guidelines, inter-annotator agreement (kappa = 0.79 for top-level, 0.76 including nested), and detailed class statistics are presented. In the experiments, FINER achieves the best macro-averaged F1 on both test sets (85.20 and 79.91 for all entities), with statistical significance reported via a Wilcoxon signed-rank test over 20 subsets.

Significance. If taken at face value, the paper is a valuable contribution: it provides a publicly available Finnish NER corpus with a comparatively rare nested annotation layer, documents a detailed annotation guideline, reports a reasonable inter-annotator agreement, and releases the FINER rule-based tool as a companion resource. The baseline experiments are useful as initial reference points, and the per-class error analysis, especially the breakdown of recall error types, is informative for future work. However, the experimental comparison is compromised by a developer-in-the-loop relationship between FINER and the gold annotations, and the neural baselines are not independently reproducible from the information given. The resource itself remains valuable, but the paper's central quantitative claim needs to be either substantiated with an independent annotation protocol or explicitly weakened.

major comments (3)
  1. [§2.2, §3.3.1, §3.1.2, §3.1.3, §2.8] The comparison between FINER and the neural systems is threatened by circularity. Section 2.2 states that annotator D extended FINER using the annotated corpus as a development set, producing 'a set of possible annotation errors' and 'refinement suggestions', after which the annotation practices were updated and the whole corpus re-annotated. Section 3.3.1 confirms that FINER rules were applied to the training data and 'inconsistent annotations were spotted and further discussed'. The test sets were then annotated 'by replicating the annotation process' (Sections 3.1.2 and 3.1.3), so the FINER-informed guidelines carry over to evaluation. The Section 2.8 inter-annotator agreement does not break this loop because the external annotator followed the same FINER-derived guidelines. This does not by itself invalidate the corpus, but it means the reported F1 advantage of FINER over the neural systems in Tables 8-11 may be partly an artifact of the gold standard being shaped to FINER's rule behavior. Please either (a) provide an independent re-annotation of at least the test sets by annotators who do not consult FINER or the FINER-derived guidelines, or (b) explicitly reframe the results as a developer-in-the-loop benchmark and avoid claiming that FINER 'outperforms' the neural baselines in an unbiased comparison.
  2. [§3.3.2, §3.4] The neural baseline experiments are not reproducible from the manuscript. Section 3.3.2 gives the model names and the pretrained embeddings, but no hyperparameters (learning rate, batch size, number of epochs, hidden dimension, dropout, early stopping criterion, random seed), no training/validation split details beyond token counts, and no code release. In addition, Tables 8-11 report F1 values without confidence intervals or bootstrap error estimates, and the Wilcoxon test in Section 3.4 is described only as 'standard 2-sided' over 20 subsets, with no p-values or description of what quantity is being paired. Since one of the paper's claims is that FINER outperforms the neural systems, this missing information is load-bearing for the experimental comparison. Please add a reproducibility appendix with hyperparameters, seeds, and p-values, and report per-run variance if any randomness is involved.
  3. [§3.1.3, Tables 10-11] The Wikipedia test set is described as out-of-domain but is acknowledged to be heavily skewed: 26 of 83 articles (31%) are biographies, and music and sports account for roughly another quarter of the articles. This creates a class distribution very different from the Digitoday corpus, and the per-class F1 differences in Tables 10-11 (e.g., ORG F1 of 61.81 for FINER versus 36.41 for GÜNGÖR-NN) may partly reflect domain shift in how the guideline is applied rather than general NER capability. The skew is disclosed, which is good, but the 'out-of-domain' claim should be nuanced: the set is more accurately a small, topic-biased sample from Wikipedia. Please state this limitation more prominently in the experimental setup and in the conclusion.
minor comments (4)
  1. [§2.6, Table 2] There is a numerical inconsistency: the text says the total number of entities increased 'from 18,863 to 19,667', and later refers to '19667 entities', but Table 2 totals to 18,867 top-level and 19,671 total. Please correct the text and table so the counts agree.
  2. [§2.3] The list of date expressions contains duplicate numbering: items 6 and 7 are both 'lokakuun 1. päivä', and items 8 and 9 are both 'vuonna 2016' and 'vuoden 2016 aikana' are assigned repeated numbers. Please renumber the list consistently.
  3. [§3.4] Please report the actual p-values for the Wilcoxon signed-rank tests rather than only saying they are significant at the 0.95 confidence level, and specify whether the test is applied to subset-level F1 scores or to per-entity binary outcomes.
  4. [Tables 8-11] There is a typographical artifact in Tables 8 and 9: the EVENT F1 values are rendered as '6 6.67' instead of '66.67'. Please fix the formatting.

Circularity Check

2 steps flagged · score 6.0 of 10

FINER's reported advantage is partially circular: the gold-standard annotation was co-developed with FINER and the test sets replicate that FINER-informed process.

  1. fitted input called prediction [Section 2.2 (Annotation Process); Section 3.3.1 (FINER)]
    "annotator D had gathered 1) a set of possible annotation errors given the annotation practices provided by the first author and 2) a set of refinement suggestions to the annotation practices. Subsequent to a discussion between annotators A, B, C, and D, the named entity categories and annotation practices were updated, and the annotation of the complete data was refined for the second and final time by annotator A. [...] The rules were applied to the training data and inconsistent annotations were spotted and further discussed."

    FINER's rule set was actively developed against the same corpus used as the gold standard: its development produced 'possible annotation errors' and 'refinement suggestions' that led to updating the categories and practices and to re-annotating the whole corpus, and its rules were then used to spot further 'inconsistent annotations'. The gold standard was therefore partly adjusted to conform to FINER's behavior. FINER's subsequently reported high F1 is partially a fit to a target that FINER itself helped define, rather than an independent measurement.

  2. other [Section 3.1.2 (Digitoday test set); Section 3.1.3 (Wikipedia test set); Section 2.8 (Inter-Annotator Agreement)]
    "The data set was annotated by replicating the annotation process described in Section 2: the text was first annotated by a single annotator following the annotation practices described in Section 2.4, after which correction suggestions were provided by a second annotator, and annotation was formed subsequent to a discussion session between the annotators. [...] We enlisted an additional annotator who, based on the annotation guideline, independently annotated 762 uniformly sampled sentences [...] from the Digitoday test set."

    The evaluation sets are not independent of FINER: they are annotated by 'replicating' the same process whose annotation practices were refined using FINER's suggestions (Section 2.2). The inter-annotator agreement check in Section 2.8 does not break this loop because the external annotator also followed the same FINER-derived guideline; the kappa scores measure consistency with the guideline, not independence from the system being evaluated. Consequently the reported F1 advantage of FINER over the neural systems (Tables 8-11) is partly an artifact of the gold standard having been shaped to FINER's rule behavior.

full rationale

The paper is transparent about the co-development loop, and the corpus itself is a useful independent resource; the F1 numbers are honestly computed empirical measurements rather than mathematical derivations from the inputs. However, the central comparative claim that FINER outperforms the neural systems on both test sets is partially circular: FINER's rule development was a direct input into the annotation guidelines used to create the gold labels, and both test sets were annotated by replicating that same FINER-informed process. The external IAA does not remove the circularity because the additional annotator followed the same FINER-derived guideline. This warrants a score of 6 rather than 0-2: the evaluation claim is not fully independent, although it is not a pure renaming or a result forced solely by definition.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's central claims rest on the quality of the manual annotations and on standard evaluation methodology; no free parameters are fitted to produce the corpus, and the F1 scores are empirical measurements. The only listed free parameter is the neural baseline configuration, which is not reported in detail.

free parameters (1)
  • Neural baseline hyperparameters (GUNGER-NN, SOHRAB-NN) = not reported in this paper
    The models are trained using the original papers' settings plus morphological features; the exact values are not listed, limiting exact reproduction but not invalidating the comparisons.
assumptions (3)
  • domain assumption The annotation guidelines in Sections 2.3 and 2.4 correctly and consistently define the six named entity classes for Finnish.
    The entire corpus and all evaluation results depend on the gold labels being trustworthy. This is checked only via one external annotator on 762 sentences (Section 2.8), yielding kappa 0.79 for top-level entities.
  • standard math Cohen's kappa as computed in Section 2.8, with partial span matches treated as errors, is a valid measure of annotation agreement.
    The recasting of NER agreement as entity classification follows Artstein and Poesio (2008), but the strict treatment of partial matches makes the metric conservative.
  • standard math The Wilcoxon signed-rank test over 20 randomly divided subsets of the test set is an appropriate significance test for the reported F1 differences.
    The test is described in Section 3.4, but the exact construction of the 20 subsets is not specified, so the significance claim may be hard to reproduce exactly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Finnish News Corpus for Named Entity Recognition." pith.science (2026). https://pith.science/paper/GLY2JMAJ

@misc{pith2026190804212,
  author       = {Pith},
  title        = {Pith review of: A Finnish News Corpus for Named Entity Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLY2JMAJ}},
  note         = {Machine review of arXiv:1908.04212}
}
read the original abstract

We present a corpus of Finnish news articles with a manually prepared named entity annotation. The corpus consists of 953 articles (193,742 word tokens) with six named entity classes (organization, location, person, product, event, and date). The articles are extracted from the archives of Digitoday, a Finnish online technology news source. The corpus is available for research purposes. We present baseline experiments on the corpus using a rule-based and two deep learning systems on two, in-domain and out-of-domain, test sets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

  1. [5]

    Nested named entity recogn ition revisited

    Arzoo Katiyar and Claire Cardie. Nested named entity recogn ition revisited. In Proceedings of The Sixteenth Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human La nguage Technolo- gies (NAACL HLT 2018), pages 861–871,

    work page 2018

  2. [7]

    Old content and modern tools-searching named enti ties in a Finnish OCRed historical newspaper collection 1771-1910

    Kimmo Kettunen, Eetu Mäkelä, Teemu Ruokolainen, Juha Kuokk ala, and Laura Löfberg. Old content and modern tools-searching named enti ties in a Finnish OCRed historical newspaper collection 1771-1910. DHQ: Digital Humanities Quarterly, 11(3),

    work page 1910

  3. [14]

    A highly accurate named entity corpus for Hungarian

    György Szarvas, Richárd Farkas, László Felföldi, András Kocsor, and János Csirik. A highly accurate named entity corpus for Hungarian. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), pages 1957–1960,

    work page 2006

  4. [16]

    Introduction to the CoNLL-2003 shared task: Language-independent named entity recogniti on

    Erik F Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recogniti on. In Proceedings of the Seventh Conference on Natural Language Learning (HLT -NAACL 2003), pages 142–147,

    work page 2003

  5. [17]

    32 Dimitrios Kokkinakis

    URL https://osf.io/w9nhb. 32 Dimitrios Kokkinakis. Swedish NER in the Nomen Nescio proje ct. Nordisk Sprogteknologi – Nordic Language Technology 2002 , pages 379–398,

    work page 2002

  6. [18]

    Semeval-2013 task 1: Tempeva l-3: Evaluating time expressions, events, and temporal relations

    Naushad UzZaman, Hector Llorens, Leon Derczynski, Marc V er hagen, James Allen, and James Pustejovsky. Semeval-2013 task 1: Tempeva l-3: Evaluating time expressions, events, and temporal relations. In Proceedings of Second Joint Conference on Lexical and Computational Semantics (SEM 201 3), pages 1–9,

    work page 2013

  7. [19]

    Semeval- 2010 task 13: Tempeval-2

    Marc V erhagen, Roser Sauri, Tommaso Caselli, and James Pust ejovsky. Semeval- 2010 task 13: Tempeval-2. In Proceedings of the Fifth International Workshop on Semantic Evaluation (SEMEVAL

    work page 2010

  8. [1977]

    Finding function in form: Compositional character models for open vocabulary word representation

    Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Languag e Processing (EMNLP 2015), pages 1520–1530,

    work page 2015

Show all 20 references
  1. [1996]

    Improving named entity recog- nition by jointly learning to disambiguate morphological t ags

    Onur Güngör, Suzan Üsküdarli, and Tunga Güngör. Improving named entity recog- nition by jointly learning to disambiguate morphological t ags. In Proceedings of the Twenty-Seventh International Conference on Computa tional Linguistics (COLING 2018), pages 2082–2092,

    2018

  2. [2002]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manni ng. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on Em- pirical Methods in Natural Language Processing (EMNLP 2015 ), pages 1532– 1543,

    2014

  3. [2003]

    Neural architectures for named en tity recogni- tion

    Guillaume Lample, Miguel Ballesteros, Sandeep Subramania n, Kazuya Kawakami, and Chris Dyer. Neural architectures for named en tity recogni- tion. In Proceedings of The Fifteenth Annual Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2016

  4. [2006]

    AnC ora: Multilevel annotated corpora for Catalan and Spanish

    Mariona Taulé, Maria Antònia Martí, and Marta Recasens. AnC ora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of the Sixth Interna- tional Conference on Language Resources and Evaluation (LR EC 2008), pages 96–101,

    2008

  5. [2007]

    The GENIA corpu s: An anno- tated research abstract corpus in molecular biology domain

    Tomoko Ohta, Y uka Tateisi, and Jin-Dong Kim. The GENIA corpu s: An anno- tated research abstract corpus in molecular biology domain . In Proceedings of the Second International Conference on Human Language Technology Research (HLT 2002), pages 82–86,

    2002

  6. [2008]

    NoSta-D named entity annotation for German: Guidelines and dataset

    Darina Benikova, Chris Biemann, and Marc Reznicek. NoSta-D named entity annotation for German: Guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pages 2524–2531,

    2014

  7. [2010]

    A neural transition-based model for nested mention recognition

    Bailin Wang, Wei Lu, Y u Wang, and Hongxia Jin. A neural transition-based model for nested mention recognition. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing (EMNLP 2018 ), pages 1011– 1017,

    2018

  8. [2013]

    Semeval-2007 Task 15: Tempeval temp oral relation identification

    Marc V erhagen, Robert Gaizauskas, Frank Schilder, Mark Hep ple, Graham Katz, and James Pustejovsky. Semeval-2007 Task 15: Tempeval temp oral relation identification. In Proceedings of the F ourth International Workshop on Semantic Evaluation (SEMEVAL 2007), pages 75–80,

    2007

  9. [2014]

    Nested named entity recognition in historical a rchive text

    Kate Byrne. Nested named entity recognition in historical a rchive text. In Proceed- ings of the First IEEE International Conference on Semantic Computing (ICSC 2007), pages 589–596,

    2007

  10. [2016]

    Deep exhaustive mode l for nested named entity recognition

    Mohammad Golam Sohrab and Makoto Miwa. Deep exhaustive mode l for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP

    2018

  11. [2017]

    Incorporating non- local information into information extraction systems by G ibbs sampling

    Jenny Rose Finkel, Trond Grenager, and Christopher Manning. Incorporating non- local information into information extraction systems by G ibbs sampling. In Proceedings of the F orty-Third Annual Meeting on Associati on for Computa- tional Linguistics (ACL 2005), pages 363–370,

    2005

  12. [2018]

    Tagging named entities in nineteenth century and modern Finnish newspaper material with a Finnish semant ic tagger

    Kimmo Kettunen and Laura Löfberg. Tagging named entities in nineteenth century and modern Finnish newspaper material with a Finnish semant ic tagger. In Pro- ceedings of the Twenty-First Nordic Conference on Computat ional Linguistics (NODALIDA 2017), pages 29–36,

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.