Pith. sign in

REVIEW 4 major objections 6 minor 25 references

MariNER: A Dataset for Historical Brazilian Portuguese Named Entity Recognition

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper builds MariNER, the first gold-standard named-entity recognition dataset for early 20th-century Brazilian Portuguese, with over 9,000 manually annotated sentences, and shows that fine-tuned XLM-RoBERTa reaches a micro-F1 of…

desk verdict Useful new resource for historical Brazilian Portuguese NER, but the gold-standard claim currently rests on trust: no IAA and no public data link. read the letter →

arxiv 2506.23051 v1 pith:GEWBFHGS submitted 2025-06-29 cs.CL

João Lucas Luz Lima Sarcinelli , Marina Lages Gonçalves Teixeira , Jade Bortot de Paiva , Diego Furtado Silva This is my paper · ORCID
classification cs.CL
keywords NamedentityrecognitionBrazilianPortugueseHistoricaltextDigitalhumanitiesGold-standarddatasetXLM-RoBERTaNERManualannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MariNER, a new resource that its authors claim to be the first gold-standard named-entity recognition dataset for early 20th-century Brazilian Portuguese. The dataset contains more than 9,000 manually annotated sentences taken from five historical travel records and articles about expeditions through northeastern Brazil. The paper's central contention is that these annotations fill a gap that previously forced researchers to rely on modern Portuguese or silver-standard automatic labels, and that they enable digital-humanities analysis of historical records. To support the claim, the authors benchmark eight models, reporting that fine-tuned XLM-RoBERTa reaches a micro-F1 of 0.922 on the test set, while large language models in zero- and few-shot settings stay far behind. If the resource is released as described, it gives historical Brazilian Portuguese a needed benchmark for entity recognition.

What carries the argument

The central object is the dataset and its annotation pipeline. Five photographs of original early 20th-century texts are OCR'ed, manually corrected, normalized to standard Brazilian Portuguese, and split into 9,649 sentences using a Portuguese tokenizer; a four-person team then labels entity mentions in three rounds, with the final round reviewing ambiguous cases. Evaluation uses the standard BIO sequence-labeling setup, where token-level predictions are scored at entity level with precision, recall, and micro-F1, across LSTM-CRF, BERT-encoder, and LLM text-to-text configurations.

What would settle it

Independently re-annotate a random sample of MariNER sentences with annotators who have not seen the gold labels and compute pairwise inter-annotator agreement; if the agreement is well below the level expected of gold-standard corpora (e.g., Cohen's kappa below 0.8), the central gold-standard claim is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a manually annotated, normalized corpus of early 20th-century Brazilian Portuguese can support high-accuracy NER: it constructs MariNER from five documents, annotates PERSON, LOCATION, DATE, and ORGANIZATION, and reports that the best model, XLM-RoBERTa with a linear classifier, achieves 0.922 micro-F1 on the test set. The paper also finds that ORGANIZATION is the hardest category, with the best F1 at 0.640, and that models trained on MariNER's own data outperform models trained on other Portuguese corpora by roughly 20 points. This last result is used to argue that domain-specific historical annotation is necessary rather than optional.

Load-bearing premise

The dataset's gold-standard claim rests on the assumption that the three-round manual annotation, performed without any reported measure of agreement between annotators, produced consistent and correct labels.

Editorial extensions

If this is right

  • Digital-humanities researchers can run automated person, place, date, and organization extraction on early 20th-century Brazilian Portuguese with a benchmark that previously did not exist.
  • The reported 0.922 micro-F1 shows that fine-tuned multilingual transformers are a practical option for this historical domain.
  • The roughly 20-point gap between in-domain and cross-dataset training implies that models will need domain-specific historical annotation rather than modern or legal Portuguese data.
  • The low ORGANIZATION scores (best 0.640) indicate that rare and context-dependent entity types remain a weak spot in historical NER.
  • Zero-shot and few-shot prompting of large language models is not competitive, with the best LLM at 0.648 micro-F1, suggesting that this task still favors fine-tuned models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the texts are normalized to modern spelling before annotation, the dataset measures NER on a cleaned version of history; a companion version preserving original spellings would test how much spelling variation matters.
  • No inter-annotator agreement is reported, so the gold-standard claim would be directly testable by re-annotating a sample and measuring agreement.
  • The heavy LOCATION majority (4,428 of 6,644 entities) means micro-F1 is dominated by the easiest class; macro-F1 or per-type analysis would expose how much harder the rarer types really are.
  • If the paper makes the underlying digitized images or raw OCR available, MariNER could become a testbed for the interaction between OCR noise and NER, which the normalization step currently removes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents MariNER, a manually annotated NER dataset for early 20th-century Brazilian Portuguese. The corpus comprises 9,649 sentences from five historical documents (travel records and articles about the Brazilian North-East), annotated with four entity types (PERSON, LOCATION, DATE, ORGANIZATION) and split into train/eval/test. The authors describe the three-round annotation process, report dataset statistics, and benchmark BiLSTM-CRF, BERTimbau, XLM-RoBERTa, and three LLMs in zero- and few-shot settings, plus cross-dataset transfer from HAREM and LeNER-BR. The best result is 0.922 micro-F1 for XLM-RoBERTa with a linear classification head.

Significance. The dataset addresses a real gap: there is currently no gold-standard NER dataset for early 20th-century Brazilian Portuguese. The paper is clearly written and the benchmark protocol is standard; the model comparisons and the demonstration that LLMs underperform task-specific models are useful empirical results. The main weakness is that the central claim that MariNER is 'gold-standard' is not yet evidenced: no inter-annotator agreement metric is reported and no public link to the dataset is provided, so label quality and reproducibility cannot be verified. If these issues are addressed, the contribution would be a valuable resource for Portuguese NLP and digital humanities.

major comments (4)
  1. [§3.5 and Abstract] The abstract and Section 3.5 call MariNER 'gold-standard', but the annotation procedure as described cannot support that claim. The first round was performed without explicit guidelines by annotators who marked only PERSON and LOCATION; the second round refined existing labels and the third reviewed ambiguous mentions. Because no round independently re-annotated a sample, no inter-annotator agreement can be computed and first-round labeling decisions may propagate into the final release. Please report a standard agreement metric (e.g., Cohen's kappa or entity-level F1 agreement) on a sample independently annotated by at least two annotators using the final guidelines, overall and per entity type. Also clarify in §3.3 how the single 'annotator' verified OCR and how disagreements between annotators were resolved.
  2. [Availability] No URL, repository, or license is given anywhere in the manuscript. For a dataset paper, the artifact is the contribution; without a persistent link and license the community cannot inspect the labels, re-annotate a sample, or reproduce the figures in Table 3. Please provide a DOI or permanent repository link, along with the full annotation guidelines and, if possible, the evaluation scripts.
  3. [§5.4] The cross-dataset experiments in Section 5.4 use HAREM and LeNER-BR, whose entity sets are supersets of MariNER's (HAREM includes Value; LeNER-BR includes Law and Jurisprudence). The paper does not explain how these extra labels are treated when training/evaluating on MariNER's four types. If extra labels are removed from the training set, this could reduce performance; if they are mapped, the mapping should be described. Please state this explicitly so Table 4 is interpretable.
  4. [§3.6] Section 3.6 does not state whether the train/eval/test split is random at the sentence level or document-level. Given that the corpus consists of only five documents and contains highly frequent repetitive mentions (e.g., Piauí accounts for 5.31% of LOCATION mentions per Table 2), a sentence-level random split may put near-duplicate context in both training and test. Please specify the split criterion, and if sentence-level, also report a document-level split to gauge generalization to unseen documents.
minor comments (6)
  1. [§4.2] 'XML-RoBERTa' is a typo for 'XLM-RoBERTa'.
  2. [Figure 1] The example sentence has irregular spacing ('Chegamosem Juazeiro em Janeiro .do Norte'); use the same tokenization shown in the BIO tags.
  3. [Figure 2] The example JSON contains a trailing comma, which is invalid JSON; since the text-to-text pipeline relies on parsing JSON, the example should be valid.
  4. [Table 1] The heading 'T able 1' has an extra space.
  5. [§4.3] The report says LLM experiments were run once because temperature was 0; consider noting that this reduces variance but does not guarantee determinism across software versions.
  6. [References] Reference 11 is listed as omitted for anonymization; in the final version, the full thesis citation is needed because it is the source of the documents and of the first annotation round.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the dataset construction and benchmark evaluation are self-contained empirical work.

full rationale

The paper's central contribution is the construction of a manually annotated NER dataset and an empirical benchmark of existing models on it. There is no derivation chain in which a predicted quantity is defined in terms of the input or vice versa. The only in-scope self-reference is provenance: the source documents and the first-round PERSON/LOCATION annotations come from the authors' own prior thesis, cited as [11] (and the OCR/normalization process is described in [20], also by the same group). This is data provenance rather than a load-bearing proof step: the final dataset extends those annotations with new entity types, additional annotation rounds, and formal guidelines, and the reported XLM-RoBERTa F1 of 0.922 is computed on a held-out test set relative to the resulting human labels. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in through citation. Concerns about the absence of inter-annotator agreement and the lack of a public dataset link are quality and reproducibility issues, not circularity, and therefore do not affect the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The axioms reflect the dataset construction assumptions: the annotation schema follows HAREM, text normalization changes the surface form, and the document selection is small and geographically focused. No free parameters or invented entities appear in this empirical resource paper.

assumptions (3)
  • domain assumption Entity definitions adapted from HAREM are appropriate for historical Brazilian Portuguese texts.
    Section 3.4 states the entity types were adapted from HAREM; this assumes HAREM's categories transfer to early 20th-century travel records without significant gaps.
  • domain assumption Normalizing original texts to modern Brazilian Portuguese preserves the NER-relevant information.
    Section 3.3 describes normalization; this assumes that spelling changes do not destroy entity boundaries or types needed for digital humanities research.
  • domain assumption The five source documents are representative of early 20th-century Brazilian Portuguese for NER purposes.
    Section 3.1 describes five documents, mainly about Piauí; the paper assumes this small selection is sufficient to support a generally useful dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MariNER: A Dataset for Historical Brazilian Portuguese Named Entity Recognition." pith.science (2026). https://pith.science/paper/GEWBFHGS

@misc{pith2026250623051,
  author       = {Pith},
  title        = {Pith review of: MariNER: A Dataset for Historical Brazilian Portuguese Named Entity Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEWBFHGS}},
  note         = {Machine review of arXiv:2506.23051}
}
read the original abstract

Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task that aims to identify and classify entity mentions in texts across different categories. While languages such as English possess a large number of high-quality resources for this task, Brazilian Portuguese still lacks in quantity of gold-standard NER datasets, especially when considering specific domains. Particularly, this paper considers the importance of NER for analyzing historical texts in the context of digital humanities. To address this gap, this work outlines the construction of MariNER: \textit{Mapeamento e Anota\c{c}\~oes de Registros hIst\'oricos para NER} (Mapping and Annotation of Historical Records for NER), the first gold-standard dataset for early 20th-century Brazilian Portuguese, with more than 9,000 manually annotated sentences. We also assess and compare the performance of state-of-the-art NER models for the dataset.

Figures

Figures reproduced from arXiv: 2506.23051 by the authors.

Figure 1
Figure 1. NER as a sequence classification task. The original sentence is split into tokens and each token is processed by the model, which outputs its class as one of the possible BIO tags. In the example, “Juazeiro do Norte” is a location entity and “Janeiro” is a date entity. 4.2 BERT Based The BERT family of models [3] are pre-trained encoder language models that are popular to perform a great number of NLP tasks. Given a… view at source ↗
Figure 2
Figure 2. NER as a text-to-text task. A prompt is constructed requiring the LLM to ex￾tract the entities of the given sentence. The model outputs a JSON formatted output, which is parsed and matched with the original input to obtain its final BIO classifica￾tion. 5 Experiments For our experiments, we train all of the previous models on MariNER’s train set, evaluate to obtain the best configurations using the eval set and test… view at source ↗
Figure 3
Figure 3. illustrates the performance of LLMs across different K-shot configura￾tions. Interestingly, adding few-shot examples to the input prompts appears to hinder performance rather than improve it. This may be attributed to the rela￾tively smaller size of these locally-run LLMs, as the increased context size from additional examples could negatively affect their capacity to effectively process and extract entities. 0 5 10… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 12 canonical work pages

  1. [1]

    Pinto, M.H., Filho, R.P.S., Costa, R., de M

    Albuquerque, H.O., Souza, E., Gomes, C., de C. Pinto, M.H., Filho, R.P.S., Costa, R., de M. Lopes, V.T., da Silva, N.F.F., de Carvalho, A.C.P.L.F., Oliveira, A.L.I.: Named Entity Recognition: a Survey for the Portuguese Language. Procesamiento del Lenguaje Natural pp. 171–185 (Mar 2023). https://doi.org/10.26342/2023-70- 14, https://doi.org/10.26342/2023-70-14

    work page doi:10.26342/2023-70- 2023

  2. [2]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised Cross-lingual Representation Learning at Scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8440–8451. Association for Computational Linguistics, Online (2020). ...

    doi:10.18653/v1/2020.acl- 2020

  3. [3]

    https://doi.org/10.48550/arXiv.1810.04805

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (May 2019). https://doi.org/10.48550/arXiv.1810.04805

    arXiv doi:10.48550/arxiv.1810.04805 2019

  4. [4]

    ACM Comput- ing Surveys 56(2), 1–47 (Feb 2024)

    Ehrmann, M., Hamdi, A., Pontes, E.L., Romanello, M., Doucet, A.: Named Entity Recognition and Classification in Historical Documents: A Survey. ACM Comput- ing Surveys 56(2), 1–47 (Feb 2024). https://doi.org/10.1145/3604931

    doi:10.1145/3604931 2024

  5. [5]

    https://doi.org/10.48550/arXiv.2407.21783

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., et al.: The Llama 3 Herd of Models (Nov 2024). https://doi.org/10.48550/arXiv.2407.21783

    arXiv doi:10.48550/arxiv.2407.21783 2024

  6. [6]

    In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (May 2018)

    Higuchi, S., Freitas, C., Cuconato, B., Rademaker, A.: Text Mining for History: first steps on building a large dataset. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (May 2018)

    work page 2018

  7. [7]

    Neural Com- put

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Com- put. 9(8), 1735–1780 (Nov 1997). https://doi.org/10.1162/neco.1997.9.8.1735, https://doi.org/10.1162/neco.1997.9.8.1735

    doi:10.1162/neco.1997.9.8.1735 1997

  8. [8]

    IEEE Transactions on Knowledge and Data Engineering34(1), 50– 70 (Jan 2022)

    Li, J., Sun, A., Han, J., Li, C.: A Survey on Deep Learning for Named Entity Recognition. IEEE Transactions on Knowledge and Data Engineering34(1), 50– 70 (Jan 2022). https://doi.org/10.1109/TKDE.2020.2981314

    arXiv 2022

Show all 25 references
  1. [9]

    In: Villavicencio, A., Moreira, V., Abad, A., Caseli, H., Gamallo, P., Ramisch, C., Gonçalo Oliveira, H., Paetzold, G.H

    Luz De Araujo, P.H., De Campos, T.E., De Oliveira, R.R.R., Stauffer, M., Couto, S., Bermejo, P.: LeNER-Br: A Dataset for Named Entity Recognition in Brazilian Legal Text. In: Villavicencio, A., Moreira, V., Abad, A., Caseli, H., Gamallo, P., Ramisch, C., Gonçalo Oliveira, H., ...

    2018 doi

  2. [10]

    Journal of Biomedical Semantics13(1), 13 (Dec 2022)

    Oliveira, L.E.S.E., Peters, A.C., Da Silva, A.M.P., Gebeluca, C.P., Gumiel, Y.B., Cintho, L.M.M., Carvalho, D.R., Al Hasan, S., Moro, C.M.C.: SemClinBr - a multi-institutional and multi-specialty semantically annotated corpus for Por- tuguese clinical NLP tasks. Journal of Bio...

    2022 doi

  3. [11]

    Omitted: The author of this thesis was omitted to avoid misinterpretations during the double-anonymized reviewing process. Ph.D. thesis, University, Brazil (2024)

    2024

  4. [12]

    https://doi.org/10.48550/arXiv.cmp-lg/9505040

    Ramshaw, L.A., Marcus, M.P.: Text Chunking using Transformation-Based Learn- ing (May 1995). https://doi.org/10.48550/arXiv.cmp-lg/9505040

    1995 doi arXiv

  5. [13]

    In: Calzolari, N., Choukri, K., Gangemi, A., Mae- gaard, B., Mariani, J., Odijk, J., Tapias, D

    Santos, D., Seco, N., Cardoso, N., Vilela, R.: HAREM: An advanced NER evalu- ation contest for Portuguese. In: Calzolari, N., Choukri, K., Gangemi, A., Mae- gaard, B., Mariani, J., Odijk, J., Tapias, D. (eds.) Proceedings of the Fifth International Conference on Language Resou...

    2006

  6. [14]

    In: Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H.G., Amaro, R

    Santos, J., Cameron, H.F., Olival, F., Farrica, F., Vieira, R.: Named entity recog- nition specialised for Portuguese 18th-century history research. In: Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H.G., Amaro, R. (eds.) Pro- ceedings of the 16th Inter...

    2024

  7. [15]

    In: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS)

    Santos, J., Consoli, B., Dos Santos, C., Terra, J., Collonini, S., Vieira, R.: Assessing the Impact of Contextual Embeddings for Portuguese Named Entity Recognition. In: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS). pp. 437–442. IEEE, Salvador, Brazil (Oct 201...

    2019

  8. [16]

    Applied Soft Computing149, 110901 (Dec 2023)

    Souza, F., Nogueira, R., Lotufo, R.: BERT models for Brazilian Portuguese: Pretraining, evaluation and tokenization analysis. Applied Soft Computing149, 110901 (Dec 2023). https://doi.org/10.1016/j.asoc.2023.110901

    2023

  9. [17]

    Foun- dations and Trends ® in Machine Learning 4(4), 267–373 (2012)

    Sutton, C.: An Introduction to Conditional Random Fields. Foun- dations and Trends ® in Machine Learning 4(4), 267–373 (2012). https://doi.org/10.1561/2200000013

    2012 doi

  10. [18]

    In: Gelbukh, A

    Tamper, M., Leskinen, P., Hyvönen, E.: Visualizing and Analyzing Networks of Named Entities in Biographical Dictionaries for Digital Humanities Re- search. In: Gelbukh, A. (ed.) Computational Linguistics and Intelligent Text Processing, vol. 13451, pp. 199–214. Springer Nature...

    2023 doi

  11. [19]

    https://doi.org/10.48550/arXiv.2408.00118

    Team, G., Riviere, M., Pathak, S., Sessa, P.G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., et al.: Gemma 2: Improving Open Language Models at a Practical Size (Oct 2024). https://doi.org/10.48550/arXiv.2408.00118

    2024 doi arXiv

  12. [20]

    Tese (doutorado em teoria e história da arquitetura e do urbanismo), Instituto de Arquitetura e Urbanismo, Universidade de São Paulo, São Carlos (2024), [no prelo]

    Teixeira, M.L.G.: Sertões do Piauhy: a construção das narrativas pela literatura de viagem (1912-1938). Tese (doutorado em teoria e história da arquitetura e do urbanismo), Instituto de Arquitetura e Urbanismo, Universidade de São Paulo, São Carlos (2024), [no prelo]

    2024

  13. [21]

    In: COLING-02: The 6th Title Suppressed Due to Excessive Length 15 Conference on Natural Language Learning 2002 (CoNLL-2002) (2002), https://aclanthology.org/W02-2024/

    Tjong Kim Sang, E.F.: Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In: COLING-02: The 6th Title Suppressed Due to Excessive Length 15 Conference on Natural Language Learning 2002 (CoNLL-2002) (2002), https://aclanthology.org/W02-2024/

    2002

  14. [22]

    https://doi.org/10.48550/arXiv.2304.10428

    Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., Wang, G.: GPT-NER: Named Entity Recognition via Large Language Models (Oct 2023). https://doi.org/10.48550/arXiv.2304.10428

    2023 doi arXiv

  15. [23]

    https://doi.org/10.48550/arXiv.2310.10035

    Xie, T., Li, Q., Zhang, J., Zhang, Y., Liu, Z., Wang, H.: Empirical Study of Zero- Shot NER with ChatGPT (Oct 2023). https://doi.org/10.48550/arXiv.2310.10035

    2023 doi arXiv

  16. [24]

    https://doi.org/10.48550/arXiv.2407.10671

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

    2024 doi arXiv

  17. [25]

    https://doi.org/10.48550/arXiv.2303.18223

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.Y., Wen, J.R.: A Survey of Large Language Models (Mar 2025). https://doi.org...

    2025 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.