{"id":"e72ce9c4-6c14-41f7-9a33-fa24aa6202d6","arxiv_id":"2506.23051","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MariNER is a new manually annotated NER dataset for early 20th-century Brazilian Portuguese, with benchmark results showing fine-tuned transformers outperform large language models.","lead":"The authors built MariNER, a hand-annotated dataset of 9,649 sentences from early 20th-century Brazilian travel records and historical articles, labeled with people, places, organizations, and dates. It aims to support digital humanities researchers working with historical Brazilian Portuguese, and the paper benchmarks several language models on it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard claim unsupported: no inter-annotator agreement is reported, and the dataset has no public link, so label quality and reproducibility cannot be checked.","rationale":"The reader's weakest assumption is the right one; the conditional verdict is appropriate. I looked for a different or more serious flaw. The novelty claim is consistent with the cited literature ([6] is silver-standard, [14] is 18th-century European Portuguese), the k-shot and cross-dataset experiments are internally coherent, and the tables agree with the prose. The weakest step is the gold-quality claim: it is not a cosmetic omission. Both the resource contribution and the 0.922 benchmark number rest on labels whose agreement was never measured, and the paper itself discloses that the initial annotations were made without guidelines and only refined later. One independent re-annotation study would settle the point; if agreement is high, the conditional can be lifted. I do not see grounds for rejection, because the process is described transparently enough to be tested and corrected.","tokens_in":10287,"tokens_out":8112,"duration_ms":90258,"concrete_test":"Have two independent annotators, blind to the existing labels and to each other, re-annotate a random sample of at least 10% of the sentences (about 965 sentences) using the Section 3.4 guidelines, and compute entity-level Cohen's kappa or pairwise F1 on exact spans, per entity type and overall. If overall agreement is below roughly 0.8, the 'gold-standard' label is not supported and the 0.922 test F1 cannot be trusted; if agreement is high, the quality concern is resolved and only the missing public release would remain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MariNER is a gold-standard NER dataset, but label quality is never demonstrated. Section 3.5 reports three annotation rounds yet gives no inter-annotator agreement, per entity type or overall. The first round was conducted 'without explicit guidelines' by a team of three computer scientists and one historian-architect and only PERSON and LOCATION were annotated; the second and third rounds 'refined' and 'reviewed ambiguous mentions' rather than independently re-annotating the data. Section 3.3 also refers to 'an annotator' manually verifying OCR, without explaining how disagreements between annotators were resolved. Because first-round decisions can propagate into the final labels, 'gold-standard' is asserted rather than measured. The reported 0.922 micro-F1 for XLM-RoBERTa is computed against these labels, so the benchmark inherits the same uncertainty. The absence of a public link is a second, independent blocker: the dataset cannot be inspected or re-annotated, so neither quality nor reproducibility can be checked by the community.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents MariNER, a manually annotated NER dataset for early 20th-century Brazilian Portuguese. The corpus comprises 9,649 sentences from five historical documents (travel records and articles about the Brazilian North-East), annotated with four entity types (PERSON, LOCATION, DATE, ORGANIZATION) and split into train/eval/test. The authors describe the three-round annotation process, report dataset statistics, and benchmark BiLSTM-CRF, BERTimbau, XLM-RoBERTa, and three LLMs in zero- and few-shot settings, plus cross-dataset transfer from HAREM and LeNER-BR. The best result is 0.922 micro-F1 for XLM-RoBERTa with a linear classification head.","tokens_in":10493,"tokens_out":6141,"duration_ms":61234,"significance":"The dataset addresses a real gap: there is currently no gold-standard NER dataset for early 20th-century Brazilian Portuguese. The paper is clearly written and the benchmark protocol is standard; the model comparisons and the demonstration that LLMs underperform task-specific models are useful empirical results. The main weakness is that the central claim that MariNER is 'gold-standard' is not yet evidenced: no inter-annotator agreement metric is reported and no public link to the dataset is provided, so label quality and reproducibility cannot be verified. If these issues are addressed, the contribution would be a valuable resource for Portuguese NLP and digital humanities.","major_comments":[{"comment":"The abstract and Section 3.5 call MariNER 'gold-standard', but the annotation procedure as described cannot support that claim. The first round was performed without explicit guidelines by annotators who marked only PERSON and LOCATION; the second round refined existing labels and the third reviewed ambiguous mentions. Because no round independently re-annotated a sample, no inter-annotator agreement can be computed and first-round labeling decisions may propagate into the final release. Please report a standard agreement metric (e.g., Cohen's kappa or entity-level F1 agreement) on a sample independently annotated by at least two annotators using the final guidelines, overall and per entity type. Also clarify in §3.3 how the single 'annotator' verified OCR and how disagreements between annotators were resolved.","section":"§3.5 and Abstract"},{"comment":"No URL, repository, or license is given anywhere in the manuscript. For a dataset paper, the artifact is the contribution; without a persistent link and license the community cannot inspect the labels, re-annotate a sample, or reproduce the figures in Table 3. Please provide a DOI or permanent repository link, along with the full annotation guidelines and, if possible, the evaluation scripts.","section":"Availability"},{"comment":"The cross-dataset experiments in Section 5.4 use HAREM and LeNER-BR, whose entity sets are supersets of MariNER's (HAREM includes Value; LeNER-BR includes Law and Jurisprudence). The paper does not explain how these extra labels are treated when training/evaluating on MariNER's four types. If extra labels are removed from the training set, this could reduce performance; if they are mapped, the mapping should be described. Please state this explicitly so Table 4 is interpretable.","section":"§5.4"},{"comment":"Section 3.6 does not state whether the train/eval/test split is random at the sentence level or document-level. Given that the corpus consists of only five documents and contains highly frequent repetitive mentions (e.g., Piauí accounts for 5.31% of LOCATION mentions per Table 2), a sentence-level random split may put near-duplicate context in both training and test. Please specify the split criterion, and if sentence-level, also report a document-level split to gauge generalization to unseen documents.","section":"§3.6"}],"minor_comments":[{"comment":"'XML-RoBERTa' is a typo for 'XLM-RoBERTa'.","section":"§4.2"},{"comment":"The example sentence has irregular spacing ('Chegamosem Juazeiro em Janeiro .do Norte'); use the same tokenization shown in the BIO tags.","section":"Figure 1"},{"comment":"The example JSON contains a trailing comma, which is invalid JSON; since the text-to-text pipeline relies on parsing JSON, the example should be valid.","section":"Figure 2"},{"comment":"The heading 'T able 1' has an extra space.","section":"Table 1"},{"comment":"The report says LLM experiments were run once because temperature was 0; consider noting that this reduces variance but does not guarantee determinism across software versions.","section":"§4.3"},{"comment":"Reference 11 is listed as omitted for anonymization; in the final version, the full thesis citation is needed because it is the source of the documents and of the first annotation round.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The main risk is that the dataset and its quality controls are not currently accessible; this is fixable. I would also verify that the final version includes the full citation for thesis [11] and explicitly states the relationship between the annotations from [11] and the new labels, to avoid any double-publication or attribution issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MariNER fills a real gap: it is the first manually annotated NER corpus for early 20th-century Brazilian Portuguese, and the authors document that gap well. The annotation process is described in detail, the entity definitions are sensible, and the benchmark evaluation is standard and competent. I would like to see this paper succeed, but two things keep me from endorsing the gold-standard claim as it stands.\n\nFirst, no inter-annotator agreement is reported. Section 3.5 describes three annotation rounds, but they are not independent re-annotations. The first round, done without guidelines and only for PERSON and LOCATION, was carried out by a team of three CS researchers and one historian; the second and third rounds refined those annotations rather than providing independent labels. That means first-round decisions can propagate into the final labels, and the reported 0.922 micro-F1 for XLM-RoBERTa inherits that uncertainty. For a resource paper, this is not fatal, but it means \"gold-standard\" is asserted rather than demonstrated. An IAA metric, even on a sample, would settle it.\n\nSecond, the dataset is not publicly accessible. No link or repository is given. For a resource contribution this is a major omission: reviewers and future users cannot inspect the data or reproduce the annotation or the benchmark. This needs to be fixed before the paper can be fully valuable.\n\nA third, minor issue: the cross-dataset mapping of entity types with HAREM and LeNER-BR is not explained. HAREM includes Value, LeNER-BR includes Law and Jurisprudence; how were those treated when training and evaluating on MariNER? A short clarification would help.\n\nThe citation pattern looks fine. The self-citation to the thesis (reference 20) is transparent because the documents were originally compiled there for discourse analysis. The NER labeling is a new extension, so there is no circularity problem.\n\nBottom line: this is a potentially valuable resource for digital humanities and Portuguese NLP. The benchmark numbers will be useful once the data is out. I would send this to peer review with a clear expectation of major revision: add IAA, release the data, and detail the entity-type mapping. The core contribution is solid enough to warrant that effort.","headline":"Useful new resource for historical Brazilian Portuguese NER, but the gold-standard claim currently rests on trust: no IAA and no public data link.","tokens_in":10991,"tokens_out":2113,"would_cite":false,"duration_ms":21706,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds MariNER, the first gold-standard named-entity recognition dataset for early 20th-century Brazilian Portuguese, with over 9,000 manually annotated sentences, and shows that fine-tuned XLM-RoBERTa reaches a micro-F1 of…","keywords":["Named entity recognition","Brazilian Portuguese","Historical text","Digital humanities","Gold-standard dataset","XLM-RoBERTa","Portuguese NER","Manual annotation"],"falsifier":"Independently re-annotate a random sample of MariNER sentences with annotators who have not seen the gold labels and compute pairwise inter-annotator agreement; if the agreement is well below the level expected of gold-standard corpora (e.g., Cohen's kappa below 0.8), the central gold-standard claim is falsified.","tokens_in":10145,"feed_emoji":"📜","tokens_out":8205,"duration_ms":75919,"temperature":0.7,"pith_summary":"This paper introduces MariNER, a new resource that its authors claim to be the first gold-standard named-entity recognition dataset for early 20th-century Brazilian Portuguese. The dataset contains more than 9,000 manually annotated sentences taken from five historical travel records and articles about expeditions through northeastern Brazil. The paper's central contention is that these annotations fill a gap that previously forced researchers to rely on modern Portuguese or silver-standard automatic labels, and that they enable digital-humanities analysis of historical records. To support the claim, the authors benchmark eight models, reporting that fine-tuned XLM-RoBERTa reaches a micro-F1 of 0.922 on the test set, while large language models in zero- and few-shot settings stay far behind. If the resource is released as described, it gives historical Brazilian Portuguese a needed benchmark for entity recognition.","feed_headline":"XLM-RoBERTa tops new historical Brazilian Portuguese NER benchmark","feed_subtitle":"The first gold-standard NER benchmark for early-1900s Brazilian Portuguese, built from travel records.","key_machinery":"The central object is the dataset and its annotation pipeline. Five photographs of original early 20th-century texts are OCR'ed, manually corrected, normalized to standard Brazilian Portuguese, and split into 9,649 sentences using a Portuguese tokenizer; a four-person team then labels entity mentions in three rounds, with the final round reviewing ambiguous cases. Evaluation uses the standard BIO sequence-labeling setup, where token-level predictions are scored at entity level with precision, recall, and micro-F1, across LSTM-CRF, BERT-encoder, and LLM text-to-text configurations.","core_discovery":"On its own terms, the paper's discovery is that a manually annotated, normalized corpus of early 20th-century Brazilian Portuguese can support high-accuracy NER: it constructs MariNER from five documents, annotates PERSON, LOCATION, DATE, and ORGANIZATION, and reports that the best model, XLM-RoBERTa with a linear classifier, achieves 0.922 micro-F1 on the test set. The paper also finds that ORGANIZATION is the hardest category, with the best F1 at 0.640, and that models trained on MariNER's own data outperform models trained on other Portuguese corpora by roughly 20 points. This last result is used to argue that domain-specific historical annotation is necessary rather than optional.","pith_inferences":["Because the texts are normalized to modern spelling before annotation, the dataset measures NER on a cleaned version of history; a companion version preserving original spellings would test how much spelling variation matters.","No inter-annotator agreement is reported, so the gold-standard claim would be directly testable by re-annotating a sample and measuring agreement.","The heavy LOCATION majority (4,428 of 6,644 entities) means micro-F1 is dominated by the easiest class; macro-F1 or per-type analysis would expose how much harder the rarer types really are.","If the paper makes the underlying digitized images or raw OCR available, MariNER could become a testbed for the interaction between OCR noise and NER, which the normalization step currently removes."],"forward_implications":["Digital-humanities researchers can run automated person, place, date, and organization extraction on early 20th-century Brazilian Portuguese with a benchmark that previously did not exist.","The reported 0.922 micro-F1 shows that fine-tuned multilingual transformers are a practical option for this historical domain.","The roughly 20-point gap between in-domain and cross-dataset training implies that models will need domain-specific historical annotation rather than modern or legal Portuguese data.","The low ORGANIZATION scores (best 0.640) indicate that rare and context-dependent entity types remain a weak spot in historical NER.","Zero-shot and few-shot prompting of large language models is not competitive, with the best LLM at 0.648 micro-F1, suggesting that this task still favors fine-tuned models."],"supporting_citations":[{"why":"Supplies the five historical documents the dataset is built from, originally compiled for discourse analysis.","marker":"[11]"},{"why":"Provides the annotation guidelines and entity definitions adapted for MariNER.","marker":"[13]"},{"why":"The closest existing gold-standard historical Portuguese NER corpus, used to motivate the resource and as template for normalizing dated spellings; also supplies an LLM prompting strategy considered in the experiments.","marker":"[14]"},{"why":"A silver-standard historical Brazilian Portuguese corpus whose existence frames the lack of a gold-standard dataset.","marker":"[6]"},{"why":"LeNER-BR, one of the two cross-dataset training sources used to test generalization.","marker":"[9]"},{"why":"XLM-RoBERTa, the encoder model that achieves the best micro-F1 on MariNER.","marker":"[2]"},{"why":"BERTimbau, the Portuguese BERT model used as the other main encoder baseline.","marker":"[16]"},{"why":"Supplies the list-generation approach used to turn LLM outputs into extracted entities for the text-to-text experiments.","marker":"[23]"}],"fun_headline_variants":["MariNER: first gold-standard NER dataset for early-1900s Portuguese","XLM-RoBERTa leads new historical Portuguese NER with 92% F1","New benchmark for historical Brazilian Portuguese NER: MariNER","Historical Portuguese NER gets 20-point boost from domain data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's gold-standard claim rests on the assumption that the three-round manual annotation, performed without any reported measure of agreement between annotators, produced consistent and correct labels.","fun_headline_variants_meta":{"raw":{"variants":["MariNER: first gold-standard NER dataset for early-1900s Portuguese","XLM-RoBERTa leads new historical Portuguese NER with 92% F1","New benchmark for historical Brazilian Portuguese NER: MariNER","Historical Portuguese NER gets 20-point boost from domain data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1912,"prompt_tokens":845,"completion_tokens":1067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":986}},"tokens_in":461,"tokens_out":1067,"duration_ms":10470,"temperature":1.0,"reasoning_tokens":986,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:50:23.524150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample of MariNER sentences with annotators who have not seen the gold labels and compute pairwise inter-annotator agreement; if the agreement is well below the level expected of gold-standard corpora (e.g., Cohen's kappa below 0.8), the central gold-standard claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the five historical documents the dataset is built from, originally compiled for discourse analysis."},{"cited_title":"In: Calzolari, N., Choukri, K., Gangemi, A., Mae- gaard, B., Mariani, J., Odijk, J., Tapias, D","cited_arxiv_id":null,"evidence_quote":"Provides the annotation guidelines and entity definitions adapted for MariNER."},{"cited_title":"In: Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H.G., Amaro, R","cited_arxiv_id":null,"evidence_quote":"The closest existing gold-standard historical Portuguese NER corpus, used to motivate the resource and as template for normalizing dated spellings; also supplies an LLM prompting strategy considered in the experiments."},{"cited_title":"In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (May 2018)","cited_arxiv_id":null,"evidence_quote":"A silver-standard historical Brazilian Portuguese corpus whose existence frames the lack of a gold-standard dataset."},{"cited_title":"In: Villavicencio, A., Moreira, V., Abad, A., Caseli, H., Gamallo, P., Ramisch, C., Gonçalo Oliveira, H., Paetzold, G.H","cited_arxiv_id":null,"evidence_quote":"LeNER-BR, one of the two cross-dataset training sources used to test generalization."}],"review_version":1}