Pith. sign in

REVIEW 4 major objections 6 minor 20 references

Token and Span Classification for Entity Recognition in French Historical Encyclopedias

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The best evaluated NER system for 18th-century French encyclopedias is fine-tuned CamemBERT with joint-label or multi-label token classification, and one-example GPT prompting is a promising low-resource alternative.

desk verdict Useful NER benchmark on 18th-century French, but the GPT few-shot results are unverified: Table 7 exactly duplicates the spaCy rows. read the letter →

arxiv 2506.02872 v1 pith:IH4RMJF6 submitted 2025-06-03 cs.CL cs.IR

Ludovic Moncla , Hédi Zeghidi This is my paper
classification cs.CLcs.IR
keywords namedentityrecognitionhistoricalFrenchnestedentitiesCamemBERTGeoEDdAfew-shotpromptingtokenclassificationspan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which named-entity recognition (NER) method actually works on text from 18th-century French encyclopedias, where spelling is not standardized and entities are frequently nested inside one another. It benchmarks six approaches on the GeoEDdA corpus: CRFs, spaCy's CNN and span-based taggers, the French transformer CamemBERT, Flair, and few-shot prompting of GPT models. It argues that fine-tuned CamemBERT, using either composite joint labels or multi-label token classification, delivers the highest overall performance, particularly on nested entities. It also argues that prompting a GPT model with a single labeled example is a promising low-resource alternative. If these claims are right, historians and digital humanists gain a concrete default model for extracting places, people, and spatial relations from early modern French, plus a cheap way to bootstrap annotations.

What carries the argument

The load-bearing mechanism is the combination of a nested-annotation dataset and label encodings that let a flat token classifier represent overlap. GeoEDdA marks entities as overlapping spans; the joint-label encoding flattens each overlap into a single composite class, such as "NP-Spatial+ENE-Spatial", and the multi-label encoding treats each of the twelve entity types as an independent binary decision per token. CamemBERT, a French pretrained transformer, carries both heads, and the comparison across architectures is essentially a test of which formulation best converts nested structure into token-level supervision.

What would settle it

Re-run all evaluated systems on the GeoEDdA test set with span-level exact-match F1, counting a prediction as correct only when its start offset, end offset, and entity label all match a gold span; if span-based models such as spaCy Spancat or multi-label CamemBERT overtake single-label CamemBERT under that metric, the paper's conclusion that token classification is the best formulation for nested historical entities is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an empirical ranking on GeoEDdA: when scored with token-level F1, fine-tuned CamemBERT and Flair occupy the top tier, with CamemBERT posting the best macro average (86.4) and the strongest results on nested labels such as ENE-Person, while CRF, spaCy NER, spaCy Spancat, and the best few-shot GPT model place lower. The paper reads this as evidence that a transformer-based token classifier can handle nested entities if overlaps are encoded into the label space, either merged into joint labels like "NP-Spatial+ENE-Spatial" or predicted independently by a multi-label sigmoid head. It further claims that one-example GPT prompting is a promising, though inconsistent, route when labeled data are scarce, especially for frequent and structurally clear entity types such as Latlong and Domain-mark.

Load-bearing premise

The ranking that carries the central claim is computed token-by-token, so a system can look good by labeling the right words even when it gets the span boundaries or nesting structure wrong; if the same models were scored by exact whole-span match, the ordering could change.

Editorial extensions

If this is right

  • A fine-tuned French transformer should be the first system to try for NER on comparable early modern French corpora, not only on GeoEDdA.
  • Nested or overlapping entities do not force a span-prediction architecture; composite-label and multi-label token classifications are competitive ways to encode them.
  • A single labeled example plus a structured-output prompt can bootstrap useful annotations for frequent entity types in low-resource settings.
  • Rare and ambiguous categories such as ENE-Misc and NP-Misc remain unsolved across every evaluated approach, so downstream applications should budget for human review on those labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the headline metric is token-level F1, a span-level exact-match evaluation would test whether the nesting advantage is real; the paper's own tables show near-zero ENE-Misc scores for spaCy Spancat, which such a metric could make even more decisive.
  • Editorial extension: since the joint-label CamemBERT head has higher precision and the multi-label head has higher recall, an ensemble of the two would plausibly beat either one on macro F1.
  • Editorial extension: the single-example GPT results point to a concrete bootstrap loop, where GPT proposes labels, a human corrects them, and the corrections fine-tune CamemBERT; the paper mentions this as an active-learning possibility but does not measure it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper benchmarks named entity recognition (NER) methods on GeoEDdA, a French 18th-century encyclopedic corpus with flat and nested entity annotations. The authors compare CRF, spaCy NER, spaCy Spancat, fine-tuned CamemBERT (joint-label and multi-label), and Flair, and they also explore few-shot prompting with proprietary GPT models. The central claims are that fine-tuned CamemBERT achieves the best overall performance, particularly on nested entities, and that few-shot GPT prompting is a promising alternative in low-resource settings. The manuscript includes a detailed per-class and aggregate evaluation, with models released publicly on HuggingFace and code on GitHub.

Significance. If the reported results are correct, the paper provides a useful and reproducible benchmark for a difficult historical French NER task, with public dataset, models, and code. The comparison of joint-label vs. multi-label token classification for nested entities, and the inclusion of span-based, sequence-labeling, and generative approaches in one study, is potentially valuable to the digital humanities and historical NLP communities. The paper ships concrete artifacts (GeoEDdA, fine-tuned model checkpoints, GitHub code) and makes falsifiable empirical predictions. However, the significance is substantially reduced by serious internal inconsistencies in the GPT evaluation tables and by the absence of any uncertainty quantification for the small test set.

major comments (4)
  1. [Section 4.5, Table 7] The GPT-4o precision/recall row in Table 7 is identical, class by class, to the fine-tuned spaCy NER row in Table 4 (e.g., NC-Spatial P=79.7/R=34.5, NP-Spatial P=93.3/R=50.3, Micro P=77.5/R=33.4, Macro P=66.9/R=28.6). Similarly, the GPT-4.1-mini row in Table 7 is identical to the spaCy Spancat row in Table 4. Because these are independent model families, exactly reproducing every value is not plausible; as written, the GPT-4o and GPT-4.1-mini few-shot results are not supported by the reported data. This is load-bearing because the few-shot contribution, highlighted in the abstract and in Section 6, rests on these numbers.
  2. [Table 8] The last column of Table 8, labeled 'GPT 4o-mini', contains values that largely match the span-level F1 scores computed from the o4-mini column of Table 7, despite Table 8 being titled 'Token-based F1-scores'. For example, NC-Spatial 62.8, NP-Spatial 85.1, and Relation 58.0 correspond to o4-mini's precision/recall. However, NP-Person is listed as 58.2, whereas the o4-mini P=82.7/R=75.4 from Table 7 gives F1≈78.9, and Macro avg is listed as 66.2 versus the computed 66.9. The table also omits GPT-4o, which Section 4.5 claims to evaluate. This mixing of granularities and model labels makes the summary comparison unreliable.
  3. [Sections 4.4, 5, 6, Table 8] The claim that CamemBERT achieves the highest overall performance is contradicted by the paper's own Table 8: the Fine-tuned Flair column has a higher Micro avg F1 (91.8) than the Fine-tuned CamemBERT column (91.2), and Flair is also higher for several classes such as NC-Person (77.1 vs. 71.6) and NP-Misc (77.1 vs. 72.7). The conclusion appears to rely on Macro avg or on selected entity types, but the manuscript does not state this. The headline 'state-of-the-art' and 'highest overall performance' claims need to be qualified or justified with a consistent evaluation criterion.
  4. [Section 4, Tables 4–8] All results are reported for a single run, and no variance, confidence intervals, or significance tests are given. The test set contains only 200 paragraphs, and several classes have very few test instances (e.g., ENE-Misc has 22). Because the central ranking differences are small—Flair and CamemBERT differ by 0.6 points in Micro F1—single-run scores cannot distinguish such differences. The ranking claims in Sections 5 and 6 should be tempered or supplemented with repeated runs and confidence intervals.
minor comments (6)
  1. [Table 1 caption] The caption contains a typo: 'entity accross' should be 'entities across'.
  2. [Table 8 header] The column header line is mangled and appears to concatenate column names: 'GPT 4o-minispaCy NER' and similar. The column structure should be clearly labeled, and each column should indicate the exact model configuration (e.g., joint-label vs. multi-label CamemBERT).
  3. [Section 4.5] The text states that GPT3.5 was evaluated, but Table 7 and Table 8 only report three models, and GPT3.5 is never shown; the paper should clarify what happened to that model or remove the mention.
  4. [Section 4.5] Model names are inconsistent: the text uses 'GPT-o4-mini' while Table 7 uses 'o4-mini' and Table 8 uses 'GPT 4o-mini'. This makes it difficult to know which model produced which numbers.
  5. [Section 3.2] The task formulation section describes joint-label and multi-label classification, but Table 8 reports only a single CamemBERT column. It should be clear whether the F1 scores come from the joint-label model, the multi-label model, or an average.
  6. [Section 4.2] Table 4 reports precision and recall but does not state whether these are token-level or span-level scores; given that Table 8 is explicitly 'Token-based F1-scores', the evaluation granularity should be defined consistently across all tables.

Circularity Check

1 steps flagged · score 6.0 of 10

GPT-4o few-shot scores are identical to the fine-tuned spaCy NER row, making part of the generative-model claim a relabeled baseline.

  1. other [Section 4.5, Table 7 (GPT-4o columns) vs. Section 4.2, Table 4 (spaCy NER columns)]
    "Table 4: 'NC-Spatial79.734.5', 'NP-Spatial93.350.3', 'Micro Avg77.533.4', 'Macro Avg66.928.6'. Table 7: '4o' row 'NC-Spatial79.734.5', 'NP-Spatial93.350.3', 'Micro Avg77.533.4', 'Macro Avg66.928.6'."

    Every precision/recall value reported for GPT-4o in Table 7 is numerically identical to the fine-tuned spaCy NER row in Table 4, including micro and macro averages and rare classes (e.g., ENE-Misc 2.9/1.2 in both). Thus the 'few-shot' GPT-4o result is the same measurement as a supervised baseline presented under a different model name; the paper's statement that GPT-4o 'attains the best precision on several tags' and the broader conclusion that generative models 'offer promising alternatives when labeled data are scarce' are, for GPT-4o, derived from spaCy NER numbers rather than from an independent generative run. That specific prediction therefore reduces by construction to a relabeled baseline.

full rationale

The core benchmark (CRF, spaCy, CamemBERT, Flair) is an empirical comparison on a fixed test split of GeoEDdA; no fitted constant defines the evaluation target, and the CamemBERT result does not reduce to any modelling input. The main circularity issue is the exact duplication of Table 4's spaCy NER precision/recall row as Table 7's GPT-4o row, including rare-class and aggregate values. As printed, the GPT-4o few-shot result is a relabeled supervised baseline, so the 'promising alternative' claim is partially supported by non-generative numbers. The o4-mini and GPT-4.1-mini rows appear numerically distinct and provide some independent evidence, which is why the score is 6 rather than higher. The dataset's construction via active learning with spaCy and several self-citations are noted but are not load-bearing circularity because the annotations were manually refined and all models share the same fixed test split.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical comparison; no free parameters or invented entities are introduced. The main hidden dependency is the evaluation protocol: token-level F1 and the GeoEDdA gold standard. Few-shot LLM results additionally depend on which proprietary models were selected by the authors.

assumptions (3)
  • domain assumption GeoEDdA's manual annotations are correct and serve as ground truth.
    Every F1 score is computed against these annotations; errors would bias all model comparisons.
  • domain assumption Token-level F1 captures the quality needed for the paper's conclusions.
    Table 8 compares all models on token-based F1; span-level exact match is never reported, yet the paper discusses nested span performance.
  • domain assumption The fixed 1800/200/200 split and single training run per model are representative.
    No repeated runs or confidence intervals are reported, so observed differences may reflect noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token and Span Classification for Entity Recognition in French Historical Encyclopedias." pith.science (2026). https://pith.science/paper/IH4RMJF6

@misc{pith2026250602872,
  author       = {Pith},
  title        = {Pith review of: Token and Span Classification for Entity Recognition in French Historical Encyclopedias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IH4RMJF6}},
  note         = {Machine review of arXiv:2506.02872}
}
read the original abstract

Named Entity Recognition (NER) in historical texts presents unique challenges due to non-standardized language, archaic orthography, and nested or overlapping entities. This study benchmarks a diverse set of NER approaches, ranging from classical Conditional Random Fields (CRFs) and spaCy-based models to transformer-based architectures such as CamemBERT and sequence-labeling models like Flair. Experiments are conducted on the GeoEDdA dataset, a richly annotated corpus derived from 18th-century French encyclopedias. We propose framing NER as both token-level and span-level classification to accommodate complex nested entity structures typical of historical documents. Additionally, we evaluate the emerging potential of few-shot prompting with generative language models for low-resource scenarios. Our results demonstrate that while transformer-based models achieve state-of-the-art performance, especially on nested entities, generative models offer promising alternatives when labeled data are scarce. The study highlights ongoing challenges in historical NER and suggests avenues for hybrid approaches combining symbolic and neural methods to better capture the intricacies of early modern French text.

Figures

Figures reproduced from arXiv: 2506.02872 by the authors.

Figure 1
Figure 1. Excerpt of annotations from the ILLESCAS entry. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. F1-scores comparison for OpenAI’s GPT models. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 10 canonical work pages

  1. [1]

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. 2024. https://arxiv.org/abs/2404.11018 Many-shot in-context learning . arXiv preprint arXiv:2404.11018

    arXiv 2024

  2. [2]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR : An easy-to-use framework for state-of-the-art NLP . In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 54--59

    work page 2019

  3. [3]

    Alice Brenon, Ludovic Moncla, and Katherine McDonough. 2022. Classifying encyclopedia articles: Comparing machine and deep learning methods and exploring their predictions. Data & Knowledge Engineering, 142:102098

    work page 2022

  4. [4]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . arXiv preprint arXiv:1810.04805

    arXiv 2018

  5. [5]

    Mauro Gaio and Ludovic Moncla. 2017. Extended named entity recognition using finite-state transducers: An application to place names. In The ninth international conference on advanced geographic information systems, applications, and services, Nice, France

    work page 2017

  6. [6]

    Carlos-Emiliano Gonz \'a lez-Gallardo, Emanuela Boros, Nancy Girdhar, Ahmed Hamdi, Jose G Moreno, and Antoine Doucet. 2023. Yes but.. can chatgpt identify entities in historical documents? In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 184--189. IEEE

    work page 2023

  7. [7]

    Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991

    arXiv 2015

  8. [8]

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64--77

    work page 2020

Show all 20 references
  1. [9]

    Mingchen Li and Rui Zhang. 2023. https://arxiv.org/abs/2307.00186 How far is language model from 100\ arXiv preprint arXiv:2307.00186

    2023 arXiv

  2. [10]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su \'a rez, Yoann Dupont, Laurent Romary, \'E ric Villemonte de La Clergerie, Djam \'e Seddah, and Beno \^i t Sagot. 2020. https://doi.org/10.18653/v1/2020.acl-main.645 C amem BERT : a tasty F rench language model . In Proceedi...

    2020 doi

  3. [11]

    Katherine McDonough, Ludovic Moncla, and Matje Van de Camp. 2019. Named entity recognition goes to old regime france: geographic text analysis for early modern french corpora. International Journal of Geographical Information Science, 33(12):2498--2522

    2019

  4. [12]

    Ludovic Moncla and Mauro Gaio. 2023. Perdido: Python library for geoparsing and geocoding french texts. In First International Workshop on Geographic Information Extraction from Texts (GeoExT)

    2023

  5. [13]

    Ludovic Moncla, Denis Vigier, and Katherine Mcdonough. 2024. GeoEDdA : A Gold Standard Dataset for Geo-semantic Annotation of Diderot & d'Alembert's Encyclop \'e die . In Second International Workshop on Geographic Information Extraction from Texts (GeoExT), Glasgow, Scotland

    2024

  6. [14]

    Abdelkrim Tafer and Mauro Gaio. 2025. Extracting spatial entities involved in the description of a movement action using deep learning methods: A comparative study of three models. In Third International Workshop on Geographic Information Extraction from Texts (GeoExT)

    2025

  7. [15]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  8. [16]

    Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. https://arxiv.org/abs/2310.10035 Empirical study of zero-shot ner with chatgpt . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7935--7956, Singapore

    2023 arXiv

  9. [17]

    Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. https://doi.org/10.18653/v1/2024.naacl-long.300 GL i NER : Generalist model for named entity recognition using bidirectional transformer . In Proceedings of the 2024 Conference of the North American Chapt...

    2024 doi

  10. [18]

    H \'e di Zeghidi and Ludovic Moncla. 2024. Evaluating named entity recognition using few-shot prompting with large language models. arXiv preprint arXiv:2408.15796

    2024 arXiv

  11. [19]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  12. [20]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.