Pith. sign in

REVIEW 4 major objections 5 minor 9 references

Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On a manually annotated sample of Russian cultural-event announcements, GPT-4.1 identifies person names at F1 = 0.94 with either a plain or JSON prompt, the best score among all models tested.

desk verdict Useful new dataset, but the headline F1 numbers are internally inconsistent and need correction before the evaluation can be trusted. read the letter →

arxiv 2506.02589 v1 pith:IX7UO4GW submitted 2025-06-03 cs.CL cs.AIcs.IR

Maria Levchenko This is my paper
classification cs.CLcs.AIcs.IR
keywords NamedEntityRecognitionRussianNLPculturalheritagetextsSPbLitGuidedatasetGPT-4.1promptengineeringPERSONextractionbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which off-the-shelf NER model can reliably extract real person names from Russian-language announcements of cultural events, where inflection, mixed-script names, pseudonyms, and names embedded in titles make the task harder than standard news NER. To answer it, the author manually annotated a 1,000-record sample of the SPbLitGuide event-announcement corpus with 5,611 PERSON labels and evaluated a set of transformer and LLM systems, including GPT-3.5, GPT-4, GPT-4o, and GPT-4.1. The central result is that GPT-4.1-2025-04-14 reaches F1 = 0.94 for PERSON extraction with both a simple prompt and a JSON-output prompt, matching or beating every other model, while GPT-4o with JSON prompting reaches F1 = 0.93 and the best non-LLM transformer reaches F1 = 0.84. If the benchmark is sound, the newest GPT-4.1 family removes the need for structured-output prompt engineering on this kind of Russian cultural text while improving accuracy.

What carries the argument

The load-bearing object is the SPbLitGuide gold-standard sample: 1,000 event announcements randomly selected from 15,012 records spanning 1999–2019, with 5,611 manually annotated PERSON mentions under explicit exclusion rules that forbid labeling names inside organization names, work titles, addresses, or references to fictional characters. The evaluation machinery then normalizes each model's raw output into the same span-label format as the annotations and computes precision, recall, and F1 per model. For the LLMs, the study varies one additional mechanism: the prompt requests either plain text output or structured JSON output, testing whether output-mode constraints change extraction quality.

What would settle it

Have two independent annotators re-label the same 1,000 records under the same rules, keep only spans where both annotators agree, and recompute each model's F1 against that adjudicated standard; if GPT-4.1 no longer reaches 0.94 or no longer beats GPT-4o, the central ranking claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is a measured ranking on the SPbLitGuide sample: among the evaluated systems, GPT-4.1-2025-04-14 achieves the best F1 of 0.94 for PERSON entities, and it does so in both the simple API-prompt mode (precision 0.94, recall 0.93) and the JSON-output mode (precision 0.93, recall 0.95). This is presented as a shift from the earlier result in the same study, where GPT-4o-2024-05-13 needed JSON prompting to reach F1 = 0.93 (precision 0.96, recall 0.90) versus F1 = 0.91 with a simple prompt. The paper also reports that GPT-4-turbo has the highest precision at 0.99, and that the strongest non-LLM baseline, RoBERTa Large NER Russian, reaches F1 = 0.84. The author interprets the convergence of simple and structured prompts as evidence that model improvements, rather than prompt engineering, drove the gain.

Load-bearing premise

The load-bearing premise is that the single annotator's 5,611 PERSON labels in the 1,000-record sample correctly apply the study's exclusion rules; if the gold standard mislabels or inconsistently handles names in titles, organizations, addresses, or fictional references, every reported precision, recall, and F1 value—and the model ranking—changes.

Editorial extensions

If this is right

  • For Russian cultural-news texts, teams can deploy GPT-4.1-2025-04-14 with a plain prompt and expect the same F1 as with JSON-structured output, cutting token overhead.
  • Applications that cannot tolerate false positives still have a precision-first option in GPT-4-turbo at 0.99 precision, at the cost of lower recall.
  • Budget- or CPU-constrained projects can use the Russian RoBERTa configuration or a CPU-oriented Russian pipeline as competitive alternatives to API-based LLMs.
  • The reported 6% residual error suggests human-in-the-loop review remains necessary if downstream literary-network analysis requires near-complete name coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to re-run the same protocol on a second independently annotated sample; without inter-annotator agreement, the 0.94 F1 may partly reflect how predictable one annotator's rule application is, rather than model quality.
  • The benchmark's exclusion of fictional characters and names inside titles means the practical NER ceiling depends on downstream goals: a system optimized for this gold standard may over-suppress names that a literary-network analysis actually wants.
  • If the convergence of simple and JSON prompting replicates on other morphologically rich languages, prompt-engineering effort in production NER pipelines should shift toward entity linking and normalization, where the paper itself identifies the next bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a comparative evaluation of named entity recognition models for extracting PERSON entities from a new Russian cultural-news dataset (SPbLitGuide). It compares several transformer-based models (DeepPavlov, RoBERTa, SpaCy) with GPT-3.5, GPT-4, GPT-4o, and, in a follow-up, GPT-4.1, on 1,000 manually annotated records containing 5,611 person labels. The headline claim is that GPT-4.1 achieves F1=0.94 with both simple and JSON-structured prompts, slightly surpassing GPT-4o's JSON-prompt F1=0.93, and that this removes the need for structured prompting. The paper also reports that GPT-4 has the highest precision (0.99) and that traditional models remain competitive, and it derives practical recommendations from the precision/recall trade-offs.

Significance. If the results can be verified, the paper would make a useful contribution: an original, publicly released benchmark for Russian cultural-news NER, a breadth of model comparison in an under-resourced domain, and the code and data are made available, which aligns with reproducibility recommendations. The observed progression from GPT-3.5 to GPT-4.1 is interesting. However, the internal inconsistencies in the metric tables and the under-documented GPT-4.1 evaluation currently prevent the findings from being used as reliable evidence. The significance of the claimed 'paradigm shift' (Section 6.1) cannot be assessed until the data are made consistent and auditable.

major comments (4)
  1. [§4.1, Table 5, §5.3, Table 7, §6.1] The GPT-4.1-2025-04-14 model, which is the basis of the paper's headline result (F1=0.94), is absent from the experimental setup and model inventory. Section 4.1 and Table 5 list only gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13; GPT-4.1 appears for the first time in the results (Section 5.3, Table 7). No API parameters, prompt text, run date, or output-parsing procedure are given for this follow-up. Because Sections 5.5 and 6.1 draw their main conclusions from this run, the central quantitative claim is not auditable from the manuscript as submitted.
  2. [Table 6, Table 7, §5.5, §6.1] The paper reports conflicting scores for the same model and prompt condition. Table 6 gives gpt-4o-2024-05-13 (simple API) P=0.96, R=0.86, F1=0.91, while Table 7 reports P=0.85, R=0.80, F1=0.82 for the same configuration, an F1 difference of 0.09. In addition, Section 5.5 cites GPT-4 recall as 0.74 while Table 6 reports 0.69; and Section 6.1 attributes an F1 of 0.78 to GPT-3.5, whereas Table 6 gives 0.81. These inconsistencies make the model rankings and the claimed progression from GPT-3.5 to GPT-4.1 impossible to verify as published.
  3. [§3.2] The description of the gold-standard sample is not reproducible. The text says the 1,000 records were 'randomly selected' but also 'based on event date and event description length,' without specifying the selection mechanism or strata. More importantly, no inter-annotator agreement is reported for the manual annotation, even though the paper itself emphasizes the difficulty of the exclusion rules (names in titles, addresses, fictional characters). Since every precision/recall/F1 number depends on this gold standard, the lack of a reliability measure is a load-bearing gap.
  4. [§5.1, §5.3] No uncertainty quantification or statistical significance testing is provided. The paper interprets small differences in F1 (e.g., 0.01–0.02 between GPT-4.1 simple and JSON; 0.02 between GPT-4o JSON and simple) as meaningful, but without confidence intervals, error bars, or paired tests (for instance, McNemar's test on entity spans), these differences could be noise. Given the inconsistencies already noted, the significance of these observed gaps cannot be evaluated.
minor comments (5)
  1. [§2.1, §5.5, footnote] The manuscript contains several encoding or transcription artifacts (e.g., 'paradigm shiffi,' 'offien,' the garbled author footnote beginning 'envel⌢pe-⌢pen...'), which should be cleaned before publication.
  2. [§3.2] The phrase 'randomly selected' and 'based on event date and event description length' are contradictory; please describe the actual sampling procedure (e.g., stratified random sampling) explicitly.
  3. [§4.1, Table 5] The model inventory should be updated to include GPT-4.1-2025-04-14, together with the version identifiers and API settings used for all models.
  4. [§5.3] The sentence 'one year affier our initial experiments' contains a typo ('affier' for 'after').
  5. [§6.1] The claim of a '21% absolute improvement' from GPT-3.5 to GPT-4.1 should be recalculated with consistent F1 values; based on Table 6, the improvement would be 0.13 (0.94 - 0.81), not 0.21.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark against an externally available gold-standard dataset, and the only self-citation is a data release that is not used as a load-bearing premise.

full rationale

This paper reports a comparative NER evaluation against a manually annotated gold standard derived from the SPbLitGuide dataset. The claimed results—GPT-4o F1=0.93 and GPT-4.1 F1=0.94—are measurements of model outputs against externally defined labels, not quantities entailed by the model definitions, by the evaluation code, or by the annotation procedure. No parameter is fitted to the test sample, no 'prediction' is constructed from the target metric, and no uniqueness theorem or model choice is justified solely from the author's prior work. The only self-citation is [6], the Zenodo release of the dataset itself; this is a data resource, not an internal premise, and it is externally downloadable and checkable. The paper's internal inconsistencies (for example, gpt-4o-2024-05-13 simple API appears with F1=0.91 in Table 6 and F1=0.82 in Table 7, and GPT-4 recall appears as 0.74 in Section 5.5 but 0.69 in Table 6) undermine auditability, but these are reproducibility and correctness concerns, not circular reasoning. The limitation section also states that the evaluation used a single dataset and no task-specific fine-tuning; those are scope limitations, not circular steps. The derivation chain is self-contained: precision, recall, and F1 are defined from true positive, false positive, and false negative counts, and the paper states that data and model outputs are shared publicly. No significant circularity was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The evaluation depends on several domain assumptions: the gold-standard annotations are correct and consistent; the 1,000-record sample represents the full dataset; the output-adjustment procedure is unbiased; and single API runs are stable. There are no fitted parameters or invented entities. The contradictions in the results tables weaken the support for the central claim.

assumptions (4)
  • domain assumption The gold-standard manual annotations of 1,000 records are correct and consistently apply the exclusion rules (e.g., excluding names in titles, organizations, addresses, and fictional characters).
    Introduced in Section 3.2; if the annotations are biased, all precision/recall/F1 values are distorted. No inter-annotator agreement is reported.
  • domain assumption The 1,000-record sample is representative of the full 15,012-record dataset, so measured performance generalizes to the whole collection.
    Section 3.2 describes the sample as randomly selected but also 'based on event date and event description length', which is ambiguous and may introduce selection bias.
  • domain assumption The procedure for converting model outputs to the gold-standard span format is unbiased and does not favor any particular model.
    Section 4.2, step 3, mentions adjusting outputs but does not specify the matching algorithm (exact span, token-level, partial overlap). If matching favors LLM output styles, the comparison is skewed.
  • domain assumption API model outputs are deterministic, so a single run per model suffices.
    Section 4.2 implies one run per configuration; no sampling temperature or repeated runs are mentioned. If variance is high, the reported differences may not be meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM." pith.science (2026). https://pith.science/paper/IX7UO4GW

@misc{pith2026250602589,
  author       = {Pith},
  title        = {Pith review of: Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IX7UO4GW}},
  note         = {Machine review of arXiv:2506.02589}
}
read the original abstract

This paper addresses the challenge of Named Entity Recognition (NER) for person names within the specialized domain of Russian news texts concerning cultural events. The study utilizes the unique SPbLitGuide dataset, a collection of event announcements from Saint Petersburg spanning 1999 to 2019. A comparative evaluation of diverse NER models is presented, encompassing established transformer-based architectures such as DeepPavlov, RoBERTa, and SpaCy, alongside recent Large Language Models (LLMs) including GPT-3.5, GPT-4, and GPT-4o. Key findings highlight the superior performance of GPT-4o when provided with specific prompting for JSON output, achieving an F1 score of 0.93. Furthermore, GPT-4 demonstrated the highest precision at 0.99. The research contributes to a deeper understanding of current NER model capabilities and limitations when applied to morphologically rich languages like Russian within the cultural heritage domain, offering insights for researchers and practitioners. Follow-up evaluation with GPT-4.1 (April 2025) achieves F1=0.94 for both simple and structured prompts, demonstrating rapid progress across model families and simplified deployment requirements.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

  1. [1]

    Biderman, S., et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Lan- guage Models. arXiv preprint arXiv:2405.14782

    arXiv 2024

  2. [2]

    L., Cabrera-Diego, L

    Boros, E., Pontes, E. L., Cabrera-Diego, L. A., Hamdi, A., Moreno, J. G., Sid `ere, N., & Doucet, A. (2020, July 17). Robust Named Entity Recognition and Linking on Historical Multilingual Documents. Conference and Labs of the Evaluation Forum (CLEF 2020). https: //ceur-ws.org/Vol-2696/paper_171.pdf

    work page 2020

  3. [3]

    Ehrmann, M., Hamdi, A., Linhares Pontes, E., Romanello, M., & Doucet, A. (2023). Named Entity Recognition and Classication in Historical Documents: A Survey. ACM Comput. Surv., 56(2), Article 27. https://doi.org/10.1145/3604931

    doi:10.1145/3604931 2023

  4. [4]

    E., Boros, E., Girdhar, N., Hamdi, A., Moreno, J

    Gonz´alez-Gallardo, C. E., Boros, E., Girdhar, N., Hamdi, A., Moreno, J. G., & Doucet, A. (2023, June). Yes but.. Can ChatGPT identify entities in historical documents?. In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) (pp. 184-189). IEEE. https://arxiv.org/pdf/2303. 17322

    work page 2023

  5. [5]

    Hu, Y., Chen, Q., Du, J., Peng, X., Kuttichi Keloth, V., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., Roberts, K., & Xu, H. (2024). Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineering. arXiv preprint arXiv:2303.16416. https: //arxiv.org/pdf/2303.16416

    arXiv 2024

  6. [6]

    Levchenko, M. (2024). Literary Events in Saint Petersburg (1999-2019) from SPbLitGuide Newsletters (1.0). Zenodo. https://doi.org/10.5281/zenodo.13753154

    work page doi:10.5281/zenodo.13753154 2024

  7. [7]

    Naguiba, M., Tannier, X., & N ´ev´eola, A. (2024). Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting. arXiv preprint arXiv:2402.12801. https://arxiv.org/pdf/2402.12801

    arXiv 2024

  8. [8]

    Peng, L., Wang, Z., Yao, F., Wang, Z., & Shang, J. (2024). MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks. arXiv preprint arXiv:2404.00457. https://arxiv.org/pdf/2404.00457

    arXiv 2024

Show all 9 references
  1. [9]

    Todorov, K., & Colavizza, G. (2020). Transfer Learning for Historical Corpora: An Assessment on Post-OCR Correction and Named Entity Recognition. Workshop on Computational Humanities Research. https://ceur-ws.org/Vol-2723/long32.pdf

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.