Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Two Spelling Normalization Approaches Based on Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Character-level statistical machine translation still beats fine-tuned large language models on historical spelling.

desk verdict Honest but narrow empirical comparison: fine-tuned LLMs don't beat character-based SMT, yet the headline claim overreaches because one corpus is a statistical tie and another is tiny. read the letter →

arxiv 2506.23288 v1 pith:J4PQC73S submitted 2025-06-29 cs.CL

Miguel Domingo , Francisco Casacuberta This is my paper · ORCID
classification cs.CL
keywords spellingnormalizationhistoricaldocumentslargelanguagemodelsmT5BARTcharacter-basedstatisticalmachinetranslationlow-resourceNLPdigitalhumanities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large pretrained multilingual language models can modernize the spelling of historical documents as reliably as dedicated machine-translation systems, in a setting where parallel normalized data is scarce. It proposes fine-tuning mT5, a model pretrained without supervised translation, and mBART, a model pretrained for machine translation, treating normalization as translation from historical to modern spelling. The central claim is that character-based statistical machine translation (CBSMT) remains the most suitable technology: it achieves the best CER, TER, and BLEU scores on two 17th-century Spanish corpora and one 18th-century Slovene corpus. A sympathetic reader would care because the task is widespread in digital humanities, and the result tells practitioners where to spend scarce training data and compute.

What carries the argument

The argument is carried by the machine-translation formalization of spelling normalization, $\hat{y} = \arg\max_y \Pr(y \mid x)$, in which the source string is the historical spelling and the target is the modern spelling. All four systems solve this equation at different granularities: character-based statistical machine translation splits words into characters and combines phrase-based alignment, reordering, and a 5-gram language model in a log-linear model; character-based neural machine translation applies a Transformer to characters; mT5 and mBART are multilingual encoder-decoder language models fine-tuned on parallel sentences. The decisive mechanism is the contrast between pretraining objectives: mBART was pretrained for translation, which the paper argues lets it adapt to the tiny Bohorič corpus, whereas mT5, pretrained without supervised translation, collapses on that corpus. The same formalism makes CER, TER, and BLEU plus significance tests the common yardstick across all systems.

What would settle it

A replicated experiment on a fourth historical corpus, with per-model hyperparameter tuning, in which fine-tuned mBART or mT5 achieves lower CER than CBSMT on the same test splits would overturn the paper's ranking.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is empirical and comparative: two ways of pressing multilingual pretrained language models into service for historical spelling normalization are workable, but neither displaces an older character-level statistical machine translation approach. Fine-tuning mT5 and mBART on historical-source to modern-target parallel sentences improves over the unnormalized baseline on the two Spanish corpora, with both language models tying CBSMT on Quijote and mBART clearly beating both character-based neural machine translation and mT5 on the small Bohorič corpus. CBSMT nevertheless posts the best CER, TER, and BLEU in nearly every configuration. The paper reads this as evidence that, for this task, the most suitable technology remains character-based statistical machine translation, with mBART the stronger language model because it was pretrained for machine translation.

Load-bearing premise

The conclusion that SMT is the most suitable technology rests on the assumption that three historical corpora, two 17th-century Spanish texts and one small 18th-century Slovene collection, represent the general spelling-normalization task closely enough.

Editorial extensions

If this is right

  • For practical normalization projects with limited parallel data, character-based statistical machine translation should remain the default baseline rather than an outdated alternative.
  • When a language model is preferred, mBART's translation-oriented pretraining makes it the safer choice over mT5 in low-resource historical settings.
  • The translation framing lets spelling normalization inherit machine-translation tooling, metrics, and significance testing directly.
  • On sufficiently large corpora the language-model approaches become statistically indistinguishable from CBSMT, so the gap is data-dependent rather than fixed.
  • Neither language model consistently beats a character-level Transformer trained from scratch, so pretraining alone is not decisive for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: because the paper reports no hyperparameter search for the language models, tuning learning rate, batch size, and fine-tuning epochs could narrow or reverse the gap; the ranking should be rechecked before treating SMT's edge as intrinsic.
  • The small Bohorič corpus and the absence of non-Spanish, non-Slovene data leave open whether the result generalizes to morphologically richer or more orthographically erratic languages; Middle English or Early Modern German corpora would be direct tests.
  • The success of mBART on low-resource Bohorič suggests a quantitative study of performance versus corpus size could separate the contribution of translation-oriented pretraining from the contribution of data scale.
  • Combining historical periods or using unnormalized old documents for general fine-tuning, which the authors propose as future work, is where language models might plausibly overtake SMT; the paper's data do not settle that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes two historical spelling normalization approaches based on fine-tuned large language models: mT5, fine-tuned from a multilingual text-to-text model, and mBART, fine-tuned from a multilingual machine translation model. Both are compared against character-based statistical machine translation (CBSMT) and character-based neural machine translation (CBNMT) on three corpora: two 17th-century Spanish texts (Entremeses y Comedias and Quijote) and one 18th-century Slovene text (Bohorič). The systems are evaluated with CER, TER, and BLEU, with statistical significance assessed by approximate randomization tests with 10,000 iterations. The results show CBSMT achieving the best overall scores, with all systems statistically tied on Quijote and with the LLMs performing poorly on the small Bohorič corpus. The paper concludes that CBSMT remains the most suitable technology for spelling normalization.

Significance. If the result holds, the paper provides a useful empirical data point: fine-tuned multilingual LLMs do not automatically outperform classical character-based SMT for historical spelling normalization, particularly in low-resource settings. The paper has several strengths: it uses standard evaluation metrics including sacreBLEU, performs significance testing, reports a reasonably detailed baseline setup for CBSMT and CBNMT, and is honest in reporting that its own proposed methods do not beat the SMT baseline. The qualitative examples in Section 5.1 also help illustrate error patterns. The main limitation is external validity: the strong conclusion about SMT superiority rests on a small number of corpora and on an asymmetric hyperparameter setup for the LLMs, so the generality of the ranking is not yet established.

major comments (3)
  1. [Section 6 and Table 2] The conclusion that CBSMT is 'still the most suitable technology for this task' is stronger than the evidence supports. On Quijote, Table 2 marks all systems with a dagger, meaning there is no statistically significant difference among any approaches on any metric; mT5 and mBART numerically match CBSMT on BLEU (95.5 vs 94.4) and achieve a better CER (1.7 vs 2.5). On Entremeses, mT5 and CBSMT are statistically tied on BLEU. The clearest CBSMT advantages are on CER/TER for Entremeses and on Bohorič, where the training set contains only 3.6K sentences (Table 1), a regime the authors themselves say is too small for the LLMs. The headline claim should be qualified to reflect that CBSMT is at least as good as fine-tuned LLMs on the larger corpora and more robust in a very low-resource setting, rather than stated as a general conclusion about the most suitable technology.
  2. [Section 4.3] The experimental setup for fine-tuning the LLMs is under-specified and asymmetric with the baselines. For CBSMT and CBNMT the paper provides many hyperparameters (e.g., 5-gram LM with Kneser-Ney smoothing, MERT, 6 layers, 512 dimensions, 8 heads, Adam with beta2=0.998, learning rate 2, Noam decay with 8000 warmup steps, label smoothing 0.1, beam size 6, 32k BPE merges). For mT5 and mBART it only states that the HuggingFace library was used and that mt5-base and mbart-large-50 were selected as starting points. No learning rate, batch size, number of epochs, warmup, early stopping, or hyperparameter search is reported. This makes the comparison potentially unfair to the LLMs and prevents reproduction. The authors should add the full fine-tuning configuration and preferably perform a small validation-based search or at least report early-stopping behavior before drawing conclusions about the relative ranking of systems.
  3. [Section 5] The comparison between the two LLM approaches is confounded by model size and capacity. The paper attributes mBART's better performance on Bohorič to its pretraining objective, stating that 'the former seems to be the most suitable for this task due to it having been pre-trained for MT.' However, mbart-large-50 is a larger model than mt5-base, and the two models also differ in pretraining data and architectural details. The observed difference could be due to model capacity or language coverage rather than the pretraining objective. A cleaner comparison would require models of comparable size or an explicit discussion of this confound.
minor comments (6)
  1. [Section 4.3] The name 'HugginFace' should be corrected to 'HuggingFace'.
  2. [Section 5] In the sentence 'Overall, the CBSMT approached yielded the best performance', 'approached' should be 'approach'.
  3. [Figures 2–4] The captions mention a symbol denoting a character removed during normalization, but this symbol does not appear in the rendered text of the figures in the manuscript; the legend should be fixed.
  4. [Table 2] The caption explains the dagger and double-dagger notation, but it would be clearer to state explicitly that unmarked differences are statistically significant at p<0.05 according to the approximate randomization test.
  5. [Section 4.2] Please clarify how the normalized versions of the corpora were produced or validated, and whether the same normalization guidelines were used for the two Spanish corpora, since this affects the comparability of the test sets.
  6. [Section 4.3] For both LLM fine-tuning runs, please specify whether the development split was used for early stopping and report the number of training steps or epochs for each corpus.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper evaluates held-out test sets with standard metrics and significance tests, and its central conclusion is not forced by any fitted parameter, self-citation chain, or definitional identity.

full rationale

This paper contains no circular derivation. The two proposed LLM-based normalization approaches are trained on the training split of each corpus and evaluated on separate test splits using CER, TER, and BLEU with approximate randomization significance tests, as described in Sections 4.1-4.3 and Table 2. No parameter is fitted to the test data and then renamed as a prediction, and no result is asserted to follow from a definition that already contains the conclusion. The central claim that CBSMT is still the most suitable technology is based on an external comparison of four independent systems on three corpora, and it is actually unfavorable to the authors' own proposed methods, which removes any incentive for a self-serving fit. The self-citations (Domingo and Casacuberta, 2018, 2019, 2021) appear only as contextual references to prior comparable findings, not as load-bearing justification: the sentence 'These results are coherent with other results reported in the literature (Tang et al., 2018; Hämäläinen et al., 2018; Domingo and Casacuberta, 2019)' uses the prior work as corroboration, while the experimental table itself is the evidence. The paper also explicitly identifies its own limitations, such as the small size of the Bohorič corpus and the data dependence of the LLM approaches, which are external-validity or robustness concerns rather than circularity. The absence of hyperparameter search for the LLMs weakens the generality of the ranking but does not make the derivation circular, because the comparison is still a fair evaluation of the configurations actually used. Overall, the derivation chain is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on the standard MT formulation and the representativeness of the chosen corpora, plus the assumption that the LLM fine-tuning is fair relative to the tuned SMT baseline.

assumptions (3)
  • domain assumption Spelling normalization can be modeled as a machine translation problem (Eq. 1).
    The paper frames normalization as finding the most likely normalized form y given original x (Eq. 1), an assumption standard in this literature but not strictly true for normalization.
  • domain assumption The three selected corpora are representative of historical spelling variation.
    Only three corpora (two Spanish, one Slovene) are used; conclusions about the most suitable technology are drawn from them.
  • domain assumption The fine-tuning setups for mT5 and mBART are a fair comparison against the strongly optimized CBSMT system.
    No hyperparameter search or ensembling was reported for the LLMs, while CBSMT used MERT tuning; this could disadvantage the LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two Spelling Normalization Approaches Based on Large Language Models." pith.science (2026). https://pith.science/paper/J4PQC73S

@misc{pith2026250623288,
  author       = {Pith},
  title        = {Pith review of: Two Spelling Normalization Approaches Based on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4PQC73S}},
  note         = {Machine review of arXiv:2506.23288}
}
read the original abstract

The absence of standardized spelling conventions and the organic evolution of human language present an inherent linguistic challenge within historical documents, a longstanding concern for scholars in the humanities. Addressing this issue, spelling normalization endeavors to align a document's orthography with contemporary standards. In this study, we propose two new approaches based on large language models: one of which has been trained without a supervised training, and a second one which has been trained for machine translation. Our evaluation spans multiple datasets encompassing diverse languages and historical periods, leading us to the conclusion that while both of them yielded encouraging results, statistical machine translation still seems to be the most suitable technology for this task.

Figures

Figures reproduced from arXiv: 2506.23288 by the authors.

Figure 1
Figure 1. Example of adapting a document’s spelling to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of modernizing a sentence from En￾tremeses y Comedias with all the different approaches. denotes a character that has been removed as part of its normalization. Unnormalized characters that should have been normalized and wrongly normalized charac￾ters are denoted in red. Characters which were success￾fully normalized are denoted in teal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example of modernizing a sentence from El Quijote with all the different approaches. denotes a character that has been removed as part of its normalization. Unnormalized characters that should have been normalized and wrongly normalized characters are denoted in red. Characters which were successfully normalized are denoted in teal. d (which is the result of changing the tense of the verb from imperative to past sim… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of modernizing a sentence from Bohoric˘ with all the different approaches. denotes a character that has been removed as part of its normalization. Unnormalized characters that should have been normalized and wrongly normalized characters are denoted in red. Cha…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages

  1. [1]

    and Rayson, P

    Baron, A. and Rayson, P. (2008). VARD2 : A tool for dealing with spelling variation in historical corpora. Postgraduate conference in corpus linguistics

    work page 2008

  2. [2]

    Bollmann, M. (2018). Normalization of Historical Texts with Neural Network Models . PhD thesis, Sprachwissenschaftliches Institut, Ruhr-Universit \"a t

    work page 2018

  3. [3]

    and S gaard, A

    Bollmann, M. and S gaard, A. (2016). Improving historical spelling normalization with bi-directional lstms and multi-task learning. In Proceedings of the International Conference on the Computational Linguistics , pages 131--139

    work page 2016

  4. [4]

    F., Pietra, V

    Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics , 19(2):263--311

    work page 1993

  5. [5]

    and Casacuberta, F

    Domingo, M. and Casacuberta, F. (2018). Spelling normalization of historical documents by using a machine translation approach. In Proceedings of the Annual Conference of the European Association for Machine Translation , pages 129--137

    work page 2018

  6. [6]

    and Casacuberta, F

    Domingo, M. and Casacuberta, F. (2019). Enriching character-based neural machine translation with modern documents for achieving an orthography consistency in historical documents. In Proceedings of the International Workshop on Pattern Recognition for Cultural Heritage , pages 59--69

    work page 2019

  7. [7]

    and Casacuberta, F

    Domingo, M. and Casacuberta, F. (2021). A comparison of character-based neural machine translations techniques applied to spelling normalization. In Proceedings of the International Conference on Pattern Recognition. International Workshop on Pattern Recognition for Cultural Heritage , pages 326--338

    work page 2021

  8. [8]

    Jehle, F

    F. Jehle, F. (2001). Works of Miguel de Cervantes in Old- and Modern-spelling . Indiana University Purdue University Fort Wayne

    work page 2001

Show all 40 references
  1. [9]

    Gage, P. (1994). A new algorithm for data compression. The C Users Journal , 12(2):23–38

    1994

  2. [10]

    a m \"a l \

    H \"a m \"a l \"a inen, M., S \"a ily, T., Rueter, J., Tiedemann, J., and M \"a kel \"a , E. (2018). Normalizing early english letters to present-day english spelling. In Proceedings of the Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanitie...

    2018

  3. [11]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9(8):1735--1780

    1997

  4. [12]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  5. [13]

    Klein , G., Kim , Y., Deng , Y., Senellart , J., and Rush , A. M. (2017). OpenNMT: Open-Source Toolkit for Neural Machine Translation . In Proceedings of the Association for Computational Linguistics: System Demonstration , pages 67--72

    2017

  6. [14]

    Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the Annual M...

    2007

  7. [15]

    J., and Marcu, D

    Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology , pages 48--54

    2003

  8. [16]

    Korchagina, N. (2017). Normalizing medieval german texts: from rules to deep learning. In Proceedings of the Nordic Conference on Computational Linguistics Workshop on Processing Historical Language , pages 12--17

    2017

  9. [17]

    Laing, M. (1993). The linguistic analysis of medieval vernacular texts: Two projects at edinburgh'. In Corpora across the Centuries: Proceedings of the First International Colloquium on English Diachronic Corpora, edited by M. Rissanen, M. Kytd, and S. Wright. St Catharine’s C...

    1993

  10. [18]

    Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. (2020). Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210

    2020 arXiv

  11. [19]

    Ljube s i \'c , N., Zupan, K., Fi s er, D., and Erjavec, T. (2016). Dataset of normalised slovene text KonvNormSl 1.0. Slovenian language resource repository CLARIN . SI . http://hdl.handle.net/11356/1068

    2016

  12. [20]

    Ljube s ic, N., Zupan, K., Fi s er, D., and Erjavec, T. (2016). Normalising slovene data: historical texts vs. user-generated content. In Proceedings of the Conference on Natural Language Processing , pages 146--155

    2016

  13. [21]

    Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 160--167

    2003

  14. [22]

    Och, F. J. and Ney, H. (2002). Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 295--302

    2002

  15. [23]

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU : a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 311--318

    2002

  16. [24]

    Porta, J., Sancho, J.-L., and G \'o mez, J. (2013). Edit transducers for spelling variation in old spanish. In Proceedings of the Workshop on Computational Historical Linguistics , pages 70--79

    2013

  17. [25]

    Post, M. (2018). A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation , pages 186--191

    2018

  18. [26]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1--67

    2020

  19. [27]

    and Maxwell, J

    Riezler, S. and Maxwell, J. T. (2005). On some pitfalls in automatic evaluation and significance testing for mt. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 57--64

    2005

  20. [28]

    and Monro, S

    Robbins, H. and Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics , pages 400--407

    1951

  21. [29]

    E., Hinton, G

    Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature , 323(6088):533--536

    1986

  22. [30]

    and Erjavec, T

    Scherrer, Y. and Erjavec, T. (2013). Modernizing historical slovene words with character-based smt. In Proceedings of the Workshop on Balto-Slavic Natural Language Processing , pages 58--62

    2013

  23. [31]

    Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas , pages 223--231

    2006

  24. [32]

    Stolcke, A. (2002). SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing , pages 257--286

    2002

  25. [33]

    Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Proceedings of the Advances in Neural Information Processing Systems , volume 27, pages 3104--3112

    2014

  26. [34]

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1--9

    2015

  27. [35]

    Tang, G., Cap, F., Pettersson, E., and Nivre, J. (2018). An evaluation of neural machine translation models on historical spelling normalization. In Proceedings of the International Conference on Computational Linguistics , pages 1320--1331

    2018

  28. [36]

    Tang, Y., Tran, C., Li, X., Chen, P.-J., Goyal, N., Chaudhary, V., Gu, J., and Fan, A. (2020). Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401

    2020 arXiv

  29. [37]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998--6008

    2017

  30. [38]

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  31. [39]

    Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021). mT5 : A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934

    2021 arXiv

  32. [40]

    J., and Ney, H

    Zens, R., Och, F. J., and Ney, H. (2002). Phrase-based statistical machine translation. In Proceedings of the Annual German Conference on Advances in Artificial Intelligence , volume 2479, pages 18--32

    2002

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.