REVIEW 2 major objections 6 minor 33 references
OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches
T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Prompt-based LLM post-correction works for historical English but not for historical Finnish with current open-weight models.
desk verdict Solid, honest empirical benchmark: open-weight LLMs help on historical English OCR but fail on Finnish; the 38.7% English gain comes with a filtering caveat that should be made plainer. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a zero-shot prompting pipeline in which the model is asked to correct a segment of OCR text, and the raw output is then post-processed by local sequence alignment against the original OCR input: leading and trailing text that does not align well—such as 'Here is the corrected text:' and appended explanations—is trimmed away. Performance is measured as relative character error rate reduction, $\text{CER\%} = (\text{CER}_{\text{orig}} - \text{CER}_{\text{post}})/\text{CER}_{\text{orig}} \times 100$, clipped at $-100\%$ and weighted by OCR character count, with WER as a secondary metric. Before scoring, the text is normalized with Unicode NFKC and, for Finnish, w is replaced by v, so that historical spelling differences do not masquerade as OCR errors. For the Llama models this overgeneration filter is decisive: without it their English CER% would be strongly negative, whereas with it they show positive gains.
What would settle it
Run the same Llama-3.1-70B prompt-based correction on the 28,907 English pages that were filtered out because their OCR and ground truth aligned poorly, and compare CER% on those pages to the 38.7% reported on the kept test set; if the excluded pages show no improvement or negative improvement, the method's practical value on uncurated collections is much smaller than the headline result.
Extended reading notes
Core claim
The paper's central claim is that in a zero-shot, prompt-based setting, open-weight LLMs can deliver substantial OCR post-correction for historical English but not for historical Finnish. On a 200-segment test set drawn from English ECCO-TCP OCR-ground-truth pairs, Llama-3.1-70B-Instruct reaches 38.7% relative CER improvement and all open models except Mixtral improve CER, while GPT-4o reaches 58.1%. On 449 pages of Finnish newspaper ground truth, all open models produce negative CER improvement, meaning their output is on average farther from the ground truth than the original OCR; GPT-4o is the only tested model with a positive Finnish result, at 11.9% CER improvement. The authors conclude that prompt-based post-correction is not practically usable for Finnish with current open-weight models, and they attribute much of the difficulty to the absence of Finnish support in the open models. They further show that removing model overgeneration with string alignment is necessary for the Llama family, and that normalizing historical spelling variants accounts for a large share of the apparent gains.
Load-bearing premise
The evaluation assumes the manually corrected ground truth is reliable and that discarding pages with poor OCR-to-ground-truth alignment—about 9% of English pages and 6% of Finnish pages—does not remove the hardest cases, so the reported gains may shrink on real, unfiltered collections.
Editorial extensions
If this is right
- A large-scale English correction run could use Llama-3.1-70B at 4-bit quantization, which fits on two GPUs and preserves most of the quality gain, to improve the ECCO collection's text.
- Any planned open-weight correction of historical Finnish collections should not be run as zero-shot prompting; fine-tuning or a Finnish-capable model is required, matching the paper's stated next step.
- Evaluation results across OCR post-correction studies are not directly comparable unless they apply the same spelling normalization and overgeneration handling, since these steps can swing scores by tens of percentage points.
- For long documents, segments of roughly 200 to 300 sub-words are preferable to shorter segments, and providing the preceding segment as context helps the largest models correct text near segment boundaries.
Reading between the lines
- Because 8.6% of English pages and 6% of Finnish pages were discarded for poor OCR-to-ground-truth alignment, the reported CER improvements describe a curated subset; on a real uncurated collection, where missing or badly garbled text is common, the practical gain could be substantially smaller, as the authors acknowledge in their limitations section.
- The Finnish failure may be primarily a language-support gap rather than a fundamental limit of LLMs, since none of the open models officially support Finnish; testing a Finnish-competent open model or a fine-tuned Finnish model is the obvious way to separate these explanations.
- Part of the measured 'correction' is actually modernization of historical orthography, not repair of OCR damage; users who need to preserve original spellings would see a smaller benefit than the reported CER% suggests.
- A multimodal extension that feeds the original page image together with the OCR text could address the missing-text cases the paper explicitly sets aside, because the current pipeline can only rewrite what the OCR already produced.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates zero-shot, prompt-based post-correction of OCR errors in historical English (ECCO-TCP, 301,937 pages after filtering) and Finnish (NLF ground truth, 449 pages) using seven LLMs, including open-weight models and GPT-4o as a reference. After filtering page pairs for alignment quality, splitting pages into fixed-length segments, optimizing temperature/top_k/top_p on a development set, and applying an alignment-based overgeneration-removal step, the authors report length-weighted relative CER and WER improvements on a held-out test set. They find that most open-weight models improve English OCR (best: Llama-3.1-70B at 38.7% CER reduction), while all tested open-weight models worsen Finnish OCR, leading to the conclusion that prompt-based post-correction is currently infeasible for historical Finnish with these models. They also study quantization, segment length, and context-aware correction at segment boundaries.
Significance. This is a carefully executed empirical study in an area with conflicting prior results. Strengths include a held-out development set for hyperparameter selection, a separate test set, public datasets and evaluation scripts, and reporting of both CER and WER along with the effect of normalization and overgeneration removal. The negative Finnish results are a practically important caution for digital humanities projects. The main caveat is that the English headline result is measured on a filtered subset of pages, so its applicability to the full ECCO collection needs quantification.
major comments (2)
- [Section 3.1 and Table 2] The 100-character-window alignment filter removes 28,907 (8.6%) English pages, and the authors list 'misidentified reading order or column layout' as one of the typical causes. This is not a case of text being missing, so the Limitations statement that the method 'cannot recover text where significant portions are missing' does not cover it; an LLM prompted to correct OCR could in principle reorder or restructure such text. Because the test set is restricted to pages that pass the alignment filter, the reported 38.7% CER improvement for Llama-3.1-70B is measured on a linearly aligned, easier subset, and the practical gain on the full ECCO collection may be substantially lower. Please analyze the discarded pages (e.g., report their CER distribution and the share attributable to each listed cause) and provide a conservative estimate for the full collection, for instance by applying the correction to all pages and falling back to the original OCR for pages where the post-processing alignment fails.
- [Section 4.3 and Table 3] The alignment-based overgeneration removal is decisive for the English results, particularly for the Llama family: for Llama-3.1-70B the CER improvement changes from -53.6% without the step to +38.7% with it. The method selects the span between the first and last characters of the model output that align to the original OCR. This can in principle also remove legitimate corrections at the beginning or end of a segment that share no characters with the noisy input. Please provide a manual or automatic check of trimmed outputs (e.g., how often the trimmed span ends inside a word that differs from OCR, and whether results are stable under variations of the gap penalties), so readers can be confident the gains are not an artifact of the trimming rule.
minor comments (6)
- [Section 4] Please clarify whether the Finnish development and test segments are page-disjoint; because each Finnish page is split into roughly 10 segments, independent segment sampling can place segments from the same page in both splits. If they are not page-disjoint, a page-level split should be used to avoid leakage.
- [Table 2 caption] The caption does not state that CER%/WER% are length-weighted averages of example-wise values clipped at -100%; please add this information to the caption or table notes.
- [Section 4.4] The description of the hyperparameter selection ('the median value of the 10 best runs of every model') is underspecified: it should state whether the median is taken per parameter across the 10 runs and whether any interaction between parameters was considered.
- [Limitations] The Limitations text says '~10% for English', but Section 3.1 reports 8.6% removed by the alignment filter and 0.4% and 1.7% by earlier filters; please align these numbers for precision.
- [Section 6] The sentence 'the gains diminishing past about 200–300 words, but our page-level data does not have long-enough examples to allow us to reach the point where the performance would start consistently decreasing' is hard to parse; please rephrase to distinguish the flattening of gains from a possible later decline.
- [General] Given that the main results are based on 200 test segments, consider reporting bootstrap confidence intervals for the CER% values in Table 2 to give a sense of the uncertainty around the reported differences.
Circularity Check
No circularity: the paper is an empirical benchmark against external ground truth, with standard dev/test separation.
full rationale
The paper makes no theoretical derivation and presents no fitted parameter as a prediction. The central measurements are CER/WER reductions computed by comparing LLM outputs to externally produced, manually corrected ground truth (ECCO-TCP and NLF), not to quantities derived from the models themselves. Hyperparameters (temperature, top_k, top_p) are optimized on a separate development set and then fixed for test reporting, so test numbers are not fitted inputs. The overgeneration-removal heuristic aligns model output to the original OCR input (not to the ground truth), and the subsequent CER is measured against GT, so no metric is defined in terms of the result it claims. The data filtering that removes roughly 8.6% of English and 6% of Finnish pages is an acknowledged limitation that bounds the method's scope, but it is not a circular step: it restricts the evaluation set rather than encoding the target outcome. No load-bearing self-citation or imported uniqueness claim appears. The paper's negative Finnish result and positive English result are direct empirical findings and remain independently falsifiable against the released data and scripts.
Assumptions & free parameters
free parameters (2)
- Temperature, top_k, top_p =
English: 0.26, 65, 0.66; Finnish: 0.14, 30, 0.60
- Segment length (200 words English, 100 words Finnish) =
200 words for English, 100 words for Finnish
assumptions (2)
- domain assumption The manually corrected ground truth (ECCO-TCP and NLF) is accurate and reliable.
- domain assumption The filtering procedure (removing pages with poor alignment) does not introduce a bias that invalidates the comparison.
Cite this review
Pith. "Pith review of OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches." pith.science (2026). https://pith.science/paper/V4HFG3EL
@misc{pith2026250201205,
author = {Pith},
title = {Pith review of: OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches},
year = {2026},
howpublished = {\url{https://pith.science/paper/V4HFG3EL}},
note = {Machine review of arXiv:2502.01205}
}
read the original abstract
Optical Character Recognition (OCR) systems often introduce errors when transcribing historical documents, leaving room for post-correction to improve text quality. This study evaluates the use of open-weight LLMs for OCR error correction in historical English and Finnish datasets. We explore various strategies, including parameter optimization, quantization, segment length effects, and text continuation methods. Our results demonstrate that while modern LLMs show promise in reducing character error rates (CER) in English, a practically useful performance for Finnish was not reached. Our findings highlight the potential and limitations of LLMs in scaling OCR post-correction for large historical corpora.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
AI@Meta. 2024 a . https://github.com/meta-llama/\ 3/blob/main/MODEL\_CARD.md Llama 3 model card
work page 2024
-
[4]
AI@Meta. 2024 b . https://github.com/meta-llama/llama-models/blob/main/models/llama3\_1/MODEL\_\ .md Llama 3.1 model card
work page 2024
-
[5]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
2019
-
[6]
Angel Beshirov, Milena Dobreva, Dimitar Dimitrov, Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2024. Post- OCR text correction for Bulgarian historical documents. ArXiv preprint arXiv:2409.00527
work page Pith review arXiv 2024
-
[7]
Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Frédéric Kaplan. 2024. Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), page...
work page 2024
-
[8]
Jonathan Bourne. 2024. CLOCR-C : Context leveraging OCR correction with pre-trained language models. ArXiv preprint arXiv:2408.17428
arXiv 2024
Show all 33 references
-
[9]
Str \"o bel
Yung-Hsin Chen and Phillip B. Str \"o bel. 2024. TrOCR meets language models: An end-to-end post-correction approach. In Proceedings of the Document Analysis and Recognition -- ICDAR 2024 Workshops, pages 12--26. Springer Nature Switzerland
2024
-
[10]
Guillaume Chiron, Antoine Doucet, Micka \"e l Coustaty, and Jean-Philippe Moreux. 2017 a . ICDAR 2017 competition on post- OCR text correction. In Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition (ICDAR 2017), volume 1, pages 1423--1428. IEEE
2017
-
[11]
Guillaume Chiron, Antoine Doucet, Mickael Coustaty, Muriel Visani, and Jean-Philippe Moreux. 2017 b . Impact of OCR errors on the use of digital libraries: Towards a better access to information. In Proceedings of the 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL),...
2017
-
[12]
Oksana Dereza, Deirdre N \' Chonghaile, and Nicholas Wolf. 2024. `` To have the ` million ' readers yet '' : Building a digitally enhanced edition of the bilingual I rish- E nglish newspaper An Gaodhal (1881-1898). In Proceedings of the Third Workshop on Language Technologies ...
2024
-
[13]
a m \"a l \
Quan Duong, Mika H \"a m \"a l \"a inen, and Simon Hengchen. 2021. An unsupervised method for OCR post-correction and spelling normalisation for F innish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 240--248. Link \"o ping Univer...
2021
-
[14]
Ali Fahandari, Fatemeh Asadi Zeydabadi, Elham Shabaninia, and Hossein Nezamabadi-pour. 2024. Enhancing Farsi text recognition via iteratively using a language model. In Proceedings of the 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP)
2024
-
[15]
https://www.gale.com/intl/primary-sources/eighteenth-century-collections-online Eighteenth Century Collections Online
Gale. https://www.gale.com/intl/primary-sources/eighteenth-century-collections-online Eighteenth Century Collections Online
-
[16]
Stephen H. Gregg. 2021. Old Books and Digital Publishing: Eighteenth-Century Collections Online. Elements in Publishing and Book Culture. Cambridge University Press
2021
-
[17]
Stephen H. Gregg. 2022. The nature of ECCO-TCP . Digital Defoe: Studies in Defoe & His Contemporaries, 14(1)
2022
-
[18]
Hill and Simon Hengchen
Mark J. Hill and Simon Hengchen. 2019. Quantifying the impact of dirty OCR on historical text analysis: Eighteenth Century Collections Online as a case study . Digital Scholarship in the Humanities, 34(4):825--843
2019
-
[19]
Albert Q. Jiang , Alexandre Sablayrolles , Antoine Roux , Arthur Mensch , Blanche Savary , Chris Bamford , Devendra Singh Chaplot , Diego de las Casas , Emma Bou Hanna , Florian Bressand , et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088
2024 arXiv
-
[20]
Kimmo Kettunen, Jukka Kervinen, and Mika Koistinen. 2018. Creating and using ground truth OCR sample data for Finnish historical newspapers and journals. In Proceedings of the Digital Humanities in the Nordic Countries Conference
2018
-
[21]
Kimmo Kettunen, Mika Koistinen, and Jukka Kervinen. 2020. Ground truth OCR sample data of Finnish historical newspapers and journals in data improvement validation of a re-OCRing process. LIBER Quarterly: The Journal of the Association of European Research Libraries, 30(1):1--20
2020
-
[22]
a \"a kk \
Kimmo Kettunen and Tuula P \"a \"a kk \"o nen. 2016. Measuring lexical quality of a historical F innish newspaper collection ― analysis of garbled OCR data with basic language technology tools and means. In Proceedings of the Tenth International Conference on Language Resource...
2016
-
[23]
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. TrOCR : Transformer-based optical character recognition with pre-trained models. In Proceedings of The Thirty-Seventh AAAI Conference on Artificial Intelligenc...
2023
-
[24]
Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatanpää, Peter Sarlin, and Sampo Pyysalo. 2024. Poro 34b and the blessing of multilinguality. ArXiv preprint arXiv:2404.01856
2024 arXiv
-
[25]
G \'a bor Madar \'a sz, No \'e mi Ligeti-Nagy, Andr \'a s Holl, and Tam \'a s V \'a radi. 2024. OCR cleaning of scientific texts with LLMs . In Natural Scientific Language Processing and Research Knowledge Graphs, pages 49--58. Springer Nature Switzerland
2024
-
[26]
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, et al. 2024. https://www.kaggle.com/m/3301 Gemma
2024
-
[27]
Thi Tuyet Hai Nguyen, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. 2021. Survey of post- OCR processing approaches. ACM Comput. Surv., 54(6)
2021
-
[28]
Robert \"O stling, Katarina Gillholm, Murathan Kurfal , Marie Mattson, and Mats Wir \'e n. 2024. Evaluation of really good grammatical error correction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (L...
2024
-
[29]
Maria Carolina Penteado and F \'a bio Perez. 2023. Evaluating GPT -3.5 and GPT -4 on grammatical error correction for Brazilian Portuguese . In Proceedings of the LatinX in AI Workshop at ICML 2023
2023
-
[30]
Christophe Rigaud, Antoine Doucet, Micka \"e l Coustaty, and Jean-Philippe Moreux. 2019. ICDAR 2019 competition on post- OCR text correction. In Proceedings of the 15th International Conference on Document Analysis and Recognition (ICDAR 2019), pages 1588--1593. IEEE
2019
-
[31]
Elizabeth Soper, Stanley Fujimoto, and Yen-Yun Yu. 2021. BART for post-correction of OCR newspaper text. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 284--290. Association for Computational Linguistics
2021
-
[32]
Mikko Tolonen, Eetu Mäkelä, Ali Ijaz, and Leo Lahti. 2021. Corpus linguistics and Eighteenth Century Collections Online (ECCO) . Research in Corpus Linguistics, 9:19--34
2021
-
[33]
Martijn Veninga. 2024. LLMs for OCR post-correction. Master's thesis, University of Twente
2024
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.