REVIEW 2 major objections 6 minor 32 references
ICDAR 2026 HIPE-OCRepair Competition on LLM-Assisted OCR Post-Correction for Historical Documents
T0 review · 2 major / 6 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read Modern LLM systems can sharply reduce OCR errors on historical newspapers and books without re-scanning images, but they still over-correct when the original text is already clean.
desk verdict Solid shared-task report: usable multilingual OCR post-correction benchmark plus clear evidence that adapted LLMs beat the no-correction baseline, with over-correction risk documented. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The HIPE-OCRepair-2026 evaluation framework: parallel OCR/ground-truth transcription units scored by character Match Error Rate (cMER) after IR-style normalization (soft-hyphen joining, lowercasing, punctuation stripping) plus a sign-based preference score that counts how often correction helps versus harms each unit, so ranking reflects searchability gains rather than diplomatic layout fidelity.
What would settle it
Re-score the same system outputs against un-normalized, strictly diplomatic references that preserve case, punctuation, and historical letterforms; if the reported gains over the no-correction baseline disappear or reverse under that protocol, the claim that these systems improve practical text quality under the paper's own framing does not hold.
Extended reading notes
Core claim
LLM-assisted OCR post-correction can substantially improve retrieval-oriented text quality across heterogeneous historical collections: the best adapted runs reach overall character Match Error Rate near 0.005 with preference scores near 0.9, outperforming zero-shot systems and a no-correction baseline on English, French, and German material, while performance still varies by dataset, language, and noise level and over-correction remains a recurring risk on low-noise inputs.
Load-bearing premise
The claim rests on the premise that lowercasing, stripping punctuation, and joining soft hyphens correctly measures what matters for search and access, so systems are not unfairly rewarded or punished for layout or historical spelling the metric deliberately ignores.
Editorial extensions
If this is right
- Task-specific adaptation (continued pre-training plus fine-tuning) outperforms pure zero-shot prompting for historical OCR repair.
- No-correction is a non-trivial baseline in low-noise settings where unnecessary edits can degrade quality.
- Public release of the dataset, scorer, and pipeline enables systematic cross-system comparison going forward.
- Evaluation must track consistency of improvement (preference score), not only average error reduction.
- Over-correction control is essential before applying LLMs at scale to mixed-quality digital collections.
Reading between the lines
- Institutions with large OCR-debt corpora may gain more searchability by adapting mid-size models on domain data than by waiting for better general zero-shot models.
- Routing high-edit or flagged cases to a larger model, as one team did, points to hybrid inference as a practical cost–fidelity trade-off.
- An IR-style metric may understate scholarly harm if historical spellings or named entities needed for citation are silently modernized.
- Extending the same protocol to more languages and to image-informed models would test whether text-only post-correction can close the gap to full re-OCR.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the design, dataset, evaluation protocol, and results of HIPE-OCRepair-2026, an ICDAR shared task on LLM-assisted OCR post-correction for historical newspapers and books in English, French, and German (17th–20th century). Systems receive noisy OCR text (paragraph/article units) plus metadata and must produce corrected transcripts without source images. Evaluation uses a retrieval-oriented normalization and character Match Error Rate (cMER), complemented by a sign-based preference score and bootstrap CIs, with a no-correction baseline. Four teams submitted systems spanning zero-shot prompting, fine-tuning, and continued pre-training plus fine-tuning; the best adapted system (BnF-Mistral) reaches overall cMER ≈ 0.005 and preference scores near 0.9, with gains varying by language, dataset, and noise level and with documented over-correction risk on low-noise inputs. The harmonized HIPE-OCRepair-2026 dataset, scorer, and evaluation pipeline are released publicly.
Significance. If the reported results hold—and the public data, scorer, and evaluation repository make them checkable—the paper supplies a timely, reproducible multilingual benchmark for a practically important digital-heritage problem (legacy OCR debt) under realistic constraints (text-only post-correction). Strengths that should be credited explicitly include: (i) a harmonized multi-source dataset with documented semi-diplomatic curation and quality filtering; (ii) a fully open scorer and Makefile-driven evaluation pipeline with bootstrap CIs and paired significance testing; (iii) a no-correction baseline and preference score that surface over-correction rather than rewarding only average error reduction; and (iv) transparent system descriptions spanning adaptation depth. These assets go beyond a one-off leaderboard and support cumulative work on LLM post-correction, hallucination control, and IR-oriented evaluation of historical text.
major comments (2)
- Section 5.3 states that the impact of individual design choices (document metadata, explicit error modeling, hallucination control) “is examined in detail in the next Section on results,” and the abstract claims “insights into the merits of different adaptation strategies.” Section 6, however, mainly reports overall/per-language/per-dataset rankings (Tables 5–7) and attributes BnF-Mistral’s lead to adaptation depth, without a systematic comparison of those design factors across runs. Either add a short, evidence-based comparison (e.g., metadata vs. no-metadata, zero-shot safeguards vs. fine-tuning, effect of judge-and-retry) or revise the 5.3/abstract wording so the claim matches what is actually shown.
- Introduction and Conclusion frame the open question as whether LLMs can reduce OCR error while avoiding over-correction and hallucination. Preference score and cMER address magnitude and consistency of improvement, and several systems describe length/judge safeguards, but the results section provides little qualitative or error-type evidence of residual hallucination (plausible but absent content) versus true correction. A brief residual-error analysis on a sample of degraded units (especially dta19-l0 and high-preference failures) would make the answer to that framing question more load-bearing rather than inferred from aggregate scores alone.
minor comments (6)
- Abstract and opening: “Large language models (LLMs) offers” → “offer”; several other subject–verb and spacing issues appear in the provided text (e.g., “thereforeremains,” “workingatthelevel”). A full copy-edit pass is needed.
- Section 4.1: cMER is clearly defined; briefly restate why insertions in the denominator are preferred over standard CER for generative systems that may over-produce, so readers unfamiliar with [18] can follow without leaving the paper.
- Table 2 / Section 3.2: impresso-nzz and overproof are correctly excluded from official ranking because test data were public; state this exclusion criterion once in a single prominent place (e.g., caption of Table 1 or ranking protocol) to avoid repeated footnotes.
- Section 6 / Table 7: When reporting statistical indistinguishability (impresso-snippets English), cite the paired bootstrap procedure and α level in the table note for self-contained reading.
- Participation is limited to four teams; a short limitations paragraph noting coverage gaps (e.g., Zakaria-ENSIAS missing dta19-l2) and the consequent caution on generalizing adaptation rankings would help readers calibrate the “merits of different strategies” claim.
- References and related work are appropriate; ensure consistent venue formatting and that the promised dedicated benchmark paper (Section 3.1) is either cited as forthcoming or the essential curation guidelines are self-contained here.
Circularity Check
No significant circularity: empirical shared-task report with independent metrics, public scorer, and no-construction predictions.
full rationale
This is an ICDAR competition report evaluating submitted LLM-based OCR post-correction systems on a newly harmonized multilingual historical dataset. The primary metric (cMER) and secondary preference score are defined from standard edit-distance alignments after explicit IR-style normalization; they are not derived from any system’s training objective or fitted parameters. The no-correction baseline simply returns the raw OCR hypothesis, providing an external anchor. Rankings (Tables 5–7) and bootstrap CIs are computed by a publicly released scorer on held-out test units whose ground truths were manually curated independently of the participating systems. Self-citations (prior HIPE/Impresso papers, authors’ earlier OCR studies) supply data sources, background, or related experimental settings; none function as load-bearing uniqueness theorems or force the reported cMER reductions by construction. There are no equations that redefine a fitted quantity as a prediction, no ansatz smuggled via self-citation, and no renaming of a known empirical pattern presented as a first-principles result. The paper is therefore self-contained against its external benchmarks; circularity score is zero.
Assumptions & free parameters
free parameters (3)
- language/dataset ranking weights
- cMER denominator including insertions
- CER quality filter threshold 0.15
assumptions (4)
- domain assumption Retrieval-oriented (IR-style) normalization—lowercasing, punctuation removal, soft-hyphen joining—is an appropriate primary evaluation target for historical OCR post-correction.
- domain assumption Systems must operate without access to source images, reflecting real OCR-debt scenarios.
- domain assumption Semi-diplomatic ground truth that prioritizes linguistically interpretable word forms over full historical graphemics is the correct reference standard.
- ad hoc to paper Character-level Match Error Rate and a sign-based preference score together adequately capture both magnitude and consistency of improvement.
Cite this review
Pith. "Pith review of ICDAR 2026 HIPE-OCRepair Competition on LLM-Assisted OCR Post-Correction for Historical Documents." pith.science (2026). https://pith.science/paper/SOFR2DF4
@misc{pith2026260708143,
author = {Pith},
title = {Pith review of: ICDAR 2026 HIPE-OCRepair Competition on LLM-Assisted OCR Post-Correction for Historical Documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/SOFR2DF4}},
note = {Machine review of arXiv:2607.08143}
}
read the original abstract
We present the results of HIPE-OCRepair-2026, an ICDAR competition on LLM-assisted OCR post-correction of historical documents. OCR post-correction remains a long-standing challenge in digital heritage: large-scale collections of digitized documents are affected by legacy OCR errors, while re-digitization at scale remains impractical. Large language models (LLMs) offers a major opportunity to revisit this challenge, yet their effectiveness across languages, document types, and noise conditions - and their tendency to hallucinate - remains insufficiently understood. HIPE-OCRepair-2026 pursues two objectives: (i) to evaluate the capabilities of modern OCR post-correction systems, and (ii) to provide a reproducible evaluation framework anchored in the HIPE-OCRepair-2026 dataset, a harmonized multilingual resource consolidating existing and newly curated historical datasets. Participants were tasked with correcting noisy OCR transcripts from historical newspapers and printed works in English, French, and German (17th-20th century), working at the level of coherent transcription units (paragraphs or articles) without access to source images. The evaluation adopts a retrieval-oriented rather than diplomatic scoring approach, reflecting the practical use case of search and access over digitized collections. Four teams submitted systems ranging from zero-shot prompting to continued pre-training and fine-tuning, offering insights into the merits of different adaptation strategies. Results show that modern LLM-assisted systems can significantly improve OCR quality, but performance varies across datasets, languages, and noise levels. Over-correction on low-noise inputs emerges as a recurring challenge, highlighting the importance of evaluation beyond character error reduction. The dataset, scorer, and evaluation pipeline are publicly released to support future research.
Reference graph
Works this paper leans on
-
[1]
Amrhein, C., Clematide, S.: Supervised OCR Error Detection and Correction Using Statistical and Neural Machine Translation Methods. Journal for Lan- guage Technology and Computational Linguistics (JLCL)33(1), 49–76 (2018), https://jlcl.org/content/2-allissues/1-heft1-2018/jlcl_2018-1_3.pdf
work page 2018
-
[2]
Balk, H., Ploeger, L.: IMPACT: Working together to address the challenges in- volving mass digitization of historical printed text. OCLC Systems & Services: International digital library perspectives (Oct 2009),https://www.emerald.com/ insight/content/doi/10.1108/10650750911001824/full/html
-
[3]
Beals, M., Bell, E., Cordell, R., Fyfe, P., Russell, I.G., Hauswedell, T., Neudecker, C., Nyhan, J., Oiva, M., Pado, S., Pimentel, M.P., Rose, L., Salmi, H., Terras, M., Viola, L.: The Atlas of Digitised Newspapers: Reports from Oceanic Exchanges (May 2020)
work page 2020
-
[4]
Beshirov, A., Dobreva, M., Dimitrov, D., Hardalov, M., Koychev, I., Nakov, P.: Post-ocr text correction for Bulgarian historical documents. International Journal on Digital Libraries26(1), 4 (Feb 2025),https://doi.org/10.1007/ s00799-025-00415-x 16 Ehrmann et al
work page 2025
-
[5]
Boros, E., Ehrmann, M., Romanello, M., Najem-Meyer, S., Kaplan, F.: Post- Correction of Historical Text Transcripts with Large Language Models: An Ex- ploratory Study. In: Proc. of the 8th Joint SIGHUM / LaTeCH-CLfL 2024 Work- shop. pp. 133–159. Association for Computational Linguistics, St. Julians, Malta (Mar 2024),https://aclanthology.org/2024.latechclfl-1.14/
work page 2024
-
[6]
International Jour- nal on Digital Libraries23(3), 241–266 (2022)
Boros, E., Nguyen, N.K., Lejeune, G., Doucet, A.: Assessing the impact of OCR noise on multilingual event detection over digitised documents. International Jour- nal on Digital Libraries23(3), 241–266 (2022)
work page 2022
-
[7]
In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)
Chiron, G., Doucet, A., Coustaty, M., Moreux, J.P.: ICDAR2017 Competition on Post-OCR Text Correction. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 01, pp. 1423–1428 (Nov 2017), https://ieeexplore.ieee.org/document/8270163
- [8]
Show all 32 references
-
[9]
IEEE Press, USA (2017),http://dl.acm.org/citation.cfm?id=3200334. 3200364
2017
-
[10]
ACM Comput- ing Surveys56(2), 27:1–27:47 (2023),https://dl.acm.org/doi/10.1145/3604931
Ehrmann, M., Hamdi, A., Pontes, E.L., Romanello, M., Doucet, A.: Named Entity Recognition and Classification in Historical Documents: A Survey. ACM Comput- ing Surveys56(2), 27:1–27:47 (2023),https://dl.acm.org/doi/10.1145/3604931
2023 doi
-
[11]
In: Proc
Evershed, J., Fitch, K.: Correcting noisy OCR: Context beats confusion. In: Proc. of the First International Conference on Digital Access to Textual Cultural Her- itage. pp. 45–51. DATeCH ’14, Association for Computing Machinery, New York, NY, USA (May 2014),https://dl.acm.org...
2014 doi
-
[12]
In: Proc
Gupta, H., Del Corro, L., Broscheit, S., Hoffart, J., Brenner, E.: Unsupervised multi-view post-OCR error correction with language models. In: Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 8647–8652 (2021)
2021
-
[13]
In: Dig- ital Libraries for Open Knowledge
Hamdi, A., Jean-Caurant, A., Sidère, N., Coustaty, M., Doucet, A.: Assessing and Minimizing the Impact of OCR Quality on Named Entity Recognition. In: Dig- ital Libraries for Open Knowledge. pp. 87–101. Lecture Notes in Computer Sci- ence,SpringerInternationalPublishing,Cham(2...
2020
-
[14]
He, Z., Zhang, C., Wu, Z., Chen, Z., Zhan, Y., Li, Y., Zhang, Z., Wang, X., Qiu, M.: Seeing is believing? mitigating ocr hallucinations in multimodal large language models (2025),https://arxiv.org/abs/2506.20168
2025
-
[15]
Jarlbrink, J., Snickars, P.: Cultural heritage as digital noise: Nineteenth century newspapersinthedigitalarchive.JournalofDocumentation73(6),1228–1243(Oct 2017),http://www.emerald.com/jd/article/73/6/1228-1243/203503
2017
-
[16]
In: Proc
Kanerva, J., Ledins, C., Käpyaho, S., Ginter, F.: OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches. In: Proc. of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains. pp. 38–47. University of Tartu Library, E...
2025
-
[17]
arXiv preprint arXiv:2501.11623 (2025)
Kim, S., Baudru, J., Ryckbosch, W., Bersini, H., Ginis, V.: Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records. arXiv preprint arXiv:2501.11623 (2025)
2025 arXiv
-
[18]
In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS)
Neudecker, C., Antonacopoulos, A.: Making Europe’s Historical Newspapers Searchable. In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS). pp. 405–410. IEEE, Santorini, Greece (Apr 2016),http://ieeexplore.ieee.org/ document/7490152/ HIPE-OCRepair 2026 17
2016
-
[19]
In: Proc
Neudecker, C., Baierer, K., Federbusch, M., Boenig, M., Würzner, K.M., Hart- mann, V., Herrmann, E.: OCR-D: An End-to-End Open Source OCR Framework for Historical Printed Documents. In: Proc. of the 3rd International Conference on Digital Access to Textual Cultural Heritage. p...
2019 doi
-
[20]
ACM Computing Surveys54(6), 124:1–124:37 (Jul 2021), https://dl.acm.org/doi/10.1145/3453476
Nguyen, T.T.H., Jatowt, A., Coustaty, M., Doucet, A.: Survey of Post-OCR Pro- cessing Approaches. ACM Computing Surveys54(6), 124:1–124:37 (Jul 2021), https://dl.acm.org/doi/10.1145/3453476
2021 doi
-
[21]
Rigaud, C., Doucet, A., Coustaty, M., Moreux, J.P.: ICDAR 2019 competition on post-OCRtextcorrection.In:2019InternationalConferenceonDocumentAnalysis and Recognition (ICDAR). pp. 1588–1593. IEEE (2019)
2019
-
[22]
In: Proc
Schaefer, R., Neudecker, C.: A Two-Step Approach for Automatic OCR Post- Correction. In: Proc. of the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature. pp. 52–57. International Committee on Computation...
2020
-
[23]
Smith, D., Cordell, R.: A Research Agenda for Historical and Multilingual Optical Character Recognition. Tech. rep., Northeastern University (Jun 2023),http:// hdl.handle.net/2047/D20297452
2023
-
[24]
In: Proc
Soper, E., Fujimoto, S., Yu, Y.Y.: BART for Post-Correction of OCR Newspaper Text. In: Proc. of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021). pp. 284–290. Association for Computational Linguistics, Online (Nov 2021), https://aclanthology.org/2021.wnut-1.31/
2021
-
[25]
Jour- nal for Language Technology and Computational Linguistics33(1), 97–114 (Jul 2018),https://jlcl.org/article/view/220
Springmann, U., Reul, C., Dipper, S., Baiter, J.: Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin. Jour- nal for Language Technology and Computational Linguistics33(1), 97–114 (Jul 2018),https://jlcl.org/article/view/220
2018
-
[26]
Ströbel, P., Clematide, S.: Ground truth for Neue Zürcher Zeitung black letter period (Jul 2019),https://zenodo.org/record/3333627
2019
-
[27]
In: Proc
Ströbel, P., Clematide, S.: Improving OCR of black letter in historical newspapers: The unreasonable effectiveness of HTR models on low-resolution images. In: Proc. of the Digital Humanities 2019, (DH2019) (2019)
2019
-
[28]
In: Workshop on Computational Methods in the Humanities 2022 (Jun 2022),https://www.zora.uzh.ch/handle/20.500.14742/207406
Ströbel, P.B., Clematide, S., Hodel, T., Volk, M.: Transformer-based HTR for Historical Documents. In: Workshop on Computational Methods in the Humanities 2022 (Jun 2022),https://www.zora.uzh.ch/handle/20.500.14742/207406
2022
-
[29]
In: Proc
Thomas, A., Gaizauskas, R., Lu, H.: Leveraging LLMs for post-OCR correction of historical newspapers. In: Proc. of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA)@ LREC-COLING-2024. pp. 116–121 (2024)
2024
-
[30]
In: Proc
van Strien, D., Beelen, K., Ardanuy, M., Hosseini, K., McGillivray, B., Colav- izza, G.: Assessing the Impact of OCR Quality on Downstream NLP Tasks:. In: Proc. of the 12th International Conference on Agents and Artificial Intelligence. pp.484–496.SCITEPRESS,Valletta,Malta(202...
2020 doi
-
[31]
Workshop presen- tation, OCR-BW, 9 June 2021 (Jun 2021),https://ocr-bw.bib.uni-mannheim
Weil, S.: Neue Modelle dank GT-Aufwertung und Anreicherung. Workshop presen- tation, OCR-BW, 9 June 2021 (Jun 2021),https://ocr-bw.bib.uni-mannheim. de/wp-content/uploads/2021/06/2021-06-09_WS_Neue_Modelle.pdf
2021
-
[32]
In: Proc
Zhang, J., Haverals, W., Naydan, M., Kernighan, B.W.: Post-OCR Correction with OpenAI’s GPT Models on Challenging English Prosody Texts. In: Proc. of the ACM Symposium on Document Engineering 2024. pp. 1–4. DocEng’24, ACM, New York, USA (Sep 2024),https://dl.acm.org/doi/10.114...
2024 doi
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.