Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On 20 pages of 1921 Belgian handwritten inheritance declarations, two-shot GPT-4o and two-shot Claude Sonnet 3.5 produce transcriptions closer to human ground truth than the conventional OCR/HTR tools tested, including fine-tuned TrOCR.

desk verdict Useful early comparison between LLMs and OCR/HTR tools, but the headline two-shot result is unverifiable without a documented train/test split for the example images. read the letter →

arxiv 2501.11623 v1 pith:NRNX2MYO submitted 2025-01-20 cs.CV cs.AIcs.LG

Seorin Kim , Julien Baudru , Wouter Ryckbosch , Hugues Bersini , Vincent Ginis This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords LargeLanguageModelsHistoricalDocumentsHandwrittenTextRecognitionOpticalCharacterErrorRateBLEUscoreFew-shotpromptingTabularrecords
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a single large language model can replace the conventional multi-stage pipeline of preprocessing, layout segmentation, and text recognition for digitising historical handwritten tables. On 20 pages of 1921 Belgian inheritance declarations, the authors compare GPT-4o and Claude Sonnet 3.5 against EasyOCR, KerasOCR, Pytesseract, and TrOCR, including fine-tuned variants. They report that both LLMs transcribe the records far more accurately, and that giving the model two example image-transcription pairs in the prompt produces the best results: two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole scans. If these results hold beyond the test set, historians and archivists could obtain high-quality transcriptions with almost no ground truth data and no per-collection fine-tuning.

What carries the argument

The central mechanism is few-shot vision-language prompting: the scanned document image is handed directly to a multimodal LLM together with one or two previous image-transcription pairs that teach the layout and handwriting style. This replaces the classical OCR/HTR chain—contrast adjustment, layout analysis, line and word segmentation, per-region recognition, and fine-tuning—with a single model call, and the paper credits this replacement for the accuracy jump over the conventional tools.

What would settle it

Inspect the example images used in the one- and two-shot prompts and compare their provenance against the 20 evaluated pages; if any example page is also a scored page, the few-shot result is invalidated. Alternatively, rerun the two best configurations with examples taken only from a separate held-out set of similar documents and check whether the CER and BLEU advantage over fine-tuned TrOCR survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that, for tabular handwritten historical records, a multimodal LLM prompted with one or two example transcriptions outperforms established OCR/HTR tools, and does so without any fine-tuning or layout pre-processing. The authors demonstrate this on Belgian probation records from 1921, scoring outputs with Character Error Rate and BLEU; they find that zero-shot LLM outputs already surpass fine-tuned TrOCR, and that adding two example image-transcription pairs yields the best transcriptions overall. They also show that the two automatic metrics disagree with human judgment on whole-scan outputs, mainly because the repetitive typed header dominates CER/BLEU, and that BLEU distinguishes method quality better than CER for this task. The conclusion is that a one-step LLM approach is a viable alternative to the traditional pipeline, needing as few as two pieces of ground truth data.

Load-bearing premise

The study's conclusion depends on the one- and two-shot example transcriptions not being drawn from the same 20 pages being scored; the paper does not state where these examples come from, so if any overlap the test set, the reported few-shot advantage would be partly circular.

Editorial extensions

If this is right

  • If the two-shot results generalize, a new historical collection can be transcribed after hand-curating just two example rows, instead of building a fine-tuning set of hundreds of lines.
  • The finding that BLEU separates good from bad transcriptions better than CER on long documents suggests future OCR/HTR benchmarks should not rely on CER alone.
  • Because the LLMs reproduce table structure in a single pass, downstream analytics can consume structured output directly, without layout-reconstruction post-processing.
  • Fine-tuned TrOCR with half the data still underperformed the best zero- and few-shot LLM outputs, indicating that per-collection labeled data may no longer be necessary for this document class.
  • Whole-scan and line-by-line inputs favor different models, so choosing the input framing is itself part of the transcription strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors never test whether the two-shot examples transfer across handwriting styles, years, or archives; based on the reported mechanism, the recipe would be example pages that resemble the target collection, so the practical protocol should include a small similarity check.
  • A hybrid pipeline is a plausible extension the paper does not run: LLM transcription for handwritten text columns plus a classical OCR pass for numeric columns, given that both LLMs struggled with digits.
  • Because the paper shows the repetitive header inflates CER/BLEU and distorts ranking, a reweighted metric that ignores known-stable regions would likely change which method looks best in future studies.
  • The observed day-to-day drift in LLM behavior suggests that reproducibility, not raw accuracy, will be the next bottleneck for applying LLM transcription at scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper compares two multimodal LLMs (GPT-4o and Claude Sonnet 3.5) with four conventional OCR/HTR tools (EasyOCR, KerasOCR, Pytesseract, TrOCR) on transcription of 20 scanned pages of a 1921 Belgian Déclaration de Succession. Experiments are run both line-by-line and on whole scans, with five prompting strategies for the LLMs and additional fine-tuned TrOCR variants. Performance is measured by CER and BLEU, supplemented by a human evaluation of whole-scan outputs. The paper claims that LLMs outperform the OCR/HTR baselines overall and that, among the tested strategies, two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole-scan images produce transcriptions closest to ground truth.

Significance. If the central claim holds, this is a useful early data point showing that a single multimodal LLM call can rival or beat traditional multi-stage OCR/HTR pipelines on a narrow tabular historical document, and that two ground-truth examples may suffice for strong few-shot performance. The paper has several strengths: it reports concrete prompts, compares multiple baselines, includes a human evaluation, and discusses metric limitations such as BLEU's sensitivity to n-gram order and CER's sensitivity to whitespace. The main significance is therefore empirical rather than methodological. However, the headline few-shot result rests on an unverifiable assumption about the origin of the example images, and the reported results would be substantially weakened if the examples come from the same pages being scored.

major comments (4)
  1. [Section 3.3 and Table 2] The one-shot and two-shot conditions are described only as 'providing the model with one or two example images and transcriptions' (Section 3.3), and Table 2's prompt inserts {example(s)} without stating whether those examples come from the same 20 pages being evaluated. Since Section 3.1 defines the dataset as exactly 20 scanned pages and the conclusion (Section 6) states that 'as few as two pieces of ground truth data' suffice, the few-shot examples may be drawn from the evaluation set. If so, the model receives ground-truth transcriptions for pages it is then asked to score, making the two-shot advantage in the abstract and Section 6 an artifact of leakage. The authors must document the provenance of the example images and, if any overlap exists, rerun the few-shot experiments with a held-out split. The zero-shot results would still be informative, but the headline best-strategy claim is currently unverifiable.
  2. [Section 4 and Appendix Fig. 6] The maximum n-gram order for BLEU differs between the two experimental settings: 3 for line-by-line and 4 for whole-scan (Section 4). As Figure 6 shows, the n-gram order materially changes BLEU magnitude. This makes the cross-setting comparison in Section 4.1 and the conclusion that 'LLMs seem to perform better when the document images are sliced per row' unsupported, because the two arms are not evaluated with the same metric. Use a common n-gram order when comparing line-by-line to whole-scan, or restrict this conclusion to within-setting comparisons.
  3. [Section 3.3 / supplementary material] The text states that 'some anomalies in the outputs are manually treated to ensure a fair and accurate comparison to the GT data (see details in the supplementary material)', but the supplementary material is not included in the manuscript. Because this manual post-processing is applied after seeing the outputs and could favor one method over another, the procedure must be specified completely: which criteria define an anomaly, which outputs were edited, and what edits were made. Without this, the reported CER/BLEU numbers are not reproducible.
  4. [Section 4 and Table 1] No statistical tests or confidence intervals accompany the headline comparisons. With 20 whole-scan samples (and 283 lines, many of which share layout and header text), the differences between the best few-shot strategies and the runners-up in Figures 1, 3, and Table 1 may be within sampling variability. At minimum, report paired tests (e.g., Wilcoxon signed-rank) for the main CER/BLEU comparisons and for the human-evaluation ranking. This is especially important because the best strategy was selected post hoc after inspecting the results.
minor comments (4)
  1. [Table 2 / Refine prompt] The refine prompt contains a typo: 'trasncription' should be 'transcription'.
  2. [Section 4.2 and Table 1] The human evaluation was conducted by the two lead authors, who also created the ground truth, but no inter-rater reliability measure is reported; please state this limitation explicitly or report agreement statistics.
  3. [Section 3.3] The practice of rerunning documents or rows that return error messages may introduce selection bias; report how often this occurred and whether the rerun outputs were used without averaging.
  4. [Section 6] The conclusion generalizes to 'LLMs' and 'conventional OCR/HTR tools' from two LLMs and four baselines; consider qualifying the claim to the specific models and document type tested.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: the paper's claims rest on empirical comparisons, and the few-shot example provenance gap is a reproducibility concern, not a circular reduction.

full rationale

The paper contains no formal derivation chain whose conclusion is equivalent to its assumptions by construction. The central result is an empirical CER/BLEU comparison between LLM prompts and OCR/HTR tools, and the metrics themselves are standard external measures applied after the fact. The one-shot and two-shot strategies are defined as in-context learning: the prompt supplies example images and their transcriptions, which is a normal few-shot protocol, not a fit-and-predict loop. For this to be circular in the sense of the analysis, the example transcriptions would have to be ground truth for the exact images being scored; the paper does not state that, and it also does not state the opposite. The dataset section describes only the 20 scanned pages, and the conclusion that 'as few as two pieces of ground truth data' suffice does not itself demonstrate that the two examples were drawn from the evaluated pages. The absence of documentation about example selection is a legitimate reproducibility and validity concern, but it is not exhibited as a specific reduction from an equation or setup in the text. Similarly, the human evaluations were performed by the authors who created the ground truth, which introduces subjectivity, but this is not circular reasoning. The TrOCR fine-tuning procedure may mix training and evaluation data, but that would only make the comparison harder for the LLMs, not circular. Overall, the paper's claims are self-contained empirical observations rather than derivations; no load-bearing step reduces to its own input on the evidence provided.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical comparison, not a derivation, so the free-parameter ledger lists methodological choices (BLEU n-gram order, fine-tuning percentages, manual anomaly handling) rather than fitted constants. The axioms are unverified assumptions about ground truth, dataset representativeness, split integrity, and evaluator objectivity.

free parameters (3)
  • BLEU maximum n-gram order = line-by-line: 3, whole-scan: 4
    Chosen by the authors based on observed text lengths and score distributions; affects comparability of BLEU between experimental types and across methods.
  • TrOCR fine-tuning data percentage = 20% and 50% for line-by-line; 20% for whole-scan
    Selected to test limited-data fine-tuning; results depend on this choice and the six training epochs.
  • Manual anomaly exclusion = unspecified
    The authors manually treated some output anomalies to ensure fair comparison; the exact rules are not provided in the main text, making the process subjective.
assumptions (4)
  • domain assumption The ground-truth transcriptions produced and double-checked by the authors are correct.
    The GT is the reference for all CER/BLEU and human scores; any errors in GT would propagate into every comparison.
  • domain assumption The 20 scanned pages are representative of historical handwritten tabular documents for comparing systems.
    Claims of general LLM superiority rest on this single corpus of one layout and one scribal hand.
  • ad hoc to paper The one-shot and two-shot example images and transcriptions are either not part of the evaluated set or do not bias the scores.
    The paper does not describe a train/test split; the conclusion that two-shot works best may be invalid if examples overlap with test pages.
  • domain assumption The human evaluations by the two lead authors are unbiased and consistent.
    The evaluators are also the GT creators, and no inter-rater reliability or blinding is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records." pith.science (2026). https://pith.science/paper/NRNX2MYO

@misc{pith2026250111623,
  author       = {Pith},
  title        = {Pith review of: Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRNX2MYO}},
  note         = {Machine review of arXiv:2501.11623}
}
read the original abstract

We explore the ability of two LLMs -- GPT-4o and Claude Sonnet 3.5 -- to transcribe historical handwritten documents in a tabular format and compare their performance to traditional OCR/HTR systems: EasyOCR, Keras, Pytesseract, and TrOCR. Considering the tabular form of the data, two types of experiments are executed: one where the images are split line by line and the other where the entire scan is used as input. Based on CER and BLEU, we demonstrate that LLMs outperform the conventional OCR/HTR methods. Moreover, we also compare the evaluated CER and BLEU scores to human evaluations to better judge the outputs of whole-scan experiments and understand influential factors for CER and BLEU. Combining judgments from all the evaluation metrics, we conclude that two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole-scan images yield the transcriptions of the historical records most similar to the ground truth.

Figures

Figures reproduced from arXiv: 2501.11623 by the authors.

Figure 1
Figure 1. BLEU score comparisons for each method on the line- [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. CER score comparisons for each method on the line-b [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. BLEU score comparisons of each method between the w [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: CER score comparisons of each method between the wh [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: BLEU and CER scores with the headers ( ) and without the headers ( ) for one-example and two-example prompts by GPT-4o and Claude Sonnet 3.5. After removing the headers, Claude Two-Example has the highest BLEU and lowest CER on average. Another challenge that we faced …
Figure 6
Figure 6. Figure 6: Comparisons between different maximum orders in n [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: BLEU and CER score comparisons for each method on th [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An HTR-LLM Workflow for High-Accuracy Transcription and Analysis of Abbreviated Latin Court Hand

    cs.DL 2025-07 conditional novelty 6.0 of 10

    A four-stage HTR plus LLM pipeline transcribed four medieval Latin court cases with word error rates between 2% and 7%, though the metric and case selection make that range optimistic.

  2. ICDAR 2026 HIPE-OCRepair Competition on LLM-Assisted OCR Post-Correction for Historical Documents

    cs.CL 2026-07 accept novelty 5.5 of 10

    LLM post-correction substantially lowers character match error on historical OCR, but gains vary by language and noise and over-correction remains a risk on clean inputs.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    I n: Bizzoni, Y ., Degaetano-Ortlieb, S., Kazantseva, A., Szpakowicz, S

    Boros, E., Ehrmann, M., Romanello, M., Najem-Meyer, S., Kaplan, F.: Post-correction of historical text tran- scripts with large language models: An exploratory study. I n: Bizzoni, Y ., Degaetano-Ortlieb, S., Kazantseva, A., Szpakowicz, S. (eds.) Proceedings of the 8th Joint SIGHU M Workshop on Computational Linguistics for Cultural Heritage, Social Scien...

    work page 2024

  2. [2]

    Fadeeva, A., Schlattner, P ., Maksai, A., Collier, M., Ko kiopoulou, E., Berent, J., Musat, C.: Representing online handwriting for recognition in large vision-language mode ls (2024), https://arxiv.org/abs/2402.15307

    arXiv 2024

  3. [3]

    Fujitake, M.: Dtrocr: Decoder-only transformer for opt ical character recognition (2023)

    work page 2023

  4. [4]

    In: International Conference on Theory and Practice of Digital Libraries

    Ghiriti, A., Göderle, W ., Kern, R.: Exploring the capabi lities of gpt4-vision as ocr engine. In: International Conference on Theory and Practice of Digital Libraries. pp. 3–12. Springer (2024)

    work page 2024

  5. [5]

    https://github.com/huggingface/evaluate (2024), https://github.com/huggingface/evaluate, version 0.4.3, Apache- 2.0 License

    HuggingFace: Evaluate: A library for easily evaluating machine learning models and datasets. https://github.com/huggingface/evaluate (2024), https://github.com/huggingface/evaluate, version 0.4.3, Apache- 2.0 License

    work page 2024

  6. [6]

    Other thesis, OST Ostschweizer Fachhochschule (May 2023), https://eprints .ost.ch/id/eprint/1189, thesis advisor: Mitra Purandare

    Löffler, K.: Digitize Historic Architectural Plans with OCR and NER Transformer Models. Other thesis, OST Ostschweizer Fachhochschule (May 2023), https://eprints .ost.ch/id/eprint/1189, thesis advisor: Mitra Purandare

    work page 2023

  7. [7]

    International Journal on Document Analysis and Recognitio n 5(1), 39–46 (2002)

    Marti, U.V ., Bunke, H.: The iam-database: an english sen tence database for offline handwriting recognition. International Journal on Document Analysis and Recognitio n 5(1), 39–46 (2002)

    work page 2002

  8. [8]

    In: Proceedings of the 40th annual meeting of the Association fo r Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W .J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association fo r Computational Linguistics. pp. 311–318 (2002)

    work page 2002

Show all 15 references
  1. [9]

    Rice, S.V ., Jenkins, F.R., Nartker, T.A.: The fifth annua l test of ocr accuracy. Tech. rep., University of Nevada, Las V egas (1996), https://www.psu.edu

    1996

  2. [10]

    Shi, Y ., Peng, D., Liao, W ., Lin, Z., Chen, X., Liu, C., Zh ang, Y ., Jin, L.: Exploring ocr capabilities of gpt- 4v(ision) : A quantitative and in-depth evaluation (2023), https://arxiv.org/abs/2310.16809

    2023 arXiv

  3. [11]

    Tam, Z.R., Wu, C.K., Tsai, Y .L., Lin, C.Y ., yi Lee, H., Ch en, Y .N.: Let me speak freely? a study on the impact of format restrictions on performance of large language mod els (2024), equal contribution, Equal advisorship

    2024

  4. [12]

    In: Sprugnoli, R., Passarotti, M

    Thomas, A., Gaizauskas, R., Lu, H.: Leveraging LLMs for post-OCR correction of historical newspapers. In: Sprugnoli, R., Passarotti, M. (eds.) Proceedings of the Thi rd Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024. pp. 11 ...

    2024

  5. [13]

    Toselli, A.H., Romero, V ., Villegas, M., Vidal, E., Sán chez, J.A.: Htr dataset icfhr 2016 (version 1.2.0) (2018). https://doi.org/10.5281/zenodo.1297399, https ://zenodo.org/record/218236, dataset from the READ project (Horizon 2020) containing annotated pages from the Ratsp...

    2018 doi

  6. [14]

    V aswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jo nes, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2023), https://arxiv.org/abs/1706.03762

    2023 arXiv

  7. [15]

    first level

    Xu, W ., Deutsch, D., Finkelstein, M., Juraska, J., Zhan g, B., Liu, Z., Wang, W .Y ., Li, L., Freitag, M.: Llmrefine: Pinpointing and refining large language models vi a fine-grained actionable feedback (2024), https://arxiv.org/abs/2311.09336 12 EARLY EVIDENCE OF HOW LLM S OUTP...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.