{"id":"ce7daccf-76b9-4b85-aab7-3bfa55d1d4dd","arxiv_id":"2501.11623","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using 20 scanned pages of 1921 Belgian handwritten tables, the authors report that GPT-4o and Claude Sonnet 3.5 transcribe more accurately than EasyOCR, Keras, Pytesseract, and TrOCR, with two-shot prompting giving the best results.","lead":"This paper compares two large language models (GPT-4o and Claude Sonnet 3.5) with traditional OCR/HTR tools on transcriptions of handwritten 1921 Belgian tax records. The authors find that the LLMs, especially with two-shot prompting, produce transcriptions closer to human-corrected ground truth than the traditional tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states whether the one/two-shot example pages come from the same 20-page evaluation set; if they do, the two-shot best-strategy claim is circular and the headline conclusion is unsupported.","rationale":"The reader identified the missing train/test documentation as the weakest assumption, and the same concern is the most load-bearing for the paper's strongest claim. The central claim has two parts: (1) LLMs outperform conventional OCR/HTR tools, and (2) the best strategy is two-shot GPT-4o for line-by-line and two-shot Claude for whole-scan. Part (1) is supported by zero-shot results and is unlikely to be overturned by example leakage, since the OCR/HTR baselines fail even on zero-shot comparisons. Part (2), however, depends entirely on the one/two-shot conditions. If the example pages are drawn from the same 20 pages being scored, the two-shot output can be copied from the prompt, making the reported CER/BLEU superiority circular. The paper's language in the conclusion ('as few as two pieces of ground truth data') makes this possibility concrete rather than hypothetical. The authors do not state whether examples are held out, and the appendix does not list example page IDs. Therefore, the specific best-strategy conclusion cannot be verified as presented. The reader's CONDITIONAL verdict remains appropriate: the paper should not be accepted without either a clear statement of the example split or a re-run with held-out examples. The concern is not an accusation of intentional leakage; it is an unresolved methodological ambiguity that determines whether the headline result is evidence or an artifact.","tokens_in":12327,"tokens_out":5655,"duration_ms":64017,"concrete_test":"Ask the authors to release, or inspect in the supplementary material, the page identifiers of the one/two example images used for each of the 20 evaluated scans. Then re-run the two-shot conditions with example pages explicitly held out (e.g., use pages 1–2 as the example set and evaluate only pages 3–20, or use a separate year's scans), and recompute the line-by-line and whole-scan CER/BLEU. If the two-shot scores drop to near the zero-shot scores, or the best model flips from GPT-4o to Claude, the headline two-shot claim is an artifact of test-set leakage. If the scores are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 describes one-shot and two-shot strategies only as 'providing the model with one or two example images and transcriptions,' and Table 2's one-/two-shot prompt fills the placeholder {example(s)} without saying where the example images come from. The dataset (§3.1) consists of exactly 20 Déclaration de Succession pages, and the conclusion (§6) states that 'as few as two pieces of ground truth data' suffice. If the example images used for the two-shot conditions are pages (or line crops) from these same 20 pages that are then evaluated, the model is effectively given ground-truth transcriptions in the prompt for pages it is asked to transcribe; the reported CER/BLEU advantage of two-shot GPT-4o (line) and two-shot Claude (whole scan) would be an artifact of answer leakage, not evidence about few-shot transcription ability. The zero-shot results would still support the qualitative claim that LLMs beat the untuned OCR/HTR baselines, but the specific best-strategy claim in the abstract and Section 6 would collapse. Because the split is not documented and the supplementary material describing example selection is not included in the text, the result is currently unverifiable. This is the weakest load-bearing link in the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two multimodal LLMs (GPT-4o and Claude Sonnet 3.5) with four conventional OCR/HTR tools (EasyOCR, KerasOCR, Pytesseract, TrOCR) on transcription of 20 scanned pages of a 1921 Belgian Déclaration de Succession. Experiments are run both line-by-line and on whole scans, with five prompting strategies for the LLMs and additional fine-tuned TrOCR variants. Performance is measured by CER and BLEU, supplemented by a human evaluation of whole-scan outputs. The paper claims that LLMs outperform the OCR/HTR baselines overall and that, among the tested strategies, two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole-scan images produce transcriptions closest to ground truth.","tokens_in":12602,"tokens_out":4525,"duration_ms":46733,"significance":"If the central claim holds, this is a useful early data point showing that a single multimodal LLM call can rival or beat traditional multi-stage OCR/HTR pipelines on a narrow tabular historical document, and that two ground-truth examples may suffice for strong few-shot performance. The paper has several strengths: it reports concrete prompts, compares multiple baselines, includes a human evaluation, and discusses metric limitations such as BLEU's sensitivity to n-gram order and CER's sensitivity to whitespace. The main significance is therefore empirical rather than methodological. However, the headline few-shot result rests on an unverifiable assumption about the origin of the example images, and the reported results would be substantially weakened if the examples come from the same pages being scored.","major_comments":[{"comment":"The one-shot and two-shot conditions are described only as 'providing the model with one or two example images and transcriptions' (Section 3.3), and Table 2's prompt inserts {example(s)} without stating whether those examples come from the same 20 pages being evaluated. Since Section 3.1 defines the dataset as exactly 20 scanned pages and the conclusion (Section 6) states that 'as few as two pieces of ground truth data' suffice, the few-shot examples may be drawn from the evaluation set. If so, the model receives ground-truth transcriptions for pages it is then asked to score, making the two-shot advantage in the abstract and Section 6 an artifact of leakage. The authors must document the provenance of the example images and, if any overlap exists, rerun the few-shot experiments with a held-out split. The zero-shot results would still be informative, but the headline best-strategy claim is currently unverifiable.","section":"Section 3.3 and Table 2"},{"comment":"The maximum n-gram order for BLEU differs between the two experimental settings: 3 for line-by-line and 4 for whole-scan (Section 4). As Figure 6 shows, the n-gram order materially changes BLEU magnitude. This makes the cross-setting comparison in Section 4.1 and the conclusion that 'LLMs seem to perform better when the document images are sliced per row' unsupported, because the two arms are not evaluated with the same metric. Use a common n-gram order when comparing line-by-line to whole-scan, or restrict this conclusion to within-setting comparisons.","section":"Section 4 and Appendix Fig. 6"},{"comment":"The text states that 'some anomalies in the outputs are manually treated to ensure a fair and accurate comparison to the GT data (see details in the supplementary material)', but the supplementary material is not included in the manuscript. Because this manual post-processing is applied after seeing the outputs and could favor one method over another, the procedure must be specified completely: which criteria define an anomaly, which outputs were edited, and what edits were made. Without this, the reported CER/BLEU numbers are not reproducible.","section":"Section 3.3 / supplementary material"},{"comment":"No statistical tests or confidence intervals accompany the headline comparisons. With 20 whole-scan samples (and 283 lines, many of which share layout and header text), the differences between the best few-shot strategies and the runners-up in Figures 1, 3, and Table 1 may be within sampling variability. At minimum, report paired tests (e.g., Wilcoxon signed-rank) for the main CER/BLEU comparisons and for the human-evaluation ranking. This is especially important because the best strategy was selected post hoc after inspecting the results.","section":"Section 4 and Table 1"}],"minor_comments":[{"comment":"The refine prompt contains a typo: 'trasncription' should be 'transcription'.","section":"Table 2 / Refine prompt"},{"comment":"The human evaluation was conducted by the two lead authors, who also created the ground truth, but no inter-rater reliability measure is reported; please state this limitation explicitly or report agreement statistics.","section":"Section 4.2 and Table 1"},{"comment":"The practice of rerunning documents or rows that return error messages may introduce selection bias; report how often this occurred and whether the rerun outputs were used without averaging.","section":"Section 3.3"},{"comment":"The conclusion generalizes to 'LLMs' and 'conventional OCR/HTR tools' from two LLMs and four baselines; consider qualifying the claim to the specific models and document type tested.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The example-leakage concern is the key load-bearing issue and is fixable: the authors should either document that few-shot examples came from a separate development set or rerun with a proper split. If the examples are drawn from the evaluation set, the abstract's two-shot claims must be withdrawn. The paper is otherwise within scope and the zero-shot comparison appears useful, but the current manuscript does not yet support the headline conclusion as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gives a useful early comparison, but the headline two-shot result may rest on answer leakage. Section 3.3 says one- and two-shot strategies provide the model with example images and transcriptions, but never where those examples come from. If they are drawn from the same 20 pages being scored, the two-shot advantage is circular, and the conclusion about \"as few as two pieces of ground truth data\" collapses. The zero-shot results likely still support the qualitative claim that LLMs beat the untuned baselines; that part is probably robust.\n\nWhat is actually new: a systematic comparison of five prompting strategies (simple, complex, one-shot, two-shot, refine) on a specific historical record type (1921 Belgian succession declarations), with both line-by-line and whole-scan inputs, plus several OCR/HTR baselines including fine-tuned TrOCR. That is a reasonable extension of earlier GPT-4V OCR work. The ground truth construction is careful, with cross-checking against archives and genealogy databases. The BLEU vs. CER analysis is thoughtful, and the author-run human evaluation, though biased, gives a useful sanity check.\n\nThe soft spots, in proportion: the missing split is the big one. It is not a stylistic omission; the main two-shot conclusion depends on it. Second, only 20 pages, so all conclusions are early evidence, as the title says. Third, manual post-hoc anomaly handling is described only by reference to a supplementary that is not in the text, which hurts reproducibility. Fourth, there are no statistical tests; the box plots suggest overlapping distributions, so \"outperform\" needs paired tests or error bars. Fifth, the BLEU n-gram order differs (3 vs. 4) between experiments; this is justified, but it complicates direct comparison.\n\nThis paper is for people working on digitizing historical documents and anyone evaluating LLMs on OCR/HTR tasks. It deserves a serious referee, but only after the authors document example provenance and ideally release data and code. For a reading group, it is a good case study in why evaluation splits matter in LLM experiments.","headline":"Useful early comparison between LLMs and OCR/HTR tools, but the headline two-shot result is unverifiable without a documented train/test split for the example images.","tokens_in":13107,"tokens_out":2694,"would_cite":false,"duration_ms":29759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On 20 pages of 1921 Belgian handwritten inheritance declarations, two-shot GPT-4o and two-shot Claude Sonnet 3.5 produce transcriptions closer to human ground truth than the conventional OCR/HTR tools tested, including fine-tuned TrOCR.","keywords":["Large Language Models","Historical Documents","Handwritten Text Recognition","Optical Character Recognition","Character Error Rate","BLEU score","Few-shot prompting","Tabular records"],"falsifier":"Inspect the example images used in the one- and two-shot prompts and compare their provenance against the 20 evaluated pages; if any example page is also a scored page, the few-shot result is invalidated. Alternatively, rerun the two best configurations with examples taken only from a separate held-out set of similar documents and check whether the CER and BLEU advantage over fine-tuned TrOCR survives.","tokens_in":12131,"feed_emoji":"📜","tokens_out":8845,"duration_ms":77051,"temperature":0.7,"pith_summary":"This paper asks whether a single large language model can replace the conventional multi-stage pipeline of preprocessing, layout segmentation, and text recognition for digitising historical handwritten tables. On 20 pages of 1921 Belgian inheritance declarations, the authors compare GPT-4o and Claude Sonnet 3.5 against EasyOCR, KerasOCR, Pytesseract, and TrOCR, including fine-tuned variants. They report that both LLMs transcribe the records far more accurately, and that giving the model two example image-transcription pairs in the prompt produces the best results: two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole scans. If these results hold beyond the test set, historians and archivists could obtain high-quality transcriptions with almost no ground truth data and no per-collection fine-tuning.","feed_headline":"With two examples, LLMs beat traditional OCR on historical records","feed_subtitle":"GPT-4o and Claude Sonnet 3.5 outperform fine-tuned TrOCR on 1921 handwritten tables.","key_machinery":"The central mechanism is few-shot vision-language prompting: the scanned document image is handed directly to a multimodal LLM together with one or two previous image-transcription pairs that teach the layout and handwriting style. This replaces the classical OCR/HTR chain—contrast adjustment, layout analysis, line and word segmentation, per-region recognition, and fine-tuning—with a single model call, and the paper credits this replacement for the accuracy jump over the conventional tools.","core_discovery":"The paper's central claim is that, for tabular handwritten historical records, a multimodal LLM prompted with one or two example transcriptions outperforms established OCR/HTR tools, and does so without any fine-tuning or layout pre-processing. The authors demonstrate this on Belgian probation records from 1921, scoring outputs with Character Error Rate and BLEU; they find that zero-shot LLM outputs already surpass fine-tuned TrOCR, and that adding two example image-transcription pairs yields the best transcriptions overall. They also show that the two automatic metrics disagree with human judgment on whole-scan outputs, mainly because the repetitive typed header dominates CER/BLEU, and that BLEU distinguishes method quality better than CER for this task. The conclusion is that a one-step LLM approach is a viable alternative to the traditional pipeline, needing as few as two pieces of ground truth data.","pith_inferences":["The authors never test whether the two-shot examples transfer across handwriting styles, years, or archives; based on the reported mechanism, the recipe would be example pages that resemble the target collection, so the practical protocol should include a small similarity check.","A hybrid pipeline is a plausible extension the paper does not run: LLM transcription for handwritten text columns plus a classical OCR pass for numeric columns, given that both LLMs struggled with digits.","Because the paper shows the repetitive header inflates CER/BLEU and distorts ranking, a reweighted metric that ignores known-stable regions would likely change which method looks best in future studies.","The observed day-to-day drift in LLM behavior suggests that reproducibility, not raw accuracy, will be the next bottleneck for applying LLM transcription at scale."],"forward_implications":["If the two-shot results generalize, a new historical collection can be transcribed after hand-curating just two example rows, instead of building a fine-tuning set of hundreds of lines.","The finding that BLEU separates good from bad transcriptions better than CER on long documents suggests future OCR/HTR benchmarks should not rely on CER alone.","Because the LLMs reproduce table structure in a single pass, downstream analytics can consume structured output directly, without layout-reconstruction post-processing.","Fine-tuned TrOCR with half the data still underperformed the best zero- and few-shot LLM outputs, indicating that per-collection labeled data may no longer be necessary for this document class.","Whole-scan and line-by-line inputs favor different models, so choosing the input framing is itself part of the transcription strategy."],"supporting_citations":[{"why":"Shows that among LLMs tested for post-OCR correction, GPT-4 performed best, motivating the choice of GPT-4o as the comparison model.","marker":"[1]"},{"why":"Introduces DTrOCR, a decoder-only transformer for OCR, representing the modern transformer-based approach the LLMs are compared against.","marker":"[3]"},{"why":"Explores GPT-4V as an OCR engine on typed texts but not handwriting or prompting strategies, defining the gap this paper fills.","marker":"[4]"},{"why":"Provides the implementation of CER used to compute the paper's primary accuracy scores.","marker":"[5]"},{"why":"Defines the BLEU metric used as the second evaluation score for comparing transcription output to ground truth.","marker":"[8]"},{"why":"Source of the Character Error Rate metric, the other core evaluation score in the study.","marker":"[9]"},{"why":"Early quantitative evaluation of GPT-4V(ision) OCR/HTR capabilities that did not explore few-shot or refinement strategies; this paper directly extends it.","marker":"[10]"}],"fun_headline_variants":["Two-shot LLMs outdo traditional OCR on handwritten tables","LLMs beat fine-tuned OCR on 1921 records with just 2 examples","No fine-tuning needed: LLMs top OCR on historical documents","GPT-4o & Claude surpass TrOCR on handwritten historical tables","Historical OCR: LLMs win with minimal prompting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study's conclusion depends on the one- and two-shot example transcriptions not being drawn from the same 20 pages being scored; the paper does not state where these examples come from, so if any overlap the test set, the reported few-shot advantage would be partly circular.","fun_headline_variants_meta":{"raw":{"variants":["Two-shot LLMs outdo traditional OCR on handwritten tables","LLMs beat fine-tuned OCR on 1921 records with just 2 examples","No fine-tuning needed: LLMs top OCR on historical documents","GPT-4o & Claude surpass TrOCR on handwritten historical tables","Historical OCR: LLMs win with minimal prompting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2770,"prompt_tokens":906,"completion_tokens":1864,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1776}},"tokens_in":522,"tokens_out":1864,"duration_ms":11211,"temperature":1.0,"reasoning_tokens":1776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:01:46.411683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the example images used in the one- and two-shot prompts and compare their provenance against the 20 evaluated pages; if any example page is also a scored page, the few-shot result is invalidated. Alternatively, rerun the two best configurations with examples taken only from a separate held-out set of similar documents and check whether the CER and BLEU advantage over fine-tuned TrOCR survives.","supporting_citations":[{"cited_title":"I n: Bizzoni, Y ., Degaetano-Ortlieb, S., Kazantseva, A., Szpakowicz, S","cited_arxiv_id":null,"evidence_quote":"Shows that among LLMs tested for post-OCR correction, GPT-4 performed best, motivating the choice of GPT-4o as the comparison model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces DTrOCR, a decoder-only transformer for OCR, representing the modern transformer-based approach the LLMs are compared against."},{"cited_title":"In: International Conference on Theory and Practice of Digital Libraries","cited_arxiv_id":null,"evidence_quote":"Explores GPT-4V as an OCR engine on typed texts but not handwriting or prompting strategies, defining the gap this paper fills."},{"cited_title":"https://github.com/huggingface/evaluate (2024), https://github.com/huggingface/evaluate, version 0.4.3, Apache- 2.0 License","cited_arxiv_id":null,"evidence_quote":"Provides the implementation of CER used to compute the paper's primary accuracy scores."},{"cited_title":"In: Proceedings of the 40th annual meeting of the Association fo r Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Defines the BLEU metric used as the second evaluation score for comparing transcription output to ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Character Error Rate metric, the other core evaluation score in the study."}],"review_version":1}