{"id":"1711c534-ae23-4de7-8a58-2c1d7ce705e5","arxiv_id":"2502.01205","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Open-weight LLMs reduce character error rates in historical English OCR by 7 to 39 percent, but none reach practical quality for historical Finnish.","lead":"This paper tests several open-weight and commercial large language models on the task of fixing OCR errors in historical English and Finnish texts, and finds the models help substantially for English but fail to reach useful accuracy for Finnish. It also shows post-processing tricks and segment context matter as much as model choice, which is why the results are useful for anyone planning large-scale digitization projects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8.6% of English pages removed by the alignment filter may include reading-order and column-layout errors that LLMs could correct, so the reported 38.7% CER gain is an upper bound on an easier subset.","rationale":"The reader's weakest_assumption identifies the same filtering/ground-truth concern that I consider most load-bearing. The paper is honest about the limitation, stating that the method cannot recover missing text, but the filter also removes pages with reading-order and column-layout errors, which are not necessarily 'missing' text and might be correctable by an LLM. This makes the English CER% an upper bound on an easier distribution, and the practical claim in the abstract ('promise in reducing CER') should be read with that caveat. I do not see an internal inconsistency or a more serious threat: the normalization choices are applied symmetrically, the overgeneration-removal method is conservative, and the Finnish negative results are large enough to withstand the 6% filtering. The proposed test—re-evaluating on discarded pages—would directly settle whether the concern lands. If it does, the headline English improvement would need to be qualified; if not, the conditional verdict can be upgraded. Therefore I recommend keeping the reader's CONDITIONAL verdict.","tokens_in":11264,"tokens_out":13262,"duration_ms":119871,"concrete_test":"Re-run the main English evaluation on a random sample of the 28,907 discarded pages using the original page-level OCR/GT pairs, the same prompt, and the same overgeneration-removal pipeline, and compute CER% for Llama-3.1-70B; if the mean CER% on this sample is far below 38.7% or negative, the reported improvement is an artifact of filtering, whereas a strongly positive result would mitigate the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the 8.6% of English pages discarded by the 100-character-window alignment filter (Section 3.1) are not the cases where post-correction matters most. The filter removes any page where a 100-character window has fewer than 10% aligned characters; the authors list as typical causes 'very noisy OCR output,' 'misidentified reading order or column layout,' and 'significant length differences.' Two of these—reading-order and column-layout errors—are not cases where text is 'missing' in the sense the Limitations section assumes; an LLM prompted to correct OCR could plausibly reorder or restructure such text. By excluding these pages, the test set is biased toward linearly aligned, relatively clean OCR, which is exactly where character-level correction succeeds. The 38.7% CER improvement (Table 2) is therefore an upper-bound estimate for the filtered subset, and the practical gain on the full ECCO collection could be substantially smaller. The authors acknowledge the issue in Limitations, but the central claim as stated—'notable CER reductions on historical English OCR'—does not carry the caveat that it applies only after removing 8.6% of pages with the most challenging alignment problems. The Finnish negative result is less threatened, since the negative margins are large and only 6% of pages were removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates zero-shot, prompt-based post-correction of OCR errors in historical English (ECCO-TCP, 301,937 pages after filtering) and Finnish (NLF ground truth, 449 pages) using seven LLMs, including open-weight models and GPT-4o as a reference. After filtering page pairs for alignment quality, splitting pages into fixed-length segments, optimizing temperature/top_k/top_p on a development set, and applying an alignment-based overgeneration-removal step, the authors report length-weighted relative CER and WER improvements on a held-out test set. They find that most open-weight models improve English OCR (best: Llama-3.1-70B at 38.7% CER reduction), while all tested open-weight models worsen Finnish OCR, leading to the conclusion that prompt-based post-correction is currently infeasible for historical Finnish with these models. They also study quantization, segment length, and context-aware correction at segment boundaries.","tokens_in":11560,"tokens_out":8492,"duration_ms":88467,"significance":"This is a carefully executed empirical study in an area with conflicting prior results. Strengths include a held-out development set for hyperparameter selection, a separate test set, public datasets and evaluation scripts, and reporting of both CER and WER along with the effect of normalization and overgeneration removal. The negative Finnish results are a practically important caution for digital humanities projects. The main caveat is that the English headline result is measured on a filtered subset of pages, so its applicability to the full ECCO collection needs quantification.","major_comments":[{"comment":"The 100-character-window alignment filter removes 28,907 (8.6%) English pages, and the authors list 'misidentified reading order or column layout' as one of the typical causes. This is not a case of text being missing, so the Limitations statement that the method 'cannot recover text where significant portions are missing' does not cover it; an LLM prompted to correct OCR could in principle reorder or restructure such text. Because the test set is restricted to pages that pass the alignment filter, the reported 38.7% CER improvement for Llama-3.1-70B is measured on a linearly aligned, easier subset, and the practical gain on the full ECCO collection may be substantially lower. Please analyze the discarded pages (e.g., report their CER distribution and the share attributable to each listed cause) and provide a conservative estimate for the full collection, for instance by applying the correction to all pages and falling back to the original OCR for pages where the post-processing alignment fails.","section":"Section 3.1 and Table 2"},{"comment":"The alignment-based overgeneration removal is decisive for the English results, particularly for the Llama family: for Llama-3.1-70B the CER improvement changes from -53.6% without the step to +38.7% with it. The method selects the span between the first and last characters of the model output that align to the original OCR. This can in principle also remove legitimate corrections at the beginning or end of a segment that share no characters with the noisy input. Please provide a manual or automatic check of trimmed outputs (e.g., how often the trimmed span ends inside a word that differs from OCR, and whether results are stable under variations of the gap penalties), so readers can be confident the gains are not an artifact of the trimming rule.","section":"Section 4.3 and Table 3"}],"minor_comments":[{"comment":"Please clarify whether the Finnish development and test segments are page-disjoint; because each Finnish page is split into roughly 10 segments, independent segment sampling can place segments from the same page in both splits. If they are not page-disjoint, a page-level split should be used to avoid leakage.","section":"Section 4"},{"comment":"The caption does not state that CER%/WER% are length-weighted averages of example-wise values clipped at -100%; please add this information to the caption or table notes.","section":"Table 2 caption"},{"comment":"The description of the hyperparameter selection ('the median value of the 10 best runs of every model') is underspecified: it should state whether the median is taken per parameter across the 10 runs and whether any interaction between parameters was considered.","section":"Section 4.4"},{"comment":"The Limitations text says '~10% for English', but Section 3.1 reports 8.6% removed by the alignment filter and 0.4% and 1.7% by earlier filters; please align these numbers for precision.","section":"Limitations"},{"comment":"The sentence 'the gains diminishing past about 200–300 words, but our page-level data does not have long-enough examples to allow us to reach the point where the performance would start consistently decreasing' is hard to parse; please rephrase to distinguish the flattening of gains from a possible later decline.","section":"Section 6"},{"comment":"Given that the main results are based on 200 test segments, consider reporting bootstrap confidence intervals for the CER% values in Table 2 to give a sense of the uncertainty around the reported differences.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the empirical work is solid. The main risk is the alignment filter's effect on the headline English result; the authors should be required to quantify it. The Finnish negative result is robust to the small sample size given the large negative margins. I would not support rejection if the authors address the filter and post-processing sensitivity concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this if you work on OCR post-correction or plan to apply LLMs to historical text. The paper is a careful empirical benchmark, not a new method. Its main finding: in a zero-shot prompt setting, open-weight LLMs cut character error rate on historical English (Llama-3.1-70B at 38.7% CER reduction), while every open-weight model made historical Finnish worse; only GPT-4o improved Finnish, and modestly. The Finnish negative result is the most valuable part—it is a credible, useful warning that success on English does not transfer.\n\nWhat is new: a clean alignment-based method for stripping LLM overgeneration (the \"here is your corrected text\" problem), a systematic look at segment length and boundary context, and a comparison of 4-bit vs fp16 quantization. The evaluation is more disciplined than most in this area: held-out dev set for hyperparameter optimization, held-out test set, both CER and WER, and explicit handling of historical spelling variants (long-s, w/v). The citation pattern is appropriate—they build on Boros et al. and Bourne without overselling. They also say code and data will be released.\n\nSoft spots, in order. First, the preprocessing filter removes 8.6% of English and 6% of Finnish pages because OCR and ground truth do not align well; causes include very noisy OCR, reading-order/column-layout errors, and missing text. The paper acknowledges this and says the method cannot recover missing text. Fair, but the caveat should sit next to the headline 38.7%: that number is measured on the well-aligned subset, and the practical gain on the full ECCO collection could be smaller. This is a real limitation, not a fatal one—the paper is transparent about it. Second, there is no significance testing; with 200-segment test sets, model differences of a few points may not be stable. The negative Finnish margins are large enough that the main conclusion holds. Third, the Finnish ground truth is small (449 pages), so treat the Finnish numbers as indicative. Fourth, I could not verify the release artifacts; they are promised, not checked.\n\nWho benefits: applied researchers and digitization teams deciding whether zero-shot LLM post-correction is worth trying, and people building benchmarks for OCR correction. It deserves a serious referee. Send it out; ask the authors to clarify the filtering effect (e.g., report results on the unfiltered or partially filtered set) and add some confidence intervals.","headline":"Solid, honest empirical benchmark: open-weight LLMs help on historical English OCR but fail on Finnish; the 38.7% English gain comes with a filtering caveat that should be made plainer.","tokens_in":12073,"tokens_out":2836,"would_cite":true,"duration_ms":27949,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt-based LLM post-correction works for historical English but not for historical Finnish with current open-weight models.","keywords":["OCR post-correction","historical documents","large language models","character error rate","zero-shot prompting","Finnish","English","open-weight models"],"falsifier":"Run the same Llama-3.1-70B prompt-based correction on the 28,907 English pages that were filtered out because their OCR and ground truth aligned poorly, and compare CER% on those pages to the 38.7% reported on the kept test set; if the excluded pages show no improvement or negative improvement, the method's practical value on uncurated collections is much smaller than the headline result.","tokens_in":11096,"feed_emoji":"📜","tokens_out":8895,"duration_ms":81998,"temperature":0.7,"pith_summary":"The paper asks whether a direct, zero-shot instruction to an LLM—'correct this OCR text'—can repair noisy transcriptions of historical documents without retraining. For eighteenth-century English, the answer is yes for several open-weight models, with the best tested model, Llama-3.1-70B, achieving a 38.7% relative reduction in character error rate. For historical Finnish, every open-weight model tested made the text worse, so the authors conclude that prompt-based post-correction is currently infeasible for Finnish with open models. The result matters because large historical collections contain millions of pages of OCR noise, and a prompt-only fix would be far cheaper than re-OCRing or fine-tuning per language. The paper also shows that two evaluation details—stripping model-added commentary and normalizing historical spellings—have a very large effect on measured performance.","feed_headline":"Open LLMs fix English OCR but worsen Finnish","feed_subtitle":"Zero-shot Llama-3.1-70B cut character errors 38.7% in English; all open-weight models degraded Finnish.","key_machinery":"The load-bearing mechanism is a zero-shot prompting pipeline in which the model is asked to correct a segment of OCR text, and the raw output is then post-processed by local sequence alignment against the original OCR input: leading and trailing text that does not align well—such as 'Here is the corrected text:' and appended explanations—is trimmed away. Performance is measured as relative character error rate reduction, $\\text{CER\\%} = (\\text{CER}_{\\text{orig}} - \\text{CER}_{\\text{post}})/\\text{CER}_{\\text{orig}} \\times 100$, clipped at $-100\\%$ and weighted by OCR character count, with WER as a secondary metric. Before scoring, the text is normalized with Unicode NFKC and, for Finnish, w is replaced by v, so that historical spelling differences do not masquerade as OCR errors. For the Llama models this overgeneration filter is decisive: without it their English CER% would be strongly negative, whereas with it they show positive gains.","core_discovery":"The paper's central claim is that in a zero-shot, prompt-based setting, open-weight LLMs can deliver substantial OCR post-correction for historical English but not for historical Finnish. On a 200-segment test set drawn from English ECCO-TCP OCR-ground-truth pairs, Llama-3.1-70B-Instruct reaches 38.7% relative CER improvement and all open models except Mixtral improve CER, while GPT-4o reaches 58.1%. On 449 pages of Finnish newspaper ground truth, all open models produce negative CER improvement, meaning their output is on average farther from the ground truth than the original OCR; GPT-4o is the only tested model with a positive Finnish result, at 11.9% CER improvement. The authors conclude that prompt-based post-correction is not practically usable for Finnish with current open-weight models, and they attribute much of the difficulty to the absence of Finnish support in the open models. They further show that removing model overgeneration with string alignment is necessary for the Llama family, and that normalizing historical spelling variants accounts for a large share of the apparent gains.","pith_inferences":["Because 8.6% of English pages and 6% of Finnish pages were discarded for poor OCR-to-ground-truth alignment, the reported CER improvements describe a curated subset; on a real uncurated collection, where missing or badly garbled text is common, the practical gain could be substantially smaller, as the authors acknowledge in their limitations section.","The Finnish failure may be primarily a language-support gap rather than a fundamental limit of LLMs, since none of the open models officially support Finnish; testing a Finnish-competent open model or a fine-tuned Finnish model is the obvious way to separate these explanations.","Part of the measured 'correction' is actually modernization of historical orthography, not repair of OCR damage; users who need to preserve original spellings would see a smaller benefit than the reported CER% suggests.","A multimodal extension that feeds the original page image together with the OCR text could address the missing-text cases the paper explicitly sets aside, because the current pipeline can only rewrite what the OCR already produced."],"forward_implications":["A large-scale English correction run could use Llama-3.1-70B at 4-bit quantization, which fits on two GPUs and preserves most of the quality gain, to improve the ECCO collection's text.","Any planned open-weight correction of historical Finnish collections should not be run as zero-shot prompting; fine-tuning or a Finnish-capable model is required, matching the paper's stated next step.","Evaluation results across OCR post-correction studies are not directly comparable unless they apply the same spelling normalization and overgeneration handling, since these steps can swing scores by tens of percentage points.","For long documents, segments of roughly 200 to 300 sub-words are preferable to shorter segments, and providing the preceding segment as context helps the largest models correct text near segment boundaries."],"supporting_citations":[{"why":"Earlier zero-shot LLM study on multilingual historical transcripts with mostly negative results; this paper's English positive result is positioned against it.","marker":"Boros et al., 2024"},{"why":"Earlier prompt-based study reporting large CER reductions on historical English; this paper's English result aligns with it and extends to open-weight models.","marker":"Bourne, 2024"},{"why":"Supplies the page-level paired ECCO OCR and ECCO-TCP ground-truth data that the English evaluation is built on.","marker":"Hill and Hengchen, 2019"},{"why":"Provides the Finnish historical newspaper ground-truth sample that the Finnish evaluation uses.","marker":"Kettunen et al., 2018"},{"why":"Documents the same Finnish ground-truth data set and its intended use for OCR quality evaluation, supporting the Finnish test setup.","marker":"Kettunen et al., 2020"},{"why":"Describes the Llama-3.1 model family, the source of the best-performing open-weight model in the study.","marker":"AI@Meta, 2024b"},{"why":"Introduces Mixtral-8x7B, one of the evaluated open-weight models and the one with the worst English CER result.","marker":"Jiang et al., 2024"},{"why":"Introduces the Gemma-2 models, which serve as the other open-weight baselines in the evaluation.","marker":"Mesnard et al., 2024"},{"why":"Defines the ICDAR 2019 post-OCR correction benchmark whose CER/WER evaluation conventions this paper follows.","marker":"Rigaud et al., 2019"}],"fun_headline_variants":["Open LLMs cut English OCR errors, worsen Finnish","English OCR gains, Finnish loses with open LLMs","No free lunch: Open LLMs fail Finnish OCR","Open model OCR: English improves, Finnish degrades","Zero-shot LLMs: English OCR better, Finnish worse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the manually corrected ground truth is reliable and that discarding pages with poor OCR-to-ground-truth alignment—about 9% of English pages and 6% of Finnish pages—does not remove the hardest cases, so the reported gains may shrink on real, unfiltered collections.","fun_headline_variants_meta":{"raw":{"variants":["Open LLMs cut English OCR errors, worsen Finnish","English OCR gains, Finnish loses with open LLMs","No free lunch: Open LLMs fail Finnish OCR","Open model OCR: English improves, Finnish degrades","Zero-shot LLMs: English OCR better, Finnish worse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1204,"prompt_tokens":874,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":253}},"tokens_in":490,"tokens_out":330,"duration_ms":3557,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:12:14.718292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Llama-3.1-70B prompt-based correction on the 28,907 English pages that were filtered out because their OCR and ground truth aligned poorly, and compare CER% on those pages to the 38.7% reported on the kept test set; if the excluded pages show no improvement or negative improvement, the method's practical value on uncurated collections is much smaller than the headline result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier zero-shot LLM study on multilingual historical transcripts with mostly negative results; this paper's English positive result is positioned against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Finnish historical newspaper ground-truth sample that the Finnish evaluation uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the same Finnish ground-truth data set and its intended use for OCR quality evaluation, supporting the Finnish test setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Gemma-2 models, which serve as the other open-weight baselines in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ICDAR 2019 post-OCR correction benchmark whose CER/WER evaluation conventions this paper follows."}],"review_version":1}