Comparing OCR Pipelines for Folkloristic Text Digitization

Alina L. Machidon; Octavian M. Machidon

REVIEW 4 major objections 5 minor 12 references

Comparing OCR Pipelines for Folkloristic Text Digitization

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Comparing two OCR approaches on three Slovene folkloristic corpora, the paper concludes that no single pipeline is universally optimal: olmOCR suits clean typewritten fairy tales, while Tesseract plus LLM post-processing suits historical…

desk verdict A useful pilot comparison of OCR pipelines on real Slovene folklore corpora, but the per-corpus recommendations outrun the evidence because the evaluation is entirely qualitative. read the letter →

arxiv 2507.19092 v1 pith:CTFHEJQT submitted 2025-07-25 cs.DL cs.MM

Octavian M. Machidon , Alina L. Machidon This is my paper · ORCID

classification cs.DLcs.MM

keywords opticalcharacterrecognitionOCRlargelanguagemodelspost-OCRcorrectionhistoricaldocumentsSlovenefolkloristictextslayoutanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares two ways of turning scanned Slovene folkloristic documents into digital text: a single-stage tool called olmOCR and a two-stage pipeline that combines Tesseract OCR with large-language-model post-processing. It argues that neither approach is best everywhere; the winning choice depends on scan quality, layout complexity, and linguistic register. For clean typewritten fairy tales, olmOCR transcribes more faithfully, while for a 19th-century newspaper and a children's magazine, Tesseract followed by an LLM produces more readable results, with layout parsing recommended for the magazine. The paper warns that LLM post-processing can quietly replace dialectal or archaic words with modern equivalents, trading linguistic authenticity for readability. The practical upshot is a document-sensitive selection strategy for folklore digitization.

What carries the argument

The comparison runs on two pipelines. olmOCR is a locally deployed vision-language model that converts scans directly into plain text; Tesseract is a conventional OCR engine whose raw output is passed to a large language model for error correction and paragraph reconstruction, with LayoutParser optionally used first to detect and segment text regions. The decisive mechanism is the interaction between these pipelines and document type: clean uniform pages favor direct recognition, while noisy or complex pages benefit from LLM repair at the cost of normalizing dialectal or archaic language.

What would settle it

A quantitative re-run of the same three corpora against manually verified ground truth, measuring character error rate and the rate at which dialectal words are changed, would settle the claim: if Tesseract plus LLM post-processing matches or beats olmOCR on typewritten fairy tales, or if olmOCR beats Tesseract plus LLM post-processing on Ciciban, the document-sensitive ranking would fail.

Watch

Extended reading notes

Core claim

Across three Slovene corpora of different eras and layouts, the paper finds that no OCR pipeline is universally optimal. olmOCR outperforms Tesseract plus LLM post-processing on mid-20th-century typewritten fairy tales, preserving sentence structure and diacritics with fewer hallucinations. Tesseract plus LLM post-processing wins on a 19th-century newspaper and on a children's magazine, where the LLM restores readable, coherent text but occasionally modernizes historical terms. The determining variables are scan quality, layout complexity, and linguistic register, and the paper recommends a document-sensitive pipeline choice rather than a single tool.

Load-bearing premise

The entire ranking rests on the authors' manual, informal reading of a small set of outputs, with no reported error rates, sample sizes, or second reviewers, so the conclusions stand only if those subjective judgments are an accurate measure of transcription quality.

Editorial extensions

If this is right

Digitization projects on Slovene folklore should not commit to a single OCR tool; the choice should follow the document's scan quality, layout, and linguistic register.
On clean, uniformly typewritten pages, a single-stage vision-language OCR such as olmOCR preserves dialectal forms with fewer hallucinations and can be run locally for privacy.
On complex layouts such as children's magazines with mixed poetry, prose, and images, Tesseract plus an LLM post-processing step, with layout parsing when needed, produces the most readable and coherent text.
LLM post-processing should be used with human validation or constrained prompts, because it can replace dialectal or archaic words with modern equivalents.
Scan quality is the dominant limiting factor: conventional preprocessing such as grayscale conversion, binarization, and dilation does not reliably salvage low-quality scans.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical extension of the paper's comparison is that archives should store both the raw OCR output and the LLM-enhanced version, so linguistic authenticity is never destroyed by normalizing post-processing.
The observed trade-off suggests a testable hypothesis for neighbouring languages: dialectal or archaic spellings will be the first casualties of LLM post-correction in any historical corpus, not just Slovene.
The authors' qualitative judgments could be turned into a quantitative benchmark by measuring character error rate and dialect-term preservation on the same corpora, allowing archives to set explicit thresholds for when to prefer fidelity over readability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a readable, honest applied comparison of olmOCR against Tesseract+LLM on three real Slovene folklore collections. What is genuinely new is the corpus work: nobody had reported these pipeline behaviors on Kmetijske in rokodelske novice, Ciciban, and the typewritten fairy-tale collection. The paper also does a good job of situating itself, and the warning about LLM post-processing normalizing dialectal words (storjice to zgodbe) is a real caution for heritage practitioners.

The soft spot is the one the stress-test names, and it is in proportion: the central rankings in Table 1 rest on unquantified manual review. No page counts, error rates, sample sizes, prompts, or model versions for the LLM calls are reported. Figures 5-7 are selected examples, and the evaluators knew which pipeline produced each output, so confirmation bias is a live concern. There is also a small but telling inconsistency: Section 4.2.1 says ChatGPT changed žlahnega to glavnega, but Figure 5 shows glahnega in the highlighted output. That makes the illustrative example harder to trust.

The deeper issue is that the pipeline recommended for Kmetijske (Tesseract+LLM) is the one whose own example shows semantic drift. So 'preferred' is doing a lot of work: preferred for readability, not for linguistic fidelity. The general conclusion that no single pipeline is universally optimal is almost certainly true, but it is not the claim that needs support; the specific per-corpus preferences are underdetermined by the published evidence.

Who gets value: digital heritage practitioners who want a pilot-level orientation before choosing OCR workflows. Methodologically oriented readers will be frustrated by the missing metrics. With revision—reporting what was actually reviewed, adding quantitative evaluation or at least a clear limitation statement, and fixing the example mismatch—this could be a solid practice paper.

My recommendation: send it to peer review with the expectation of heavy revision. The question matters and the authors did real work on real archives, so referee time is not wasted. But it should not be accepted as-is, and the authors should be pushed to either add error metrics or scale the conclusions back.

Referee Report

4 major / 5 minor

Summary. The paper compares two OCR pipelines for digitizing Slovene folkloristic and historical materials: a single-stage olmOCR pipeline and a two-stage Tesseract-plus-LLM pipeline, with optional LayoutParser for complex layouts. The evaluation covers three corpora: a 19th-century newspaper (Kmetijske in rokodelske novice), a mid-20th-century children's magazine (Ciciban), and typewritten fairy tales from the Institute of Folklore Studies. Based on qualitative manual review of OCR outputs, the paper concludes that no single pipeline is universally optimal and recommends Tesseract+LLM for the newspaper and children's magazine, and olmOCR for clean typewritten fairy tales, while cautioning about LLM-induced semantic drift and normalization of dialectal terms.

Significance. If the conclusions were supported by rigorous evidence, the paper would provide practically useful guidance for digital-heritage practitioners working with historical Slavic-language materials, and its emphasis on the trade-off between readability and linguistic authenticity is a worthwhile contribution. The authors also deserve credit for testing on authentic, publicly available corpora rather than synthetic data, and for explicitly discussing privacy and local deployment advantages of olmOCR. However, the central comparative claims currently rest entirely on manual review with no quantitative metrics, no sample sizes, no gold-standard comparison, and no inter-annotator reliability check, so the specific recommendations in Table 1 are not yet established by the published evidence.

major comments (4)

[§3.2 (Output Evaluation), §4.3, Table 1] The per-corpus rankings in Table 1 and the central conclusion in Section 5 rest on the manual visual review described in Section 3.2, but the paper reports no number of pages evaluated per corpus, no character error rates or word error rates, no comparison against a ground-truth transcription, and no inter-annotator agreement statistic. Because the ranking claims are load-bearing, this lack of quantitative support is a major issue. For the revision, the authors should report the size of each evaluation sample, compute standard error metrics against a manually verified transcription for at least a random subset of each corpus, and have at least two independent reviewers score outputs on the stated criteria, with agreement reported.
[§4.2.1, Figure 5] Section 4.2.1 states that ChatGPT changed 'žlahnega' to 'glavnega', but the highlighted enhanced output in Figure 5 contains 'glahnega' rather than 'glavnega'. This is a concrete inconsistency in the key example used to illustrate semantic drift. The text or the figure must be corrected, and all highlighted tokens in Figure 5 should be verified against the actual model outputs before the example is used as evidence.
[§4.1.2, §4.2.1, Table 1] The paper acknowledges that the Tesseract+LLM pipeline introduced 'storjice → zgodbe' normalization and 'semantic drift' on Kmetijske in rokodelske novice, yet Table 1 still recommends this pipeline for that corpus. Since Section 3.2 lists 'language integrity' as an evaluation criterion, the trade-off between readability and authenticity is never made explicit. The authors should define a scoring rubric that includes authenticity preservation and show that the recommended pipeline wins on that rubric; otherwise the recommendation is underdetermined by the stated criteria.
[§3.2 (Pipeline B), §4.1.2] The LLM post-processing stage is not described reproducibly: the paper says 'ChatGPT-4 (GPT-4.o version model)' and mentions 'other LLMs (such as Gemini and Claude)' in Section 4.1.2, but it never provides the exact model identifiers, the prompts used, decoding parameters, or any results separated by model. This makes the experiments impossible to reproduce and leaves unclear whether the findings are specific to one model configuration or general across LLMs. The revision should include the full prompts, model versions and access dates, and per-model results.

minor comments (5)

[§4.3, bullet list] The phrase 'the use olmOCR' is missing 'of'; it should read 'the use of olmOCR'.
[§3.1.3] The phrase 'collected by theInštitut za narodopisje' is missing a space after 'the'.
[Figure 5 caption] The caption says 'Top: original scan. Middle: raw OCR output using Tesseract. Bottom: ChatGPT-4 enhanced output', but the visible figure appears to contain two text paragraphs rather than three clearly separated panels; the layout should be clarified, and the yellow highlights may not be visible in grayscale print.
[§3.2 (Pipeline A)] The cost estimate 'under $190 per million pages' lacks a price date and a specification of the hardware or API assumptions; please add a reference or a calculation footnote.
[Table 1] The corpus name is spelled 'Kmetijske in Rokodelske novice' in Table 1 but 'Kmetijske in rokodelske novice' elsewhere; please standardize the capitalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical OCR comparison whose conclusions are inductive summaries of manual observations, not derivations from their own inputs.

full rationale

The paper contains no derivation chain, fitted parameters, or mathematical reduction. Its central claims — that no single OCR pipeline is universally optimal and that per-corpus preferences differ (Table 1) — are empirical conclusions drawn from manual review of OCR outputs described in Section 3.2 ('Output Evaluation'). The conclusion in Section 5 ('We conclude that no single OCR pipeline is universally optimal') is a generalization of the reported observations, not a quantity predicted from the same data by construction. There are no equations equating an output to an input, no parameter fitted to a subset and then called a prediction, and no load-bearing self-citation: the cited works are external and used for background or tool description, not to justify the paper's comparative result. The weaknesses identified by a skeptical reading — small sample sizes, lack of inter-rater agreement, possible confirmation bias, and an inconsistency between Section 4.2.1 ('glavnega') and Figure 5 ('glahnega') — concern the reliability and evidence quality of the evaluation, not circularity. Under the stated rules, unsupported or underpowered evidence is a correctness risk, not a circularity finding. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters and no invented entities. Its load is borne by unquantified manual evaluation and unspecified sampling, which are captured as domain assumptions.

assumptions (3)

domain assumption Manual review is a sufficiently reliable measure of OCR accuracy and linguistic authenticity.
Section 3.2 'Output Evaluation': the entire comparison rests on subjective manual review with no quantitative metrics or inter-rater checks.
domain assumption The selected sample pages are representative of each corpus.
Section 3.1 describes the corpora generally but gives no page counts, sampling strategy, or selection criteria, so the claims assume representativeness without evidence.
ad hoc to paper The LLM prompts and model versions are fixed and consistent across all experiments.
Prompt engineering is mentioned but prompts are not disclosed; the paper assumes consistency sufficient for comparison, though no verification is possible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Comparing OCR Pipelines for Folkloristic Text Digitization." pith.science (2026). https://pith.science/paper/CTFHEJQT

@misc{pith2026250719092,
  author       = {Pith},
  title        = {Pith review of: Comparing OCR Pipelines for Folkloristic Text Digitization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CTFHEJQT}},
  note         = {Machine review of arXiv:2507.19092}
}

read the original abstract

The digitization of historical folkloristic materials presents unique challenges due to diverse text layouts, varying print and handwriting styles, and linguistic variations. This study explores different optical character recognition (OCR) approaches for Slovene folkloristic and historical text digitization, integrating both traditional methods and large language models (LLMs) to improve text transcription accuracy while maintaining linguistic and structural integrity. We compare single-stage OCR techniques with multi-stage pipelines that incorporate machine learning-driven post-processing for text normalization and layout reconstruction. While LLM-enhanced methods show promise in refining recognition outputs and improving readability, they also introduce challenges related to unintended modifications, particularly in the preservation of dialectal expressions and historical structures. Our findings provide insights into selecting optimal digitization strategies for large-scale folklore archives and outline recommendations for developing robust OCR pipelines that balance automation with the need for textual authenticity in digital humanities research.

Figures

Figures reproduced from arXiv: 2507.19092 by the authors.

**Figure 1.** A sample page from Kmetijske in rokodelske novice, dated November 7, 1855. This historical periodical, written in 19th-century Slovene and printed using antiquated fonts and multicolumn layouts, poses significant challenges for OCR due to typographic complexity, degraded print quality, and early orthographic conventions. 3.1.2. Ciciban The second dataset comprises issues from Ciciban [Cic45], a children’s magazin… view at source ↗

**Figure 3.** Sample page from the Inštitut za narodopisje – skenirane pravljice collection. The document is typewritten and shows typical layout and character spacing of mid-20th century folkloristic transcriptions. Despite visual uniformity, OCR can be challenged by faded ink, uneven typewriter impressions, and page-level degradation. 3.2. OCR Pipeline and Processing Workflow We evaluated two distinct OCR pipelines to assess … view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Comparison of scan quality. Left: low-quality scan with [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 6.** Figure 6: Comparison between raw Tesseract OCR and ChatGPT [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 5.** Figure 5: OCR and post-processing comparison on a historical [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 8 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry.original add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 i...
[3]

: Post-correction of historical text transcripts with large language models: An exploratory study

Boros E., Ehrmann M., Romanello M., Najem-Meyer S., Kaplan F. : Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature (LaTeCH-CLfL 2024) (2024), Association for Computatio...

work page 2024
[4]

: CLOCR-C : Context leveraging OCR correction with pre-trained language models

Bourne J. : CLOCR-C : Context leveraging OCR correction with pre-trained language models. arXiv preprint arXiv:2408.17428 (2024)

arXiv 2024
[5]

https://www.dlib.si/details/URN:NBN:SI:DOC-1R8Y8Z7L, from 1945

Ciciban : Ciciban magazine collection. https://www.dlib.si/details/URN:NBN:SI:DOC-1R8Y8Z7L, from 1945. Digitized by the National and University Library of Slovenia (NUK)

work page 1945
[6]

Historical German Text Normalization Using Type- and Token-Based Language Modeling

Ehrmanntraut A. : Historical G erman text normalization using type-and token-based language modeling. arXiv preprint arXiv:2409.02841 (2024)

work page Pith review arXiv 2024
[7]

: Enhancing OCR in historical documents with complex layouts through machine learning

Fleischhacker D., Kern R., G \"o derle W. : Enhancing OCR in historical documents with complex layouts through machine learning. International Journal on Digital Libraries 26, 1 (2025), 3

work page 2025
[8]

OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches

Kanerva J., Ledins C., K \"a pyaho S., Ginter F. : OCR error post-correction with llms in historical documents: No free lunches. arXiv preprint arXiv:2502.01205 (2025)

work page Pith review arXiv 2025

Show all 12 references

[9]

https://nl.ijs.si/imp/nuk/dl/NUKP14041-1843.html, (1843–1902)

Kmetijske in rokodelske novice. https://nl.ijs.si/imp/nuk/dl/NUKP14041-1843.html, (1843–1902). Digitized by the National and University Library of Slovenia (NUK)

1902
[10]

: Building an efficient OCR system for historical documents with little training data

Mart \' nek J., Lenc L., Kr \'a l P. : Building an efficient OCR system for historical documents with little training data. Neural Computing and Applications 32 (2020), 17209--17227

2020
[11]

: olm OCR : Unlocking trillions of tokens in pdfs with vision language models

Poznanski J., Borchardt J., Dunkelberger J., Huff R., Lin D., Rangapur A., Wilhelm C., Lo K., Soldaini L. : olm OCR : Unlocking trillions of tokens in pdfs with vision language models. arXiv preprint arXiv:2502.18443 (2025)

2025
[12]

: State of the art optical character recognition of 19th century fraktur scripts using open source engines

Reul C., Springmann U., Wick C., Puppe F. : State of the art optical character recognition of 19th century fraktur scripts using open source engines. arXiv preprint arXiv:1810.03436 (2018)

2018 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry.original add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 i...

[3] [3]

: Post-correction of historical text transcripts with large language models: An exploratory study

Boros E., Ehrmann M., Romanello M., Najem-Meyer S., Kaplan F. : Post-correction of historical text transcripts with large language models: An exploratory study. In Proceedings of the 8th joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature (LaTeCH-CLfL 2024) (2024), Association for Computatio...

work page 2024

[4] [4]

: CLOCR-C : Context leveraging OCR correction with pre-trained language models

Bourne J. : CLOCR-C : Context leveraging OCR correction with pre-trained language models. arXiv preprint arXiv:2408.17428 (2024)

arXiv 2024

[5] [5]

https://www.dlib.si/details/URN:NBN:SI:DOC-1R8Y8Z7L, from 1945

Ciciban : Ciciban magazine collection. https://www.dlib.si/details/URN:NBN:SI:DOC-1R8Y8Z7L, from 1945. Digitized by the National and University Library of Slovenia (NUK)

work page 1945

[6] [6]

Historical German Text Normalization Using Type- and Token-Based Language Modeling

Ehrmanntraut A. : Historical G erman text normalization using type-and token-based language modeling. arXiv preprint arXiv:2409.02841 (2024)

work page Pith review arXiv 2024

[7] [7]

: Enhancing OCR in historical documents with complex layouts through machine learning

Fleischhacker D., Kern R., G \"o derle W. : Enhancing OCR in historical documents with complex layouts through machine learning. International Journal on Digital Libraries 26, 1 (2025), 3

work page 2025

[8] [8]

OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches

Kanerva J., Ledins C., K \"a pyaho S., Ginter F. : OCR error post-correction with llms in historical documents: No free lunches. arXiv preprint arXiv:2502.01205 (2025)

work page Pith review arXiv 2025