Pith. sign in

REVIEW 2 major objections 2 minor 6 references

A pipeline that extracts glyphs from manuscript images, clusters them, and builds distance-based stemmas can reconstruct copying relationships without any text transcription.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-10 16:01 UTC

load-bearing objection The OCR comparisons give a useful snapshot of current tools for Church Slavonic, but the visual stemma pipeline only shows it runs on small sets without evidence it recovers actual copying history. the 2 major comments →

arxiv 2604.11724 v1 submitted 2026-04-13 cs.CV

The Devil is in the Details -- From OCR for Old Church Slavonic to Purely Visual Stemma Reconstruction

Armin Hoenen This is my paper
classification cs.CV
keywords OCRstemmaticsOld Church Slavonicvisual analysismanuscriptsglyph clusteringdistance matriximage processing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper first benchmarks multiple OCR approaches on late handwritten Old Church Slavonic manuscripts and reports that basic-letter character error rates can reach 2-3 percent with current systems including large language models, while diacritics remain difficult. It then introduces and tests a purely visual stemmatology method that automatically extracts letter shapes from page images, groups similar glyphs, compares pairs statistically, and converts the resulting distances into a family tree. If the method works, researchers could trace manuscript lineages directly from scans of texts whose language or script makes full transcription expensive or error-prone. The demonstration on two small corpora, one Church Slavonic and one French, shows that the pipeline produces plausible stemmas from image data alone.

Core claim

The author presents a complete image-only pipeline for stemma construction: visual glyph extraction from manuscript pages, unsupervised clustering of letter forms, pairwise statistical comparison to form a distance matrix, and conversion of that matrix into a stemma diagram. When applied to a small set of 14th-16th century Church Slavonic Gospel of Mark copies and a set of 14th-15th century Roman de la Rose manuscripts, the pipeline produces tree structures that the author treats as evidence of basic functionality. The claim is that visual shape information alone can serve as a workable proxy for historical copying relationships.

What carries the argument

The visual glyph extraction, clustering, and pairwise statistical comparison pipeline that produces a distance matrix and stemma directly from page images.

Load-bearing premise

That letter shapes captured in images, without any textual content or expert judgment, contain enough information to recover accurate historical copying relationships on the tested corpora.

What would settle it

Running the visual pipeline on a corpus whose copying relationships are already known from textual collation and finding that the resulting stemma differs substantially from the established tree.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • OCR output can feed into stemmatology even when full transcription is imperfect.
  • Stemma reconstruction becomes feasible for manuscripts whose scripts are hard to transcribe reliably.
  • The distance matrix from glyph comparisons offers a quantitative starting point that can later be refined with textual or codicological data.
  • The same visual approach can be tested on additional small corpora to check consistency across languages and periods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the method scales beyond the two small test sets, it could reduce the bottleneck of manual transcription in digital humanities projects that aim to map manuscript traditions.
  • The approach might be combined with existing textual stemmatology tools so that visual distances serve as one data layer among others rather than the sole input.
  • Limitations observed with diacritics in the OCR section suggest that glyph features involving fine marks may need special handling in any larger deployment of the visual pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript compares a range of OCR systems (classical, ML-based, and LLMs such as GPT-5 and Gemini-3-flash) on approximately 6,000 characters from 18th-century handwritten Old Church Slavonic manuscripts, evaluates post-processing and agentic architectures, and reports that CER for basic letters can reach 2-3% while noting persistent issues with diacritics. It then introduces a purely visual stemma reconstruction pipeline consisting of automated glyph extraction, clustering, pairwise statistical comparisons to produce a distance matrix, and applies the method to two small corpora (14th-16th century Church Slavonic Gospel of Mark and 14th-15th century French Roman de la Rose), claiming that basic functioning of the method can be demonstrated.

Significance. If the visual stemma pipeline can be shown to recover genealogical relationships rather than merely scribal style or script family, it would constitute a novel contribution to computational stemmatology by operating independently of textual variants. The OCR experiments provide practical insights into current LLM capabilities for historical scripts. However, the current lack of validation metrics, ground-truth comparisons, or controls for confounding visual factors substantially limits the demonstrated impact.

major comments (2)
  1. [stemma reconstruction pipeline] In the stemma reconstruction section, the pipeline is applied to two small corpora and 'basic functioning' is claimed, yet no quantitative validation metrics (e.g., agreement with expert stemmata), error analysis, or controls for non-genetic visual similarity (shared scribal training, regional hand, or layout) are reported. This is load-bearing for the central claim that the distance matrix reconstructs historical copying relationships.
  2. [stemma reconstruction pipeline] No ablation isolating visual glyph statistics from textual content, and no baseline comparison against traditional variant-based stemmatology, is presented. Without such tests it remains unclear whether the pairwise distances encode descent or merely visual similarity, directly affecting the validity of the downstream stemma.
minor comments (2)
  1. [abstract] The abstract states that 'more than 10 CS OCR-systems among which 2 LLMs (GPT5 and Gemini3-flash) are being compared' but does not list the full set of systems or the precise conditions under which the 2-3% CER is achieved; this should be clarified with a table of results.
  2. [abstract and introduction] Several sentences contain awkward or passive phrasing (e.g., 'Focussing on basic letter correctness, more than 10 CS OCR-systems ... are being compared' and 'With new technology elaborated, experiments suggest...'). Consider revising for directness and readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. The primary concerns relate to the validation and interpretation of the visual stemma reconstruction pipeline. We address these points directly below while maintaining that the work offers a proof-of-concept demonstration of a text-independent approach, consistent with the limited scope and small corpora described.

read point-by-point responses

  1. Referee: In the stemma reconstruction section, the pipeline is applied to two small corpora and 'basic functioning' is claimed, yet no quantitative validation metrics (e.g., agreement with expert stemmata), error analysis, or controls for non-genetic visual similarity (shared scribal training, regional hand, or layout) are reported. This is load-bearing for the central claim that the distance matrix reconstructs historical copying relationships.

    Authors: We agree that quantitative validation metrics, error analysis, and explicit controls for non-genetic factors would strengthen the presentation. For the small historical corpora examined, however, no consensus expert stemmata exist for direct quantitative comparison, and constructing reliable ground truth would require extensive philological expertise outside the computational scope of this study. The claim of 'basic functioning' is supported by the pipeline producing distance matrices and clusterings that align qualitatively with known manuscript dates and provenances. We will revise the manuscript to include a more detailed discussion of these limitations, potential confounders such as shared scribal training or layout, and directions for future controlled validation. revision: partial

  2. Referee: No ablation isolating visual glyph statistics from textual content, and no baseline comparison against traditional variant-based stemmatology, is presented. Without such tests it remains unclear whether the pairwise distances encode descent or merely visual similarity, directly affecting the validity of the downstream stemma.

    Authors: The pipeline is constructed to use only visual glyph features extracted from images, with no textual content or OCR output entering the clustering or distance matrix computation; an ablation separating visual statistics from text is therefore satisfied by design. A direct baseline against traditional variant-based methods is not provided because those approaches require accurate transcriptions, whose challenges for these scripts are precisely the subject of the OCR experiments in the first half of the paper. The visual method is offered as an alternative precisely when transcription is unreliable. We will revise the text to clarify this motivation and the interpretive scope of the resulting distance matrix. revision: partial

Circularity Check

0 steps flagged

No circularity: experimental OCR comparisons and visual pipeline demonstration

full rationale

The paper reports empirical comparisons of multiple OCR systems (classical, ML, LLMs) on ~6000 characters of 18th-century Church Slavonic manuscripts, followed by a described pipeline of visual glyph extraction, clustering, pairwise statistical distances, and stemma construction applied to two small corpora. It concludes only that basic functioning can be demonstrated. No equations, derivations, fitted parameters, or predictions are presented. No self-citations are invoked as load-bearing premises. The work is self-contained experimental description without any reduction of claimed outputs to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is experimental and introduces no mathematical axioms, free parameters, or invented entities in the abstract; any clustering thresholds or comparison statistics are unspecified.

pith-pipeline@v0.9.0 · 5591 in / 1217 out tokens · 64536 ms · 2026-05-10T16:01:44.604070+00:00 · methodology

0 comments
read the original abstract

The age of artificial intelligence has brought many new possibilities and pitfalls in many fields and tasks. The devil is in the details, and those come to the fore when building new pipelines and executing small practical experiments. OCR and stemmatology are no exception. The current investigation starts comparing a range of OCR-systems, from classical over machine learning to LLMs, for roughly 6,000 characters of late handwritten church slavonic manuscripts from the 18th century. Focussing on basic letter correctness, more than 10 CS OCR-systems among which 2 LLMs (GPT5 and Gemini3-flash) are being compared. Then, post-processing via LLMs is assessed and finally, different agentic OCR architectures (specialized post-processing agents, an agentic pipeline and RAG) are tested. With new technology elaborated, experiments suggest, church slavonic CER for basic letters may reach as low as 2-3% but elaborated diacritics could still present a problem. How well OCR can prime stemmatology as a downstream task is the entry point to the second part of the article which introduces a new stemmatic method based solely on image processing. Here, a pipeline of automated visual glyph extraction, clustering and pairwise statistical comparison leading to a distance matrix and ultimately a stemma, is being presented and applied to two small corpora, one for the church slavonic Gospel of Mark from the 14th to 16th centuries, one for the Roman de la Rose in French from the 14th and 15th centuries. Basic functioning of the method can be demonstrated.

Figures

Figures reproduced from arXiv: 2604.11724 by Armin Hoenen.

Figure 1
Figure 1. Figure 1: Here, Red2Black made the last word appear in the transcript but deteriorated the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Model similarities. Base Models (→ families cluster such as LLMs, Transkribus models, fine-tuned models), the only exception being Gem 2. Model families clearly cluster, but the prompt optimization pushes Gem 2 into a very different direction showing how much influence prompt optimization can have. 2.8 Single letters vs. letters in context In order to see, for instance, how strong the editorial urge of lar… view at source ↗
Figure 3
Figure 3. Figure 3: Thinking moment, a look into the inner workings of the thinking LLM. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model similarities of post-corrected. No families discernible. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Architecture and results of the full agentic OCR approach executed with GPT5. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Icon photo of photo, book O. Zonowa, The treasures of the Kremlin. Moscow:1963, No. 36. Warlaam, end of the 15th century. Not enhanced with upscayl [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Left: Forged by Gemini by substituting Jesus with Vlad and Prophets with Sorcerers [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GPT generated with an example image; note that letters are not print-exact twins. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Obfuscating forgeries by making the image less sharp, degraded, badly lighted and [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of generated forgeries, including output from a post-edited unrestricted [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Cluster example snapshots, left good cluster, right bad cluster. [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: An unrooted topology obtained by Neighbour Joining for the 4 manuscripts. [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

  1. [2]

    Multimodal

    URL https://arxiv.org/abs/2504.00414. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778,

    work page arXiv

  2. [3]

    doi: 10.30687/mag/2724-3923/2025/02/005

    ISSN 2724-3923. doi: 10.30687/mag/2724-3923/2025/02/005. URL https://edizionicafoscari.it/en/edizioni4/riviste/magazen/2025/ 2/llm-mining-pre-stemmatological-philological-litera/ . Published on- line January 16,

    work page doi:10.30687/mag/2724-3923/2025/02/005 2025

  3. [4]

    OCR er- ror post-correction with LLMs in historical documents: No free lunches

    Jenna Kanerva, Cassandra Ledins, Siiri Käpyaho, and Filip Ginter. OCR er- ror post-correction with LLMs in historical documents: No free lunches. In Špela Arhar Holdt, Nikolai Ilinykh, Barbara Scalvini, Micaella Bruton, Iben Ny- holm Debess, and Crina Madalina Tudor, editors, Proceedings of the Third Workshop on Resources and Representations for Under-Res...

    work page 2025

  4. [5]

    ISBN 978-9908-53-121-2

    University of Tartu Library, Estonia. ISBN 978-9908-53-121-2. URL https://aclanthology.org/2025.resourceful-1.8/. Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83–97,

    work page 2025

  5. [6]

    Levchenko

    Maria A. Levchenko. Evaluating LLMs for historical document OCR: A method- ological framework for digital humanities. In Isuri Nanomi Arachchige, Francesca Frontini, Ruslan Mitkov, and Paul Rayson, editors, Proceedings of the First Work- shop NLP and Language Models for Digital Humanities associated with RANLP 2025, pages 75–85, Varna, Bulgaria, September

    work page 2025

  6. [7]

    URL https://aclanthology.org/2025.lm4dh-1.7/

    INCOMA Ltd., Shoumen, Bulgaria. URL https://aclanthology.org/2025.lm4dh-1.7/. Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129–137,

    work page 2025