REVIEW 2 major objections 6 minor 26 references
Transcribing Spanish Texts from the Past: Experiments with Transkribus, Tesseract and Granite
T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A 2-billion-parameter multimodal model, fine-tuned on a consumer GPU with QLoRA, produced the team's strongest OCR transcriptions in the 2025 PastReader shared task, placing second in five of eight official metrics.
desk verdict A modest, honest shared-task report whose real payload is the competitive Granite 2B fine-tuning result on consumer hardware; the rank claim needs the full leaderboard but the measurements are worth taking seriously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the fine-tuned compact vision-language model Granite3.2-vision:2b, a 2-billion-parameter multimodal model whose vision tower processes document images in 384×384 crops. Fine-tuning uses QLoRA, which loads the base model in 4-bit NF4 precision and updates only lightweight LoRA adapters injected into the language model's projection layers, leaving the vision tower and output head in full precision; with a per-device batch size of 1, 8 gradient-accumulation steps, and bfloat16, this fits the whole run into 16 GB of VRAM. The model is prompted through a chat-format dataset with a Spanish-specific OCR system prompt instructing it to return only raw text, and page images are resized to 414×585 with aspect-ratio-preserving padding to stay near the processor's crop size. This mechanism is what carries the paper's claim that a compact multimodal model can rival much larger systems on semantic and word-level metrics while remaining trainable on ordinary hardware.
What would settle it
Recompute the official metrics after re-annotating the test-set ground truth with explicit philological transcription rules—covering diacritics, historical spellings, and line-break decisions—and see whether Granite3.2-vision:2b still places second in WER and the ROUGE variants; if the margins to the winner move beyond the reported thousandths-level gaps, the claimed ranking is an artifact of the noisy ground truth. A complementary check is to measure inter-annotator agreement on a sample of the ground-truth files, since low agreement would directly undermine the metric comparisons.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that Granite3.2-vision:2b—a 2-billion-parameter open multimodal model—can be fine-tuned for end-to-end transcription of historical Spanish documents using QLoRA 4-bit quantization and LoRA adapters on a consumer GPU with 16 GB of VRAM, and the resulting system is 'arguably our strongest submission overall'. In the official leaderboard it took second place in WER and in all four ROUGE variants, with scores within a few thousandths of the winning system, and third in BLEU; only character-level Levenshtein distance and normalised edit distance trailed, placing fourth. The paper attributes the character-level gap to the model's tendency to join hyphen-split words, 'correct' historical Spanish spellings, and hallucinate in illegible areas, and it notes preliminary evidence that dev/train and test splits differ in source diversity and background colour. The same run also demonstrates the feasibility of the whole pipeline—PDF-to-PNG conversion, padding to a 414×585 resolution, prompt-based chat formatting, and quantised fine-tuning—on consumer hardware.
Load-bearing premise
The rankings depend on the shared task's ground-truth transcriptions being consistently annotated, but the authors themselves report finding labelling inconsistencies during fine-tuning, so if those inconsistencies distort the metrics the central comparison could change.
Editorial extensions
If this is right
- A 2B-parameter multimodal model fine-tuned with QLoRA on a 16 GB consumer GPU can reach second place in five of eight official OCR metrics, so competitive historical transcription no longer requires high-performance computing.
- Tesseract fine-tuning on this dataset did not improve over the off-the-shelf model, suggesting that traditional OCR transfer learning plateaus unless the training data is clean, consistent, and well aligned.
- For Transkribus, preserving line breaks helps precision metrics (Levenshtein, WER, NED), while joining lines helps semantic metrics (BLEU, ROUGE), showing that output formatting alone shifts scores across metric families.
- Granite's best results are in word-level and semantic metrics, so further gains should target literal character accuracy: diacritics, historical spellings, and hyphenated line breaks.
- The apparent distribution mismatch between dev/train and test sets means future fine-tuning should control for source diversity and background colour before drawing conclusions about model generalisation.
Reading between the lines
- A natural testable extension: once the authors release their planned transcription guidelines, re-running the same QLoRA recipe on a cleaned dataset should specifically improve Levenshtein and NED, since current Granite errors concentrate in diacritics and historical spelling 'corrections'.
- The paper's line-break findings imply that OCR evaluation campaigns should report metrics on both raw and normalised text, because a model's formatting choices can change BLEU and ROUGE scores without changing transcription content.
- If the recipe transfers, the same 2B-model-plus-QLoRA pipeline could be applied to other low-resource historical languages; the open question, which the paper does not test, is whether its success depends on Spanish being close enough to the model's English-centric pretraining.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the GRESEL team's participation in the IberLEF 2025 PastReader shared task, which asks systems to transcribe historical Spanish documents from PDF images. Three approaches are compared: Transkribus using the Coloso Español model, Tesseract both off-the-shelf and fine-tuned with tesstrain, and a 2B-parameter Granite3.2-vision model fine-tuned with QLoRA on an RTX 5080. Using the official leaderboard results (Table 3), the authors find that the fine-tuned Granite run places second in five of eight metrics, including WER and all ROUGE variants, and they interpret this as evidence that a compact multimodal model fine-tuned on consumer hardware can be competitive for historical-document OCR. The paper also reports that Tesseract fine-tuning did not improve over the baseline, analyzes failure patterns on difficult pages, discusses the effect of line-break preservation on different metrics, and includes energy/emission estimates. The authors are careful to label several explanations as hypotheses and to acknowledge dataset-quality issues.
Significance. If the ranking claim holds, the paper provides a useful empirical data point: a 2B-parameter vision-language model fine-tuned with QLoRA on a single consumer GPU can outperform traditional OCR engines on WER and semantic overlap metrics for historical Spanish texts, while remaining much cheaper than large-scale HPC training. The paper is also transparent about limitations: it explicitly states that ground-truth inconsistencies exist, that hyperparameter exploration was minimal, that the train/test distribution may differ, and that some observed phenomena are only hypotheses. The practical engineering detail—PDF-to-PNG conversion, padding to fixed resolution, NF4 quantization, gradient checkpointing, and LoRA adapter placement—will be helpful to practitioners who want to reproduce or adapt this pipeline. The main value is therefore a reproducible system-description contribution with an honest assessment of what did and did not work.
major comments (2)
- [Section 4.1, Table 3] The claim that GRESEL2_run1 achieved "a solid second place in five out of the eight evaluation metrics" is only verifiable if Table 3 is the complete official leaderboard for the task. The table lists seven rows, but the manuscript never states that this is the set of all submitted runs, and the cited overview paper [1] is not reproduced. If any other team's run lies between OCRTITS_run1 and GRESEL2_run1 on WER or any ROUGE metric, the stated rank would be incorrect. Please add an explicit sentence that Table 3 contains the complete official leaderboard (with the total number of participating teams/runs), or otherwise qualify the rank claim.
- [Sections 2.2, 3.2, and 4.1] The authors themselves report "inconsistencies in labelling" in the training data (Section 3.2) and state that "it is essential that the dataset is clean and well organised, as well as the application of systematic transcription rules" (Section 2.2), which they say is still pending. Since all leaderboard metrics are computed against this ground truth, the central quantitative claims—including the Granite second-place result—are potentially affected by label noise or inconsistent transcription conventions. The manuscript should explicitly discuss how such inconsistencies could bias the comparison, for example by favoring models that reproduce the annotators' errors or by penalizing models that transcribe the page literally. This is a load-bearing caveat rather than a stylistic one, and it should appear in or near Section 4.1.
minor comments (6)
- [Section 4.1] The text says "even achieving second place in BLUE"; the metric name should be BLEU. The same typo appears in the description of the Transkribus joined-lines run.
- [Section 3.3] The 90/10 random split of the combined training and development sets is described as producing "training" and "testing" partitions. Since the official test set is used later, calling this second split "testing" is confusing; recommend "validation" or "held-out subset."
- [Section 3.2] The conclusion that fine-tuned Tesseract "consistently" underperforms the baseline is mainly supported by Figure 3 on the development set, but the figure's caption does not identify which metrics are shown or how scores are normalized. Adding axis labels and a legend would make the comparison easier to verify.
- [Section 4.1, Figure 8] The emission analysis is described as a "simulated 10-hour execution window," but the method for estimating energy consumption and CO2 is not specified. A one-sentence description of the estimation formula or tool would improve reproducibility.
- [Section 3.3, Table 1] The hyperparameter row "Dataset text field \"\"" is unclear; it likely refers to the text column being empty for image-only inputs. Please clarify what this setting does.
- [Throughout] There are several minor wording and formatting errors, including "an useful point," "As it can bee seen," "theColoso Español" (missing space), and the email address in the author affiliation block appears garbled. These do not affect the technical content but should be corrected in a final revision.
Circularity Check
No significant circularity: the Granite result is anchored to an external official leaderboard, not derived from the paper's own fitted inputs.
full rationale
The paper's central empirical claim is the ranking of GRESEL2_run1 on the PastReader 2025 test set, as reported in Table 3 with results provided by the task organization. The fine-tuned Granite model was trained on a random 90/10 split of the development and training sets and then run on the test set released later by the organization (Section 3.3); the authors explicitly state that hyperparameters remained largely unexplored and were mostly defaults or initial guesses, so no parameter was fitted to the test labels. The leaderboard is an external benchmark, and the paper does not reconstruct the ranking from any fitted quantity of its own. The self-citations to the companion papers [3] and [4] are descriptive references to separate system descriptions rather than load-bearing evidence for the validity of the current results. The dataset citation [21] includes two of the present authors as contributors, but the dataset and official evaluation are products of the shared task organization, and no argument in the paper reduces to that citation. The observation that Table 3 lists only seven runs and therefore cannot by itself prove 'second place' against the complete official leaderboard is an evidence-completeness concern, not circularity. Likewise, the authors' admission of labeling inconsistencies in the ground truth is a data-quality limitation that could distort all metric comparisons, but it does not make any claimed prediction equivalent to its input by construction. No equation or claimed result in the paper reduces to a fitted parameter renamed as a prediction, and no uniqueness or validity claim is imported from the authors' prior work.
Assumptions & free parameters
free parameters (6)
- Granite learning rate =
1e-4
- Number of training epochs =
1
- Gradient accumulation steps =
8
- Target image resolution =
414x585
- Train/dev split ratio for fine-tuning =
90/10 random
- Tesseract page segmentation mode =
psm 6
assumptions (4)
- domain assumption The official evaluation metrics (WER, BLEU, ROUGE family, edit distances) are valid proxies for OCR transcription quality.
- domain assumption Ground-truth transcriptions in the PastReader dataset are correct and consistently transcribed.
- domain assumption The shared task test set is representative of historical Spanish documents and is not leaked into fine-tuning.
- domain assumption QLoRA 4-bit quantization with NF4 preserves enough model quality for OCR.
Cite this review
Pith. "Pith review of Transcribing Spanish Texts from the Past: Experiments with Transkribus, Tesseract and Granite." pith.science (2026). https://pith.science/paper/K4RL4FTL
@misc{pith2026250704878,
author = {Pith},
title = {Pith review of: Transcribing Spanish Texts from the Past: Experiments with Transkribus, Tesseract and Granite},
year = {2026},
howpublished = {\url{https://pith.science/paper/K4RL4FTL}},
note = {Machine review of arXiv:2507.04878}
}
read the original abstract
This article presents the experiments and results obtained by the GRESEL team in the IberLEF 2025 shared task PastReader: Transcribing Texts from the Past. Three types of experiments were conducted with the dual aim of participating in the task and enabling comparisons across different approaches. These included the use of a web-based OCR service, a traditional OCR engine, and a compact multimodal model. All experiments were run on consumer-grade hardware, which, despite lacking high-performance computing capacity, provided sufficient storage and stability. The results, while satisfactory, leave room for further improvement. Future work will focus on exploring new techniques and ideas using the Spanish-language dataset provided by the shared task, in collaboration with Biblioteca Nacional de Espa\~na (BNE).
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
A. Montejo-Ráez, E. Sánchez-Nogales, G. Expósito-Álvarez, L. A. Ureña-López, M. T. Martín- Valdivia, J. Collado-Montañez, I. Cabrera-de Castro, M. V. Cantero-Romero, R. Ortuño-Casanova, Overview of pastreader shared task in iberlef 2025: Transcribing texts from the past, Procesamiento del Lenguaje Natural 75 (2025)
work page 2025
-
[2]
J. Á. González-Barba, L. Chiruzzo, S. M. Jiménez-Zafra, Overview of IberLEF 2025: Natural Language Processing Challenges for Spanish and other Iberian Languages, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2025), co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025), CEUR-WS. org, 2025
2025
-
[3]
J. Macicior-Mitxelena, Transcribing History with Tesseract: A Monomodal OCR Approach in the PastReader 2025 Shared Task, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2025) co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025), volume TBD of CEUR Workshop Proceedings, CEUR-WS, Zaragoza...
work page 2025
-
[4]
Y. A. Torterolo-Orta, M. Miguez-Lamanuzzi, Fine-Tuning a Compact Multimodal Model on Consumer-Grade Hardware at PastReader 2025, in: Proceedings of the Iberian Languages Evalua- tion Forum (IberLEF 2025) co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025), volume TBD of CEUR Workshop Proceedings , CEUR-W...
work page 2025
-
[5]
E. Sánchez-Salido, A. Menta, A. García-Serrano, Seeking information in spanish historical newspa- pers: The case of diario de madrid (18th and 19th centuries)., DHQ: Digital Humanities Quarterly (2023)
work page 2023
-
[6]
J. J. Lastra-Díaz, J. Goikoetxea, M. A. Hadj Taieb, A. Garcia-Serrano, M. Ben Aouicha, E. Agirre, D. Sánchez, A large reproducible benchmark of ontology-based methods and word embeddings for word similarity, Information Systems 96 (2021) 101636. URL: https://www.sciencedirect.com/ science/article/pii/S0306437920301058. doi:https://doi.org/10.1016/j.is.2020.101636
-
[7]
E. Garcia-Arias, A. Garcia-Serrano, Creación de un modelo de descripciones de imágenes especial- izado en arqueología griega (pendiente de publicación), Procesamiento del Lenguaje Natural 75 (2025)
work page 2025
-
[8]
A. Garcia-Serrano, X. Benavent, R. Granados, J. M. Goñi-Menoyo, Some results using different approaches to merge visual and text-based features in clef’08 photo collection, in: C. Peters, T. Dese- laers, N. Ferro, J. Gonzalo, G. J. F. Jones, M. Kurimo, T. Mandl, A. Peñas, V. Petras (Eds.), Evaluating Systems for Multilingual and Multimodal Information Acc...
Show all 26 references
-
[9]
J. L. Martínez-Fernández, J. V. Román, A. M. Garcia-Serrano, J. C. González-Cristóbal, Combining textual and visual features for image retrieval, in: C. Peters, F. C. Gey, J. Gonzalo, H. Müller, G. J. F. Jones, M. Kluck, B. Magnini, M. de Rijke (Eds.), Accessing Multilingual I...
2006 doi
-
[10]
Moreno-Sandoval, J
A. Moreno-Sandoval, J. Porta, B. Carbajo-Coronado, Y. Torterolo, D. Samy, The financial document causality detection shared task (FinCausal 2025), in: C.-C. Chen, A. Moreno-Sandoval, J. Huang, Q. Xie, S. Ananiadou, H.-H. Chen (Eds.), Proceedings of the Joint Workshop of the 9t...
2025
-
[11]
Porta-Zamorano, Y
J. Porta-Zamorano, Y. Torterolo-Orta, A. Moreno-Sandoval, LLI-UAM Team at FinancES 2023: Noise, Data Augmentation and Hallucinations, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023) co-located with the Conference of the Spanish Society for Natural Lang...
2023
-
[12]
Y. A. Torterolo-Orta, S. M. Roseti, A. Moreno-Sandoval, Assessing a literary rag system with a human-evaluated synthetic qa dataset generated by an llm: Experiments with knowledge graphs, Procesamiento del Lenguaje Natural 75 (2025). In press
2025
-
[13]
H. Liu, C. Li, Q. Wu, Y. J. Lee, Visual instruction tuning, 2023. URL: https://arxiv.org/abs/2304.08485. arXiv:2304.08485
2023 arXiv
-
[14]
Grattafiori, A
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, (...), Z. Ma, The llama 3 herd of models, 2024. URL: http...
2024 arXiv
-
[15]
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. bastien Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, (...), L. Hussenot, Gemma 3 technical report, 2025. U...
2025 arXiv
-
[16]
Poznanski, J
J. Poznanski, J. Borchardt, J. Dunkelberger, R. Huff, D. Lin, A. Rangapur, C. Wilhelm, K. Lo, L. Soldaini, olmocr: Unlocking trillions of tokens in pdfs with vision language models, 2025. URL: https://arxiv.org/abs/2502.18443. arXiv:2502.18443
2025
-
[17]
P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, J. Lin, Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, arXiv preprint arXiv:2409.12191 (2024)
2024 arXiv
-
[18]
G. V. Team, L. Karlinsky, A. Arbelle, A. Daniels, A. Nassar, A. Alfassi, B. Wu, E. Schwartz, D. Joshi, J. Kondic, N. Shabtay, P. Li, R. Herzig, S. Abedin, S. Perek, S. Harary, U. Barzelay, A. R. Goldfarb, A. Oliva, B. Wieles, (...), R. Feris, Granite vision: a lightweight, ope...
2025 arXiv
-
[19]
Hegghammer, Ocr with tesseract, amazon textract, and google document ai: a benchmarking experiment, Journal of Computational Social Science 5 (2022) 861–882
T. Hegghammer, Ocr with tesseract, amazon textract, and google document ai: a benchmarking experiment, Journal of Computational Social Science 5 (2022) 861–882
2022
-
[20]
Fleischhacker, W
D. Fleischhacker, W. Goederle, R. Kern, Improving ocr quality in 19th century historical documents using a combined machine learning based approach, arXiv preprint arXiv:2401.07787 (2024)
2024 arXiv
-
[21]
Montejo-Ráez, E
A. Montejo-Ráez, E. Sánchez Nogales, G. Expósito Álvarez, A. Ureña López, M. T. Martín-Valdivia, J. Collado-Montañez, I. Cabrera de Castro, M. V. Cantero Romero, A. García Serrano, R. Or- tuño Casanova, Y. A. Torterolo Orta, Pastreader 2025, https://doi.org/10.5281/zenodo.15084265,
2025 doi
-
[22]
Schwartz, Fine-tuning granite vision with trl and peft (lora), https://colab.research.google
E. Schwartz, Fine-tuning granite vision with trl and peft (lora), https://colab.research.google. com/github/huggingface/cookbook/blob/main/notebooks/en/fine_tuning_granite_vision_sft_trl. ipynb, 2024. URL: https://colab.research.google.com/github/huggingface/cookbook/blob/main...
2024
-
[23]
Zhang, O
J. Zhang, O. Liu, T. Yu, J. Hu, W. Neiswanger, Euclid: Supercharging multimodal llms with synthetic high-fidelity visual descriptions, arXiv preprint arXiv:2412.08737 (2024)
2024 arXiv
-
[24]
Dettmers, A
T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms, 2023. URL: https://arxiv.org/abs/2305.14314. arXiv:2305.14314
2023 arXiv
-
[25]
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adapta- tion of large language models, 2021. URL: https://arxiv.org/abs/2106.09685. arXiv:2106.09685
2021 arXiv
-
[26]
Narbona, S
A. Narbona, S. Ros, Beyond traditional ocr: Exploring the efficiency of llms in document processing, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2025) co-located with the 41st Conference of the Spanish Society for Natural Language Processing (SEPLN 2025)...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.