REVIEW 5 major objections 4 minor 13 references
Making History Readable
T0 review · 5 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that feeding historical documents through text extraction, correction, and LLM summarization makes them searchable and understandable in a digital library.
desk verdict A clear work-in-progress system description for three historical collections, but the accessibility improvement is asserted, not measured – no accuracy data, no user study, and the one shown example is an error at 66.79% confidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pipeline's load-bearing parts are three named components: Textract, BERT, and Llama-3.1-8B-Instruct, together with a confidence-threshold correction step. Textract is a cloud OCR service that returns extracted text, layout blocks, and per-word confidence scores. BERT is a masked-language model that predicts the most probable words in context; the paper uses its top-three predictions to replace low-confidence extractions. Llama-3.1-8B-Instruct is a generative large language model that writes simplified summaries. For maps, an additional mechanism is multi-angle image rotation before extraction, with future tiling planned for large images.
What would settle it
Take a random sample of pages from the three collections, have human transcribers produce reference text, run the pipeline, and compare: if the extracted text differs from the reference often enough to make search unreliable, or if the summaries alter or invent historical facts, the claimed accessibility gain is falsified.
Extended reading notes
Core claim
The central claim is that out-of-the-box AI tools can be adapted into a working pipeline for archival materials. For the letters, a cloud OCR service produces words with confidence scores, and any word below a threshold is replaced using BERT's whole-word-masking predictions; the corrected text is then condensed by Llama-3.1-8B-Instruct into a summary in more current language. For the newspapers, layout analysis is used to handle multi-column text. For the maps, the document is rotated at multiple angles before extraction so that slanted or curved labels become readable. The authors present these adaptations as the reason the three collections can become searchable and navigable, and they state plans to refine the map pipeline with tiling and rotation.
Load-bearing premise
The whole accessibility claim rests on the unmeasured assumption that the OCR service, the BERT correction step, and the Llama summarizer are accurate enough on these particular historical documents that the resulting text can be searched and read without misleading users.
Editorial extensions
If this is right
- If the pipeline performs as described, the three collections become full-text searchable instead of being accessible only as page images.
- Simplified summaries in modern language lower the reading barrier for handwritten Civil War-era letters, making primary sources usable by non-specialists.
- The confidence-threshold correction scheme provides a reusable way to clean OCR output when extracted words come with low confidence scores.
- The planned tiling-plus-rotation approach for maps would allow very large map files to be processed without downsampling them below readability.
Reading between the lines
- An editorial inference: the same three-stage recipe could be tested on other degraded archival formats, such as census schedules or handwritten marginalia, where layout and handwriting noise are similar.
- An editorial inference: a public interface built on this pipeline would benefit from linking each summary to the original page image so readers can verify the simplified wording against the source.
- An editorial inference: a user study comparing corrected text with human transcription would tell whether the BERT correction step actually reduces errors or only replaces low-confidence words with plausible but wrong ones.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes an AI-assisted workflow at Virginia Tech University Libraries for converting three challenging historical collections (Civil War-era handwritten letters, historic newspapers, and digitized topographic maps) into machine-readable, searchable text. The proposed pipeline uses Amazon Textract for text extraction, a BERT whole-word-masking model to correct low-confidence extractions, and Llama-3.1-8B-Instruct to generate simplified summaries. The authors claim that this integration enhances user experience by making the materials easier to search, navigate, and understand. However, the manuscript is a system description without any quantitative evaluation of extraction accuracy, correction effectiveness, or summary faithfulness, and the central accessibility claim is asserted rather than demonstrated.
Significance. If the described pipeline were validated, it would address a real and widespread problem in digital libraries, and the collection-scale specifics (e.g., 1.44 million newspaper items, 23 TB) suggest the work is operationally relevant. The paper honestly identifies concrete technical challenges, such as archaic cursive glyphs, multi-column newspaper layouts, and non-linear text on maps, and it sketches a plausible combination of existing tools. The main weakness is the complete absence of evaluation: no word error rates, no retrieval precision/recall, no summary faithfulness assessment, no user study, and no comparison to baseline OCR. Because the claimed benefit is an empirical outcome, the significance of the work remains wholly unsubstantiated. The paper also ships no code, data, or machine-checked artifacts, so the reproducibility of the described corrections cannot be assessed.
major comments (5)
- [III-A, Fig. 4] The paper claims that the text correction pipeline 'ensures that the most accurate text is used in summarization,' but no evaluation of the correction mechanism is provided. The only quantitative artifact is a single extraction error ('hand' as 'fund') with a confidence score of 66.79%, and the confidence threshold triggering BERT correction is never disclosed. Without a word error rate, a comparison against the uncorrected Textract output, or a baseline such as Tesseract, the claim that the pipeline improves accuracy is load-bearing and unsupported.
- [III-B] The newspaper section describes layout analysis on multi-column pages (Fig. 2) but reports no accuracy or error measurements for text extraction on this collection. The claim that 'users can search and navigate events from historical time periods' implicitly requires that the extracted text be reliable for full-text search, yet no retrieval precision, recall, or manual spot-check results are provided. This is a central claim that depends entirely on unverified extraction quality.
- [III-C] The multi-angle rotation strategy for topographic maps is stated to 'improve the overall success of text extraction,' but no success metric is reported for the map collection. The tiling ensemble method is only mentioned as future work in Section IV, so the described end-to-end effect on extraction quality is unmeasured. This outcome claim is load-bearing because the map collection is one of the three case studies the paper uses to support its accessibility argument.
- [IV] The paper's central conclusion—'By extracting text from challenging-to-read materials, we improve accessibility, while the summarization service creates clear, concise summaries to enhance user understanding, and thus improve overall engagement with the material'—is an empirical claim that is asserted without any user study, usage analytics, or task-based evaluation. Moreover, Section IV refers to 'results discussed in Section III,' but Section III contains no results, only procedural descriptions and a single illustrative figure. This missing support is the core weakness of the manuscript.
- [III-A] The summarization service uses Llama-3.1-8B-Instruct without any faithfulness or quality evaluation. Since the LLM is not fine-tuned for historical or archaic language, there is a realistic risk that summaries are fluent but factually inaccurate, particularly for cursive Civil War-era text with uncertain OCR output. The manuscript provides no ROUGE scores, no human evaluation, and no example of a summary being checked against the source, so the claim that summaries 'enhance user understanding' is unsupported.
minor comments (4)
- [IV] The phrase 'Our results discussed in Section III' is misleading because Section III presents no experimental results; consider renaming Section III to 'Use Cases' or 'Case Studies' to avoid implying a measured outcome.
- [III-A, Fig. 4] The paper references Fig. 4 but does not explain what the reader should observe beyond the red box and the corrected word; clarify the relationship between the extracted text, the correction, and the generated summary in the figure caption or body.
- [References] Reference [9] cites Llama 3.2 model card, but the text and the model name in Section III-A refer to Llama-3.1-8B-Instruct; align the reference with the actual model used.
- [II] The term 'custom AI agents' is used repeatedly but never defined; specify the components, versions, and any bespoke code or prompting strategies to make the pipeline reproducible.
Circularity Check
No circular reasoning: the paper is a workflow description with no derivation chain, fitted parameters, or quantitative prediction that reduces to its inputs.
full rationale
This poster describes an AI pipeline (Textract for extraction, BERT whole-word masking for correction, Llama-3.1-8B for summarization) applied to three historical collections. There are no equations, no fitted parameters, no quantitative predictions, and no derivation that could be equivalent to its inputs by construction. The claimed improvement in accessibility and engagement is an empirical assertion, not a derived result; the paper offers no accuracy measurements, but absence of validation is a correctness or evidence concern, not circularity. The only self-references are to the authors' own DLP and collection pages, which serve as descriptive pointers rather than load-bearing evidence for any technical claim. No step reduces to a self-citation chain or to a definition. Therefore the appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
free parameters (1)
- Confidence threshold for text block acceptance =
not stated in paper
assumptions (3)
- domain assumption Textract provides accurate enough text extraction for the three collections
- ad hoc to paper BERT word-masking correction improves the extracted text
- domain assumption Llama-3.1-8B produces faithful, useful summaries
Cite this review
Pith. "Pith review of Making History Readable." pith.science (2026). https://pith.science/paper/C2TR5JRN
@misc{pith2026241117600,
author = {Pith},
title = {Pith review of: Making History Readable},
year = {2026},
howpublished = {\url{https://pith.science/paper/C2TR5JRN}},
note = {Machine review of arXiv:2411.17600}
}
read the original abstract
The Virginia Tech University Libraries (VTUL) Digital Library Platform (DLP) hosts digital collections that offer our users access to a wide variety of documents of historical and cultural importance. These collections are not only of academic importance but also provide our users with a glance at local historical events. Our DLP contains collections comprising digital objects featuring complex layouts, faded imagery, and hard-to-read handwritten text, which makes providing online access to these materials challenging. To address these issues, we integrate AI into our DLP workflow and convert the text in the digital objects into a machine-readable format. To enhance the user experience with our historical collections, we use custom AI agents for handwriting recognition, text extraction, and large language models (LLMs) for summarization. This poster highlights three collections focusing on handwritten letters, newspapers, and digitized topographic maps. We discuss the challenges with each collection and detail our approaches to address them. Our proposed methods aim to enhance the user experience by making the contents in these collections easier to search and navigate.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Digital Libraries & Repositories,
V . D. L. Platform, “Digital Libraries & Repositories,” 2024. [Online]. Available: https://lib.vt.edu/content/lib vt edu/en/find-borrow/ digital-library.html
work page 2024
-
[2]
S. N. Srihari, A. Shekhawat, and S. W. Lam, Optical character recog- nition (OCR) . GBR: John Wiley and Sons Ltd., 2003, p. 1326–1333
work page 2003
-
[3]
The 6 Biggest OCR Problems and How to Overcome Them
C. Marketing, “The 6 Biggest OCR Problems and How to Overcome Them.” [Online]. Available: https://conexiom.com/blog/the-6-biggest- ocr-problems-and-how-to-overcome-them
-
[4]
pytesseract: Python-tesseract is a python wrapper for Google’s Tesseract-OCR
S. Hoffstaetter, “pytesseract: Python-tesseract is a python wrapper for Google’s Tesseract-OCR.” [Online]. Available: https://github.com/ madmaze/pytesseract
-
[5]
Google, “Tesseract documentation.” [Online]. Available: https: //tesseract-ocr.github.io/
-
[6]
OCR Software, Data Extraction Tool - Amazon Textract - AWS,
A. W. Services, “OCR Software, Data Extraction Tool - Amazon Textract - AWS,” 2019. [Online]. Available: https://aws.amazon.com/textract/
work page 2019
-
[7]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips.cc/paper/ 2017/hash/3f5ee243547dee91fbd053c1c4...
work page 2017
-
[8]
BERT: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers) . Minneapolis, Minnesota: Association for ...
work page 2019
Show all 13 references
-
[9]
Llama 3.2 model card,
AI@Meta, “Llama 3.2 model card,” 2024. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.2-3B
2024
-
[10]
Improving language understanding by generative pre-training,
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://s3-us-west-2.amazonaws.com/openai-assets/research- covers/language-unsupervised/language understanding paper.pdf
2018
-
[11]
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,
M. e. a. Abdin, “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” Aug. 2024, arXiv:2404.14219. [Online]. Available: http://arxiv.org/abs/2404.14219
2024 arXiv
-
[12]
Virginia Tech Digital Libraries | Montgomery Museum
VTDLP, “Virginia Tech Digital Libraries | Montgomery Museum.” [Online]. Available: https://digital.lib.vt.edu/collection/92992h2p
-
[13]
Virginia Tech Digital Libraries | Newman Library Map Collection
D. L. P. VT, “Virginia Tech Digital Libraries | Newman Library Map Collection.” [Online]. Available: https://digital.lib.vt.edu/collection/ cq35qv9s
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.