REVIEW 4 major objections 5 minor 31 references
Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The pipeline reduces character error rate from 0.43 to 0.18 and from 0.33 to 0.15 on the two evaluated Kwak'wala books, and structural error rate from 25 to 2 and from 18 to 3.
desk verdict A genuinely useful digitization pipeline for Kwak'wala legacy texts, but the headline error reductions rest on an underspecified evaluation sample that may overlap with the post-correction model's training data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the masking–post-correction–reconstruction loop. A binary fastText language identification model (trained on 1000 sentences per language) labels each OCR bounding box as English or Kwak'wala; the masking layer then hides English text, numbers, and parentheses, storing token-level indices so they can be reinserted after correction. The post-correction model is the multi-source neural architecture of Rijhwani et al. (2021), trained from scratch on the labeled Boas-Hunt dataset and pretrained on unlabeled first-pass outputs. By isolating Kwak'wala text, the pipeline limits post-correction to the language that needs it, avoiding damage to already-good English OCR.
What would settle it
Take a random sample of pages from all five books, including pages with inline figures, single and double columns, and the vocabulary appendices; have an expert produce gold references; run the pipeline; and compare CER and SER. If the error reductions are substantially smaller on that sample than the 0.18/0.15 CER and 2/3 SER reported, the central claim is overstated.
Extended reading notes
Core claim
On the book pages tested, the proposed mixed-methods pipeline reduces the character error rate by roughly half relative to raw first-pass OCR and cuts the structural error rate by about 87.5%, bringing the digitized text close to gold-reference quality. The discovery is that language identification can serve as a structure proxy in a multilingual legacy document: by labeling each bounding box as English or Kwak'wala, the pipeline can mask non-Kwak'wala content, run post-correction only on the Kwak'wala text where errors concentrate, and then reconstruct the original layout by reinserting masked tokens at stored indices. Additionally, the final reconstruction stage converts the text into modern orthographies via grapheme-to-phoneme rules, producing output that is directly useful to the community.
Load-bearing premise
The reported error reductions rest on the assumption that the handful of gold-reference pages used for evaluation represent the full five-book collection, including the difficult layouts with inline figures and mixed columns.
Editorial extensions
If this is right
- The evaluated pages of both books become machine-readable at a character error rate around 0.15–0.18, a level that supports transliteration into U'mista and SD-72 orthographies.
- The digitized line numbers are preserved, so the texts can be cross-referenced against Boas's dictionary and grammar, creating the analog concordance the community requested.
- The five-book collection can be processed with the same pipeline, assuming the gold-reference pages are representative of the remaining pages.
- The masking approach is reusable for other multilingual legacy documents with parallel English translations.
- The pipeline enables downstream tools such as speech-to-text, word processing, and predictive typing for Kwak'wala.
Reading between the lines
- An open question the paper leaves implicit is whether the post-correction model trained on one book's data transfers to the other books without retuning; a quick test would be to train on Jesup 5.1 and evaluate on Kwakiutl 1909, and if the transfer holds, the pipeline's cost drops considerably.
- The reported 99.84% sentence-level accuracy of the binary language identifier suggests that in cleanly typed bilingual scans, language identification may substitute for explicit layout analysis, which could simplify similar digitization projects for other Indigenous languages with English parallel text.
- The evaluation bottleneck is expert manual correction to create gold references; quantifying correction time per page would make the pipeline's cost model explicit and help other projects decide whether the approach is worth adopting.
- A caution: the gold-reference pages were selected as 'representative' without a stated sampling procedure, so the reported error reductions should be re-estimated on a random sample of pages, especially those with inline figures and mixed single and double columns.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a digitization pipeline for five volumes of Boas-Hunt Kwak'wala legacy texts, combining Google Vision first-pass OCR with a fastText language identification model, masking of non-Kwak'wala content, a post-correction model based on Rijhwani et al. (2021), and a reconstruction stage that reinserts masked tokens and performs grapheme-to-phoneme transliteration. The central empirical claim, reported in Table 1 of Section 4.6, is that the full pipeline reduces character error rate from 0.43 to 0.18 on Jesup 5.1 and from 0.33 to 0.15 on Kwakiutl 1909, with structural error rate dropping from 25 to 2 and from 18 to 3, respectively. The paper frames the work as community-oriented, noting consultation with language programs and plans to release outputs with community consent.
Significance. If the reported error reductions are reliable, the paper would provide a valuable, low-cost recipe for making legacy Indigenous-language texts machine-readable, and the replication of the prior post-correction model before applying it is a commendable methodological step. The pipeline itself is simple and clearly described, and the emphasis on community consent and dissemination is a strength. However, the quantitative evidence for the central claim is currently under-specified: the evaluation in Section 4.6 rests on an unreported number of pages with no stated selection criteria, no error bars, and no explicit demonstration of separation from training or pretraining data. The claim is therefore plausible but not yet established at the level needed to support the paper's conclusions.
major comments (4)
- [§4.6 / Table 1] The load-bearing evaluation does not state how many gold-reference pages were annotated, how they were chosen, or whether they are representative of the five-book collection described in Section 2. The text says only 'a few representative sample pages for two books.' It also does not report whether these pages cover the difficult layouts (inline figures, alternating single/double columns) that motivated the masking and reconstruction steps. Without this information, the CER/SER reductions in Table 1 cannot be generalized to the full collection, and the reader cannot assess whether the improvement would persist on harder pages.
- [§4.4 / §4.6] The paper does not establish that the evaluation pages are disjoint from the labeled Boas-Hunt training data and the unlabeled first-pass OCR pretraining data used in Section 4.4. Since the two evaluated books are Boas-Hunt texts and the shared dataset from Rijhwani et al. (2021) is drawn from the same collection, overlap is plausible. If the evaluation pages or their first-pass OCR outputs were used in training or pretraining, the reported error reductions could reflect memorization rather than generalization. The Limitations section acknowledges the small sample but does not address this train/evaluation separation issue, which is a distinct and more load-bearing risk. Please state explicitly how the evaluation pages were held out from all training and pretraining stages.
- [§4.6] The gold references are created by manually correcting the post-corrected output, rather than by independently transcribing the source image. This procedure can bias the evaluation in favor of the pipeline: any error in the post-corrected output that the annotator overlooks is inherited by the gold reference and therefore never counted as an error. The paper should describe the annotation protocol in more detail, including whether a second annotator reviewed corrections, whether annotators worked blind to the pipeline stage, and whether any inter-annotator agreement was measured. Without this, the corrected CER/SER numbers may be optimistically biased.
- [§4.2 / §4.6] The fastText language identification model is reported to achieve 99.84% sentence-level accuracy, but the evaluation that supports this figure is not described: there is no mention of a held-out test set, its size, its provenance, or its overlap with the pages evaluated in Table 1. Since langID errors propagate into the masking and reconstruction stages, a more fully specified evaluation of langID is needed to interpret the end-to-end results.
minor comments (5)
- [§1 / §4.5] The term 'mixed-methods' in the title is not reflected in the body of the paper: the work is an end-to-end NLP pipeline with a community consultation component, but no formal qualitative/quantitative mixed-methods analysis is presented. Consider either revising the title or adding a brief description of how community feedback was systematically incorporated.
- [§4.6] The structural error metric from Kanai et al. (1995) is described as a weighted sum of insertions, deletions, and moves, but the weighting scheme and the normalization to 0-100 are not specified. Please provide the exact formula or a citation to the precise variant used.
- [§1] There is a typo 'over a century of of legacy documentation' in the first paragraph; please correct it.
- [References] The references list contains a verbatim duplicate of Berg-Kirkpatrick et al. (2013); one copy should be removed.
- [Footnote 1] Footnote 1 says 'Relevant code and data resources are available here' but no URL or repository identifier is provided. Please add a permanent link or a statement that resources will be released upon acceptance.
Circularity Check
Table 1's error reductions are not shown to be on unseen pages: the post-correction model's pretraining data includes the evaluated collection, so the reported improvements may reduce to memorization.
-
fitted input called prediction
[Section 4.4 (Post-Correction) and Section 4.6 (Evaluation), Table 1]
"We train the model from scratch on the labeled Boas-Hunt dataset shared in the paper, with pre-training conducted on the unlabeled first-pass OCR outputs for the collection. ... So, for the moment, we evaluate on a few representative sample pages for two books."
The post-correction model is pretrained on unlabeled first-pass OCR outputs for the entire collection, which includes the two books used for evaluation (Jesup 5.1 and Kwakiutl 1909, per Section 2). The paper does not state that the evaluation pages are excluded from this pretraining data or from the labeled Boas-Hunt dataset. Therefore, the model's parameters have been exposed to the exact first-pass OCR outputs it is later evaluated on. The Table 1 CER/SER reductions are thus not demonstrated to be predictions on unseen pages; they are at least partially determined by the model's memorization of the training data, making the reported improvement statistically forced by construction.
-
self definitional
[Section 4.6 (Evaluation)]
"Gold reference pages are created by inspecting the post-corrected output, comparing it with the source image, and manually correcting any errors."
The gold reference is produced by editing the system's own post-corrected output. This makes the reference text dependent on the pipeline's output rather than an independent ground truth. Any systematic error that the human corrector fails to notice (for example, a consistent misrecognition of a diacritic) remains in the gold and is treated as correct, artificially lowering the reported error rates. The evaluation metric is therefore defined in terms of the very output it is meant to validate, undermining the claim of a 50% CER reduction as an objective measure of quality.
full rationale
The paper's central empirical claim is the Table 1 reduction in character error rate and structural error rate after applying the proposed pipeline. The derivation chain for this claim relies on the post-correction model described in Section 4.4, which is trained on the labeled Boas-Hunt dataset and pretrained on first-pass OCR outputs for the entire collection. The evaluation in Section 4.6 uses 'a few representative sample pages for two books' from that same collection, with no stated data split excluding these pages from training. By the paper's own description, the model has therefore likely seen the evaluation pages during pretraining, and possibly during supervised training if the labeled dataset overlaps. This means the reported improvements are not established as predictions on unseen data; they may reduce to memorization of the training input. Additionally, the gold references are constructed by correcting the post-corrected output, so the evaluation metric is not independent of the system being evaluated. The self-citation to Rijhwani et al. (2021) is not load-bearing because the model is published and the authors state they replicated its results; this does not add circularity. However, the train/eval overlap and self-referential gold construction directly affect the central claim, warranting a score of 6. Other components (langID, masking, reconstruction) are not circular in themselves, but their contribution is not independently evaluated outside this compromised framework.
Assumptions & free parameters
free parameters (1)
- langID training sentences per class =
1000 sentences per language
assumptions (3)
- domain assumption Google Vision's first-pass OCR output on these Boas-Hunt scans is good enough for language identification and post-correction to improve upon it.
- domain assumption The post-correction model from Rijhwani et al. (2021), trained on the shared Boas-Hunt dataset, transfers to the newly processed books.
- ad hoc to paper A few manually corrected gold-reference pages are sufficient to estimate CER and SER for entire books.
Cite this review
Pith. "Pith review of Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts." pith.science (2026). https://pith.science/paper/BDBD6YNK
@misc{pith2026250601775,
author = {Pith},
title = {Pith review of: Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/BDBD6YNK}},
note = {Machine review of arXiv:2506.01775}
}
read the original abstract
Kwak'wala is an Indigenous language spoken in British Columbia, with a rich legacy of published documentation spanning more than a century, and an active community of speakers, teachers, and learners engaged in language revitalization. Over 11 volumes of the earliest texts created during the collaboration between Franz Boas and George Hunt have been scanned but remain unreadable by machines. Complete digitization through optical character recognition has the potential to facilitate transliteration into modern orthographies and the creation of other language technologies. In this paper, we apply the latest OCR techniques to a series of Kwak'wala texts only accessible as images, and discuss the challenges and unique adaptations necessary to make such technologies work for these real-world texts. Building on previous methods, we propose using a mix of off-the-shelf OCR methods, language identification, and masking to effectively isolate Kwak'wala text, along with post-correction models, to produce a final high-quality transcription.
Figures
Reference graph
Works this paper leans on
-
[1]
Milind Agarwal, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.895 LIMIT : Language identification, misidentification, and translation using hierarchical models in 350+ languages . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14496--14519, Singapore. As...
-
[2]
Milind Agarwal and Antonios Anastasopoulos. 2024. https://doi.org/10.18653/v1/2024.americasnlp-1.10 A concise survey of OCR for low-resource languages . In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), pages 88--102, Mexico City, Mexico. Association for Computational Linguistics
-
[3]
Taylor Berg - Kirkpatrick, Greg Durrett, and Dan Klein. 2013. https://aclanthology.org/P13-1021/ Unsupervised transcription of historical documents . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers , pages 207--217. The Association for Computer Li...
work page 2013
-
[4]
Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. https://aclanthology.org/P13-1021 Unsupervised transcription of historical documents . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 207--217, Sofia, Bulgaria. Association for Computational Linguistics
work page 2013
-
[5]
Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. https://arxiv.org/abs/2308.13418 Nougat: Neural optical understanding for academic documents . Preprint, arXiv:2308.13418
arXiv 2023
-
[6]
Franz Boas. 1909. The Kwakiutl of Vancouver Island. Leiden, New York: E.J. Brill; G.E. Stechert & Co
work page 1909
-
[7]
Franz Boas and George Hunt. 1902 a . Volume 5, Part 1. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co
work page 1902
-
[8]
Franz Boas and George Hunt. 1902 b . Volume 5, Part 2. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co
work page 1902
Show all 31 references
-
[9]
Franz Boas and George Hunt. 1902 c . Volume 5, Part 3. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co
1902
-
[10]
Franz Boas and George Hunt. 1906. Jesup North Pacific Expedition - Kwakiutl Texts, Second Series, Volume 10. Leiden, New York: E.J. Brill; G.E. Stechert & Co
1906
-
[11]
Johanna Cordova and Damien Nouvel. 2021. https://doi.org/10.18653/v1/2021.americasnlp-1.18 Toward creation of A ncash lexical resources from OCR . In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 163--167, Onli...
2021 doi
-
[12]
Rui Dong and David Smith. 2018. https://doi.org/10.18653/v1/P18-1220 Multi-input attention for unsupervised OCR correction . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2363--2372, Melbourne, Austral...
2018 doi
-
[13]
Yasuhisa Fujii, Karel Driesen, Jonathan Baccash, Ash Hurst, and Ashok C. Popat. 2017. https://doi.org/10.1109/ICDAR.2017.35 Sequence-to-label script identification for multilingual OCR . In 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Ky...
2017 doi
-
[14]
Lenore A Grenoble and Lindsay J Whaley. 2005. Saving languages: An introduction to language revitalization. Cambridge University Press
2005
-
[15]
Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow
Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow. 2021. https://doi.org/10.18653/v1/2021.naacl-main.201 A survey on recent approaches for natural language processing in low-resource scenarios . In Proceedings of the 2021 Conference of the...
2021 doi
-
[16]
Nikolaus P Himmelmann. 1998. Documentary and descriptive linguistics
1998
-
[17]
Benjamin Hunt, Lane Schwartz, Sylvia Schreiner, and Emily Chen. 2023. https://doi.org/10.18653/v1/2023.americasnlp-1.14 Community consultation and the development of an online akuzipik- E nglish dictionary . In Proceedings of the Workshop on Natural Language Processing for Ind...
2023 doi
-
[18]
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. https://aclanthology.org/E17-2068 Bag of tricks for efficient text classification . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume ...
2017
-
[19]
Kanai, S.V
J. Kanai, S.V. Rice, T.A. Nartker, and G. Nagy. 1995. https://doi.org/10.1109/34.368146 Automated evaluation of ocr zoning . IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1):86--90
1995 doi
-
[20]
Okan Kolak and Philip Resnik. 2005. https://aclanthology.org/H05-1109 OCR post-processing for low density languages . In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 867--874, Vancouver, British C...
2005
-
[21]
Kimberley L. Lawson. 2004. https://doi.org/http://dx.doi.org/10.14288/1.0091657 Precious fragments: First Nations materials in archives, libraries and museums . Ph.D. thesis, University of British Columbia
2004 doi
-
[22]
Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin. 2020. https://doi.org/10.1109/ACCESS.2020.3012542 Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr) . IEEE Access, 8:142642--142668
2020
-
[23]
Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. https://doi.org/10.1145/3476887.3476888 A survey of ocr evaluation tools and metrics . In Proceedings of the 6th International Workshop on Historica...
2021
-
[24]
Thi Tuyet Hai Nguyen, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. 2021. https://doi.org/10.1145/3453476 Survey of post-ocr processing approaches . ACM Comput. Surv., 54(6)
2021 doi
-
[25]
Aidan Pine, Patrick William Littell, Eric Joanis, David Huggins-Daines, Christopher Cox, Fineen Davis, Eddie Antonio Santos, Shankhalika Srikanth, Delasie Torkornoo, and Sabrina Yu. 2022. https://doi.org/10.18653/v1/2022.computel-1.7 G _i 2 P _i rule-based, index-preserving gr...
2022 doi
-
[26]
Christian Reul, Uwe Springmann, and Frank Puppe. 2017. https://arxiv.org/abs/1701.07396 LAREX - A semi-automatic open-source tool for layout analysis and region extraction on early printed books . CoRR, abs/1701.07396
2017 arXiv
-
[27]
Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.478 OCR P ost C orrection for E ndangered L anguage T exts . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages ...
2020 doi
-
[28]
Shruti Rijhwani, Daisy Rosenblum, Antonios Anastasopoulos, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00427 Lexically aware semi-supervised learning for OCR post-correction . Transactions of the Association for Computational Linguistics, 9:1285--1302
2021 doi
-
[29]
Shruti Rijhwani, Daisy Rosenblum, Michayla King, Antonios Anastasopoulos, and Graham Neubig. 2023. https://aclanthology.org/2023.computel-1.4 User-centric evaluation of OCR systems for kwak ' wala . In Proceedings of the Sixth Workshop on the Use of Computational Methods in th...
2023
-
[30]
Kevin Schlottmann. 2023. https://blogs.library.columbia.edu/rbml/2023/04/04/george-hunt-kwakwala-manuscripts/ Description and digitization of the george hunt kwak’wala ethnographic manuscripts . Accessed: 2025-01-10
2023
-
[31]
R. Smith. 2007. https://doi.org/10.1109/ICDAR.2007.4376991 An overview of the tesseract OCR engine . In 9th International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September, Curitiba, Paran \' a , Brazil , pages 629--633. IEEE Computer Society
2007
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.