Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The pipeline reduces character error rate from 0.43 to 0.18 and from 0.33 to 0.15 on the two evaluated Kwak'wala books, and structural error rate from 25 to 2 and from 18 to 3.

desk verdict A genuinely useful digitization pipeline for Kwak'wala legacy texts, but the headline error reductions rest on an underspecified evaluation sample that may overlap with the post-correction model's training data. read the letter →

arxiv 2506.01775 v1 pith:BDBD6YNK submitted 2025-06-02 cs.CL

classification cs.CL
keywords Kwak'walaOCRpost-correctionlanguageidentificationmaskinglegacytextsIndigenousdigitizationendangeredlanguagesorthographytransliteration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a pipeline built from existing OCR components can make century-old scanned Kwak'wala texts usable by machines. On the two books evaluated, character error rate drops from 0.43 to 0.18 (Jesup 5.1) and from 0.33 to 0.15 (Kwakiutl 1909), while structural error rate drops from 25 to 2 and from 18 to 3. The components are first-pass OCR, binary language identification trained on Kwak'wala and English, masking of non-Kwak'wala tokens, a post-correction model, and reconstruction that reinserts the masked tokens. The goal is to give the Kwak'wala community and researchers machine-readable text that can be transliterated into modern orthographies and used for language technology.

What carries the argument

The central mechanism is the masking–post-correction–reconstruction loop. A binary fastText language identification model (trained on 1000 sentences per language) labels each OCR bounding box as English or Kwak'wala; the masking layer then hides English text, numbers, and parentheses, storing token-level indices so they can be reinserted after correction. The post-correction model is the multi-source neural architecture of Rijhwani et al. (2021), trained from scratch on the labeled Boas-Hunt dataset and pretrained on unlabeled first-pass outputs. By isolating Kwak'wala text, the pipeline limits post-correction to the language that needs it, avoiding damage to already-good English OCR.

What would settle it

Take a random sample of pages from all five books, including pages with inline figures, single and double columns, and the vocabulary appendices; have an expert produce gold references; run the pipeline; and compare CER and SER. If the error reductions are substantially smaller on that sample than the 0.18/0.15 CER and 2/3 SER reported, the central claim is overstated.

Watch

Extended reading notes

Core claim

On the book pages tested, the proposed mixed-methods pipeline reduces the character error rate by roughly half relative to raw first-pass OCR and cuts the structural error rate by about 87.5%, bringing the digitized text close to gold-reference quality. The discovery is that language identification can serve as a structure proxy in a multilingual legacy document: by labeling each bounding box as English or Kwak'wala, the pipeline can mask non-Kwak'wala content, run post-correction only on the Kwak'wala text where errors concentrate, and then reconstruct the original layout by reinserting masked tokens at stored indices. Additionally, the final reconstruction stage converts the text into modern orthographies via grapheme-to-phoneme rules, producing output that is directly useful to the community.

Load-bearing premise

The reported error reductions rest on the assumption that the handful of gold-reference pages used for evaluation represent the full five-book collection, including the difficult layouts with inline figures and mixed columns.

Editorial extensions

If this is right

  • The evaluated pages of both books become machine-readable at a character error rate around 0.15–0.18, a level that supports transliteration into U'mista and SD-72 orthographies.
  • The digitized line numbers are preserved, so the texts can be cross-referenced against Boas's dictionary and grammar, creating the analog concordance the community requested.
  • The five-book collection can be processed with the same pipeline, assuming the gold-reference pages are representative of the remaining pages.
  • The masking approach is reusable for other multilingual legacy documents with parallel English translations.
  • The pipeline enables downstream tools such as speech-to-text, word processing, and predictive typing for Kwak'wala.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open question the paper leaves implicit is whether the post-correction model trained on one book's data transfers to the other books without retuning; a quick test would be to train on Jesup 5.1 and evaluate on Kwakiutl 1909, and if the transfer holds, the pipeline's cost drops considerably.
  • The reported 99.84% sentence-level accuracy of the binary language identifier suggests that in cleanly typed bilingual scans, language identification may substitute for explicit layout analysis, which could simplify similar digitization projects for other Indigenous languages with English parallel text.
  • The evaluation bottleneck is expert manual correction to create gold references; quantifying correction time per page would make the pipeline's cost model explicit and help other projects decide whether the approach is worth adopting.
  • A caution: the gold-reference pages were selected as 'representative' without a stated sampling procedure, so the reported error reductions should be re-estimated on a random sample of pages, especially those with inline figures and mixed single and double columns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a digitization pipeline for five volumes of Boas-Hunt Kwak'wala legacy texts, combining Google Vision first-pass OCR with a fastText language identification model, masking of non-Kwak'wala content, a post-correction model based on Rijhwani et al. (2021), and a reconstruction stage that reinserts masked tokens and performs grapheme-to-phoneme transliteration. The central empirical claim, reported in Table 1 of Section 4.6, is that the full pipeline reduces character error rate from 0.43 to 0.18 on Jesup 5.1 and from 0.33 to 0.15 on Kwakiutl 1909, with structural error rate dropping from 25 to 2 and from 18 to 3, respectively. The paper frames the work as community-oriented, noting consultation with language programs and plans to release outputs with community consent.

Significance. If the reported error reductions are reliable, the paper would provide a valuable, low-cost recipe for making legacy Indigenous-language texts machine-readable, and the replication of the prior post-correction model before applying it is a commendable methodological step. The pipeline itself is simple and clearly described, and the emphasis on community consent and dissemination is a strength. However, the quantitative evidence for the central claim is currently under-specified: the evaluation in Section 4.6 rests on an unreported number of pages with no stated selection criteria, no error bars, and no explicit demonstration of separation from training or pretraining data. The claim is therefore plausible but not yet established at the level needed to support the paper's conclusions.

major comments (4)
  1. [§4.6 / Table 1] The load-bearing evaluation does not state how many gold-reference pages were annotated, how they were chosen, or whether they are representative of the five-book collection described in Section 2. The text says only 'a few representative sample pages for two books.' It also does not report whether these pages cover the difficult layouts (inline figures, alternating single/double columns) that motivated the masking and reconstruction steps. Without this information, the CER/SER reductions in Table 1 cannot be generalized to the full collection, and the reader cannot assess whether the improvement would persist on harder pages.
  2. [§4.4 / §4.6] The paper does not establish that the evaluation pages are disjoint from the labeled Boas-Hunt training data and the unlabeled first-pass OCR pretraining data used in Section 4.4. Since the two evaluated books are Boas-Hunt texts and the shared dataset from Rijhwani et al. (2021) is drawn from the same collection, overlap is plausible. If the evaluation pages or their first-pass OCR outputs were used in training or pretraining, the reported error reductions could reflect memorization rather than generalization. The Limitations section acknowledges the small sample but does not address this train/evaluation separation issue, which is a distinct and more load-bearing risk. Please state explicitly how the evaluation pages were held out from all training and pretraining stages.
  3. [§4.6] The gold references are created by manually correcting the post-corrected output, rather than by independently transcribing the source image. This procedure can bias the evaluation in favor of the pipeline: any error in the post-corrected output that the annotator overlooks is inherited by the gold reference and therefore never counted as an error. The paper should describe the annotation protocol in more detail, including whether a second annotator reviewed corrections, whether annotators worked blind to the pipeline stage, and whether any inter-annotator agreement was measured. Without this, the corrected CER/SER numbers may be optimistically biased.
  4. [§4.2 / §4.6] The fastText language identification model is reported to achieve 99.84% sentence-level accuracy, but the evaluation that supports this figure is not described: there is no mention of a held-out test set, its size, its provenance, or its overlap with the pages evaluated in Table 1. Since langID errors propagate into the masking and reconstruction stages, a more fully specified evaluation of langID is needed to interpret the end-to-end results.
minor comments (5)
  1. [§1 / §4.5] The term 'mixed-methods' in the title is not reflected in the body of the paper: the work is an end-to-end NLP pipeline with a community consultation component, but no formal qualitative/quantitative mixed-methods analysis is presented. Consider either revising the title or adding a brief description of how community feedback was systematically incorporated.
  2. [§4.6] The structural error metric from Kanai et al. (1995) is described as a weighted sum of insertions, deletions, and moves, but the weighting scheme and the normalization to 0-100 are not specified. Please provide the exact formula or a citation to the precise variant used.
  3. [§1] There is a typo 'over a century of of legacy documentation' in the first paragraph; please correct it.
  4. [References] The references list contains a verbatim duplicate of Berg-Kirkpatrick et al. (2013); one copy should be removed.
  5. [Footnote 1] Footnote 1 says 'Relevant code and data resources are available here' but no URL or repository identifier is provided. Please add a permanent link or a statement that resources will be released upon acceptance.

Circularity Check

2 steps flagged · score 6.0 of 10

Table 1's error reductions are not shown to be on unseen pages: the post-correction model's pretraining data includes the evaluated collection, so the reported improvements may reduce to memorization.

  1. fitted input called prediction [Section 4.4 (Post-Correction) and Section 4.6 (Evaluation), Table 1]
    "We train the model from scratch on the labeled Boas-Hunt dataset shared in the paper, with pre-training conducted on the unlabeled first-pass OCR outputs for the collection. ... So, for the moment, we evaluate on a few representative sample pages for two books."

    The post-correction model is pretrained on unlabeled first-pass OCR outputs for the entire collection, which includes the two books used for evaluation (Jesup 5.1 and Kwakiutl 1909, per Section 2). The paper does not state that the evaluation pages are excluded from this pretraining data or from the labeled Boas-Hunt dataset. Therefore, the model's parameters have been exposed to the exact first-pass OCR outputs it is later evaluated on. The Table 1 CER/SER reductions are thus not demonstrated to be predictions on unseen pages; they are at least partially determined by the model's memorization of the training data, making the reported improvement statistically forced by construction.

  2. self definitional [Section 4.6 (Evaluation)]
    "Gold reference pages are created by inspecting the post-corrected output, comparing it with the source image, and manually correcting any errors."

    The gold reference is produced by editing the system's own post-corrected output. This makes the reference text dependent on the pipeline's output rather than an independent ground truth. Any systematic error that the human corrector fails to notice (for example, a consistent misrecognition of a diacritic) remains in the gold and is treated as correct, artificially lowering the reported error rates. The evaluation metric is therefore defined in terms of the very output it is meant to validate, undermining the claim of a 50% CER reduction as an objective measure of quality.

full rationale

The paper's central empirical claim is the Table 1 reduction in character error rate and structural error rate after applying the proposed pipeline. The derivation chain for this claim relies on the post-correction model described in Section 4.4, which is trained on the labeled Boas-Hunt dataset and pretrained on first-pass OCR outputs for the entire collection. The evaluation in Section 4.6 uses 'a few representative sample pages for two books' from that same collection, with no stated data split excluding these pages from training. By the paper's own description, the model has therefore likely seen the evaluation pages during pretraining, and possibly during supervised training if the labeled dataset overlaps. This means the reported improvements are not established as predictions on unseen data; they may reduce to memorization of the training input. Additionally, the gold references are constructed by correcting the post-corrected output, so the evaluation metric is not independent of the system being evaluated. The self-citation to Rijhwani et al. (2021) is not load-bearing because the model is published and the authors state they replicated its results; this does not add circularity. However, the train/eval overlap and self-referential gold construction directly affect the central claim, warranting a score of 6. Other components (langID, masking, reconstruction) are not circular in themselves, but their contribution is not independently evaluated outside this compromised framework.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities or fitted physical parameters are introduced. The pipeline depends on off-the-shelf OCR and a previously published post-correction model, plus a handful of design choices; the fragile assumptions are about data representativeness and model transfer.

free parameters (1)
  • langID training sentences per class = 1000 sentences per language
    Chosen by the authors for training the fastText language identifier; not fitted to the target CER or SER results, but it affects masking accuracy and hence downstream corrections.
assumptions (3)
  • domain assumption Google Vision's first-pass OCR output on these Boas-Hunt scans is good enough for language identification and post-correction to improve upon it.
    The entire pipeline assumes the first-pass quality permits reliable langID and that post-correction errors dominate; introduced in Sections 4.1 and 4.4.
  • domain assumption The post-correction model from Rijhwani et al. (2021), trained on the shared Boas-Hunt dataset, transfers to the newly processed books.
    The paper trains the model on that dataset and then applies it to the five-volume collection (Section 4.4) without reporting per-book fine-tuning or transfer validation.
  • ad hoc to paper A few manually corrected gold-reference pages are sufficient to estimate CER and SER for entire books.
    Section 4.6 states evaluation is on 'a few representative sample pages' without specifying count or sampling, yet Table 1 reports collection-level conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts." pith.science (2026). https://pith.science/paper/BDBD6YNK

@misc{pith2026250601775,
  author       = {Pith},
  title        = {Pith review of: Developing a Mixed-Methods Pipeline for Community-Oriented Digitization of Kwak'wala Legacy Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDBD6YNK}},
  note         = {Machine review of arXiv:2506.01775}
}
read the original abstract

Kwak'wala is an Indigenous language spoken in British Columbia, with a rich legacy of published documentation spanning more than a century, and an active community of speakers, teachers, and learners engaged in language revitalization. Over 11 volumes of the earliest texts created during the collaboration between Franz Boas and George Hunt have been scanned but remain unreadable by machines. Complete digitization through optical character recognition has the potential to facilitate transliteration into modern orthographies and the creation of other language technologies. In this paper, we apply the latest OCR techniques to a series of Kwak'wala texts only accessible as images, and discuss the challenges and unique adaptations necessary to make such technologies work for these real-world texts. Building on previous methods, we propose using a mix of off-the-shelf OCR methods, language identification, and masking to effectively isolate Kwak'wala text, along with post-correction models, to produce a final high-quality transcription.

Figures

Figures reproduced from arXiv: 2506.01775 by the authors.

Figure 1
Figure 1. Example two-column text from the Kwakiutl of Vancouver Island (1909) collection. Notice the abundance [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    Milind Agarwal, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.895 LIMIT : Language identification, misidentification, and translation using hierarchical models in 350+ languages . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14496--14519, Singapore. As...

  2. [2]

    Milind Agarwal and Antonios Anastasopoulos. 2024. https://doi.org/10.18653/v1/2024.americasnlp-1.10 A concise survey of OCR for low-resource languages . In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), pages 88--102, Mexico City, Mexico. Association for Computational Linguistics

  3. [3]

    Taylor Berg - Kirkpatrick, Greg Durrett, and Dan Klein. 2013. https://aclanthology.org/P13-1021/ Unsupervised transcription of historical documents . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers , pages 207--217. The Association for Computer Li...

  4. [4]

    Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. https://aclanthology.org/P13-1021 Unsupervised transcription of historical documents . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 207--217, Sofia, Bulgaria. Association for Computational Linguistics

  5. [5]

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. https://arxiv.org/abs/2308.13418 Nougat: Neural optical understanding for academic documents . Preprint, arXiv:2308.13418

  6. [6]

    Franz Boas. 1909. The Kwakiutl of Vancouver Island. Leiden, New York: E.J. Brill; G.E. Stechert & Co

  7. [7]

    Franz Boas and George Hunt. 1902 a . Volume 5, Part 1. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co

  8. [8]

    Franz Boas and George Hunt. 1902 b . Volume 5, Part 2. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co

Show all 31 references
  1. [9]

    Franz Boas and George Hunt. 1902 c . Volume 5, Part 3. Kwakiutl Texts - Memoirs of The American Museum of Natural History. Leiden, New York: E.J. Brill; G.E. Stechert & Co

  2. [10]

    Franz Boas and George Hunt. 1906. Jesup North Pacific Expedition - Kwakiutl Texts, Second Series, Volume 10. Leiden, New York: E.J. Brill; G.E. Stechert & Co

  3. [11]

    Johanna Cordova and Damien Nouvel. 2021. https://doi.org/10.18653/v1/2021.americasnlp-1.18 Toward creation of A ncash lexical resources from OCR . In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 163--167, Onli...

  4. [12]

    Rui Dong and David Smith. 2018. https://doi.org/10.18653/v1/P18-1220 Multi-input attention for unsupervised OCR correction . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2363--2372, Melbourne, Austral...

  5. [13]

    Yasuhisa Fujii, Karel Driesen, Jonathan Baccash, Ash Hurst, and Ashok C. Popat. 2017. https://doi.org/10.1109/ICDAR.2017.35 Sequence-to-label script identification for multilingual OCR . In 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Ky...

  6. [14]

    Lenore A Grenoble and Lindsay J Whaley. 2005. Saving languages: An introduction to language revitalization. Cambridge University Press

  7. [15]

    Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow

    Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow. 2021. https://doi.org/10.18653/v1/2021.naacl-main.201 A survey on recent approaches for natural language processing in low-resource scenarios . In Proceedings of the 2021 Conference of the...

  8. [16]

    Nikolaus P Himmelmann. 1998. Documentary and descriptive linguistics

  9. [17]

    Benjamin Hunt, Lane Schwartz, Sylvia Schreiner, and Emily Chen. 2023. https://doi.org/10.18653/v1/2023.americasnlp-1.14 Community consultation and the development of an online akuzipik- E nglish dictionary . In Proceedings of the Workshop on Natural Language Processing for Ind...

  10. [18]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. https://aclanthology.org/E17-2068 Bag of tricks for efficient text classification . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume ...

  11. [19]

    Kanai, S.V

    J. Kanai, S.V. Rice, T.A. Nartker, and G. Nagy. 1995. https://doi.org/10.1109/34.368146 Automated evaluation of ocr zoning . IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1):86--90

  12. [20]

    Okan Kolak and Philip Resnik. 2005. https://aclanthology.org/H05-1109 OCR post-processing for low density languages . In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 867--874, Vancouver, British C...

  13. [21]

    Kimberley L. Lawson. 2004. https://doi.org/http://dx.doi.org/10.14288/1.0091657 Precious fragments: First Nations materials in archives, libraries and museums . Ph.D. thesis, University of British Columbia

  14. [22]

    Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin. 2020. https://doi.org/10.1109/ACCESS.2020.3012542 Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr) . IEEE Access, 8:142642--142668

  15. [23]

    Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. https://doi.org/10.1145/3476887.3476888 A survey of ocr evaluation tools and metrics . In Proceedings of the 6th International Workshop on Historica...

  16. [24]

    Thi Tuyet Hai Nguyen, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. 2021. https://doi.org/10.1145/3453476 Survey of post-ocr processing approaches . ACM Comput. Surv., 54(6)

  17. [25]

    Aidan Pine, Patrick William Littell, Eric Joanis, David Huggins-Daines, Christopher Cox, Fineen Davis, Eddie Antonio Santos, Shankhalika Srikanth, Delasie Torkornoo, and Sabrina Yu. 2022. https://doi.org/10.18653/v1/2022.computel-1.7 G _i 2 P _i rule-based, index-preserving gr...

  18. [26]

    Christian Reul, Uwe Springmann, and Frank Puppe. 2017. https://arxiv.org/abs/1701.07396 LAREX - A semi-automatic open-source tool for layout analysis and region extraction on early printed books . CoRR, abs/1701.07396

  19. [27]

    Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.478 OCR P ost C orrection for E ndangered L anguage T exts . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages ...

  20. [28]

    Shruti Rijhwani, Daisy Rosenblum, Antonios Anastasopoulos, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00427 Lexically aware semi-supervised learning for OCR post-correction . Transactions of the Association for Computational Linguistics, 9:1285--1302

  21. [29]

    Shruti Rijhwani, Daisy Rosenblum, Michayla King, Antonios Anastasopoulos, and Graham Neubig. 2023. https://aclanthology.org/2023.computel-1.4 User-centric evaluation of OCR systems for kwak ' wala . In Proceedings of the Sixth Workshop on the Use of Computational Methods in th...

  22. [30]

    Kevin Schlottmann. 2023. https://blogs.library.columbia.edu/rbml/2023/04/04/george-hunt-kwakwala-manuscripts/ Description and digitization of the george hunt kwak’wala ethnographic manuscripts . Accessed: 2025-01-10

  23. [31]

    R. Smith. 2007. https://doi.org/10.1109/ICDAR.2007.4376991 An overview of the tesseract OCR engine . In 9th International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September, Curitiba, Paran \' a , Brazil , pages 629--633. IEEE Computer Society

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.