REVIEW 4 major objections 4 minor 46 references
Advancing Uto-Aztecan Language Technologies: A Case Study on the Endangered Comanche Language
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Three examples teach GPT-4o to identify Comanche at 100 percent.
desk verdict The Comanche dataset is a real contribution; the few-shot accuracy claim is not yet supported and needs a clean re-evaluation before it means anything. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is in-context few-shot prompting: before asking the model to classify a phrase, the prompt supplies one to three Comanche phrases paired with randomized English translations drawn from the same dataset. These pairs teach the model the orthographic and morphological shape of Comanche, letting it recognize the language without fine-tuning. The paper also uses the same few-shot prompting to generate synthetic Comanche translations, filtered by a normalized Levenshtein similarity threshold, to expand the dataset.
What would settle it
Re-run the three-shot identification experiment with a test set built from Comanche phrases that are provably disjoint from the few-shot examples, and repeat with different random example draws. If accuracy falls well below 100 percent in any disjoint condition, the reported result reflected memorization of the prompt examples rather than generalization to Comanche.
Extended reading notes
Core claim
The paper's central claim is that a small number of Comanche-English example pairs placed in the prompt is enough for GPT-4o to identify Comanche text almost perfectly, even though the model fails without such examples. In zero-shot prompting, GPT-4o correctly identified 56 of 412 entries, or 13.5 percent. Adding one Comanche example raised accuracy to 91 percent, and a three-shot prompt produced 100 percent accuracy on the 100-entry evaluation sample. English identification stayed at 97-100 percent across conditions. On the paper's terms, this shows that minimal targeted prompting can overcome the default exclusion of an endangered language from LLM-based identification.
Load-bearing premise
The central result assumes that the Comanche phrases used as few-shot examples are never the same as the phrases in the 100-item test set; if a phrase appears in both, the reported near-perfect accuracy could be the model repeating in-context examples rather than recognizing unseen Comanche.
Editorial extensions
If this is right
- If three examples suffice, any endangered language with a few documented phrases could be made identifiable to LLMs at near-zero cost.
- The same few-shot approach could extend to other Uto-Aztecan and polysynthetic languages that currently lack language-identification support.
- The synthetic data pipeline offers a route to bootstrap parallel corpora for downstream tasks such as machine translation and language-learning tools.
- The result suggests the main barrier to including endangered languages in NLP is data visibility, not model capability.
Reading between the lines
- A stress test that varies the random draw of few-shot examples and test items across repeated trials would show whether 100 percent accuracy is stable or an artifact of a favorable sample.
- Comanche is polysynthetic, so the few-shot examples may teach surface spelling patterns rather than grammar; a version of the prompt that also supplies a small morpheme or pronunciation lexicon might generalize better to unseen word forms.
- Because the synthetic pipeline accepts translations at a low similarity threshold (0.11), testing identification separately on synthetic versus human-curated Comanche would clarify whether the few-shot benefit comes from real linguistic signal or from prompt artifacts.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a first computational case study of Comanche, an endangered Uto-Aztecan language. The authors contribute a manually curated dataset of 412 Comanche-English phrases, a synthetic data generation pipeline that uses GPT-4o with normalized Levenshtein filtering, and zero-shot and few-shot language identification experiments on GPT-4o and, per the abstract, GPT-4o-mini. The main empirical claim is that GPT-4o identifies Comanche poorly in the zero-shot setting (13.5% accuracy) but dramatically better with a few in-context examples, reaching 91% one-shot and 100% three-shot accuracy. The paper also reports a community interview and advocates for community-informed, low-cost NLP for endangered languages.
Significance. The dataset and the focus on Comanche are genuinely valuable: there is essentially no prior NLP resource for this endangered language, and the community-engagement framing is a positive contribution. If the few-shot result were validated on a clean held-out set, it would provide a useful data point for low-resource language identification. The authors also state that the dataset and code are publicly available, which supports reproducibility. However, the central empirical claim is currently under-supported because the few-shot evaluation does not document a separation between the in-context examples and the test entries, and the synthetic data pipeline's quality threshold is so low that it admits near-random strings.
major comments (4)
- [Section 5] The few-shot evaluation does not establish that the 100 test entries are disjoint from the few-shot examples. The text says the test sample consists of '100 randomly selected entries from our original dataset' and that 'each few-shot pair included one Comanche phrase and a randomized English entry from the dataset,' but it never states that the in-context examples were removed from the test pool. If the same 412-entry pool was used for both, the reported 100% three-shot accuracy may reflect the model reading the answer from the prompt rather than generalizing to unseen Comanche text. Please provide an explicit train/test split for the few-shot experiments and report accuracy on examples that were never shown in context.
- [Section 4.2] The synthetic data pipeline uses a normalized Levenshtein similarity threshold of 0.11, which is only marginally above the 'baseline threshold of 0.1' mentioned in the footnote. At this threshold, outputs that share almost no characters with the reference translation can be accepted, so the pipeline does not currently demonstrate that the generated Comanche is linguistically usable. The paper should report the distribution of similarity scores, show representative accepted and rejected examples, and justify why 0.11 is an appropriate quality bar.
- [Abstract / Section 5] The abstract claims 'near-perfect accuracy with just five examples,' while Section 5 reports 91% one-shot accuracy and '100% accuracy' with a three-shot strategy. These statements are inconsistent and should be reconciled, with the reported number of examples and the exact few-shot conditions stated precisely in both places.
- [Abstract / Section 5] The abstract states that the paper contains an 'empirical evaluation of GPT-4o and GPT-4o-mini,' but Section 5 reports results only for GPT-4o. No GPT-4o-mini results appear anywhere in the manuscript. Please either report the GPT-4o-mini results or revise the abstract and any related claims.
minor comments (4)
- [Section 4.2] The text contains an unresolved 'Appendix ??' reference, which should be corrected to the appropriate appendix.
- [References] The reference to Charney (1993) lists the title as 'A Grammer of Comanche'; this should be 'A Grammar of Comanche.'
- [Section 5 / Figure 5] The paper uses the word 'consistently' for the three-shot result but reports no repeated runs, variance, or confidence intervals; please provide standard deviations or run-level results across multiple random few-shot example selections.
- [Ethics Statement] The GitHub link provided for the dataset and code should include a version identifier or repository snapshot so that the exact released artifacts can be cited and verified.
Circularity Check
Few-shot language-ID evaluation draws examples and test items from the same dataset without documenting a held-out split, so the headline 91–100% accuracy is not established as a prediction.
-
fitted input called prediction
[Section 5, Language Identification (few-shot experiment description and results)]
"For this experiment, we used a sample of 100 randomly selected entries from our original dataset. Each few-shot pair included one Comanche phrase and a randomized English entry from the dataset, as shown in Figure 4. With just one Comanche example, GPT-4o achieved 91% accuracy in identification of Comanche. Extending to a three-shot strategy consistently yielded 100% accuracy, as shown in Figure 5."
The test sample and the few-shot examples are both drawn from the same 'original dataset,' and the paper never states that the few-shot items were excluded from the 100 test entries. For any test item that is also used as an in-context example, the correct Comanche label is provided verbatim in the prompt, so that item is not a prediction but a copy of the input. The reported 91%/100% figures are therefore not demonstrated to be held-out accuracies; the central few-shot claim is at least partly an artifact of the evaluation design. No separate train/test split for this experiment is described.
full rationale
The central load-bearing claim is that few-shot prompting gives GPT-4o near-perfect Comanche language identification. The paper's own protocol states that a sample of 100 entries from the original dataset is used for the test, while the few-shot pairs also come from that dataset, with no statement that the examples were removed from the test sample. For any overlap, the correct answer is in the prompt, so the reported accuracy is partly in-sample. The magnitude of this effect is limited (three examples out of 100 test items), so the broad conclusion is not wholly forced, but the specific near-perfect numbers are not a clean held-out prediction. I found no load-bearing self-citation: the Yang et al. (2025a,b) citations support background claims about digital exclusion and are not used to justify the experimental design. The synthetic data pipeline is also not circular: it generates translations and filters them with a Levenshtein threshold, which is an external (if weak) quality signal. The abstract/body discrepancy (five examples vs three-shot) is a reporting inconsistency, not circularity. Overall, the paper has one partially circular evaluation step, so score 6.
Assumptions & free parameters
free parameters (1)
- Levenshtein similarity threshold =
0.11
assumptions (3)
- domain assumption The 412 manually curated phrases are accurate Comanche-English alignments.
- domain assumption Normalized Levenshtein similarity is a meaningful proxy for translation quality.
- domain assumption The 100-entry subset used for few-shot evaluation is representative and independent from the few-shot examples.
Cite this review
Pith. "Pith review of Advancing Uto-Aztecan Language Technologies: A Case Study on the Endangered Comanche Language." pith.science (2026). https://pith.science/paper/RHWXPPJT
@misc{pith2026250518159,
author = {Pith},
title = {Pith review of: Advancing Uto-Aztecan Language Technologies: A Case Study on the Endangered Comanche Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/RHWXPPJT}},
note = {Machine review of arXiv:2505.18159}
}
read the original abstract
The digital exclusion of endangered languages remains a critical challenge in NLP, limiting both linguistic research and revitalization efforts. This study introduces the first computational investigation of Comanche, an Uto-Aztecan language on the verge of extinction, demonstrating how minimal-cost, community-informed NLP interventions can support language preservation. We present a manually curated dataset of 412 phrases, a synthetic data generation pipeline, and an empirical evaluation of GPT-4o and GPT-4o-mini for language identification. Our experiments reveal that while LLMs struggle with Comanche in zero-shot settings, few-shot prompting significantly improves performance, achieving near-perfect accuracy with just five examples. Our findings highlight the potential of targeted NLP methodologies in low-resource contexts and emphasize that visibility is the first step toward inclusion. By establishing a foundation for Comanche in NLP, we advocate for computational approaches that prioritize accessibility, cultural sensitivity, and community engagement.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Thejaswi Adimulam, Swetha Chinta, and Suprit Kumar Pattanayak. 2022. Transfer learning in natural language processing: Overcoming low-resource challenges. International Journal of Enhanced Research In Science Technology & Engineering, 11:65--79
work page 2022
-
[2]
James Richard Andrews. 2003. Introduction to classical Nahuatl, volume 1. University of Oklahoma Press
work page 2003
-
[3]
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.376 Systematic inequalities in language technology performance across the world ' s languages . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486--5505, Dublin, Ireland. Asso...
-
[4]
Rodrigo C \'a mara-Leret and Jordi Bascompte. 2021. Language extinction triggers the loss of unique medicinal knowledge. Proceedings of the National Academy of Sciences, 118(24):e2103683118
work page 2021
-
[5]
Joseph B Casagrande. 1955. Comanche linguistic acculturation iii. International Journal of American Linguistics, 21(1):8--25
work page 1955
-
[6]
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588--6608
2020
-
[7]
O. Chaika, N. Sharmanova, and O. Makaruk. 2024. https://doi.org/10.57125/FS.2024.06.20.03 Revitalising endangered languages: Challenges, successes, and cultural implications . Futurity of Social Sciences, 2(2):38--61
-
[8]
Jean Ormsbee Charney. 1993. A Grammer of Comanche. University of Nebraska Press
work page 1993
Show all 46 references
-
[9]
Nguyen Dinh, Thanh Dang, Luan Thanh Nguyen, and Kiet Nguyen. 2024. Multi-dialect vietnamese: Task, dataset, baseline models and challenges. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7476--7498
2024
-
[10]
Ingo Glaser, Shabnam Sadegharmaki, Basil Komboz, and Florian Matthes. 2021. Data scarcity: Methods to improve the quality of text classification. In ICPRAM, pages 556--564
2021
-
[11]
Xin Guan, Nate Demchak, Saloni Gupta, Ze Wang, Ediz Ertekin Jr, Adriano Koshiyama, Emre Kazim, and Zekun Wu. 2025. Saged: A holistic bias-benchmarking pipeline for language models with customisable fairness calibration. In Proceedings of the 31st International Conference on Co...
2025
-
[12]
Xiaobo Guo, Neil Potnis, Melody Yu, Nabeel Gillani, and Soroush Vosoughi. 2024. The computational anatomy of humility: Modeling intellectual humility in online public discourse. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5701--5723
2024
-
[13]
Alexandru-Iulius Jerpelea, Alina Radoi, and Sergiu Nisioi. 2025. Dialectal and low resource machine translation for aromanian. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7209--7228
2025
-
[14]
Yiren Jian, Chongyang Gao, and Soroush Vosoughi. 2022. Contrastive learning for prompt-based few-shot language learners. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5577--5587
2022
-
[15]
Kevin Kelly. 2020. An Evaluation of Parallel Text Extraction and Sentence Alignment for Low-Resource Polysynthetic Languages. Ph.D. thesis, University of Groningen
2020
-
[16]
Michael Krauss. 1992. The world's languages in crisis. Language, 68(1):4--10
1992
-
[17]
Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation. arXiv preprint arXiv:2410.20774
2024 arXiv
-
[18]
Abelardo Carlos Mart \' nez Lorenzo, Pere-Lu \' s Huguet Cabot, Karim Ghonim, Lu Xu, Hee-Soo Choi, Alberte Fern \'a ndez Castro, and Roberto Navigli. 2024. Mitigating data scarcity in semantic parsing across languages: the multilingual semantic layer and its dataset. In The 62...
2024
-
[19]
Dylan Scott Low, Isaac Mcneill, and Michael Day. 2022. Endangered languages: A sociocognitive approach to language death, identity loss, and preservation in the age of artificial intelligence. Sustainable Multilingualism, 21(1):1--25
2022
-
[20]
Manuel Mager, Arturo Oncevay, Annette Rios, Jamshidbek Mirzakhalov, and Katharina Kann. 2023. https://aclanthology.org/2023.c3nlp-1.4.pdf The role of computational linguistics in indigenous language revitalization: Challenges and opportunities . In Proceedings of the 1st Works...
2023
-
[21]
Ravindra Mangar, Cesar Arguello, David Inyangson, Tina Pavlovich, Karen Gareis, and Tushar M Jois. 2025. Engaging students from under-represented groups to pursue graduate school in computer science and engineering. In Proceedings of the 56th ACM Technical Symposium on Compute...
2025
-
[22]
Paul J Meighan. 2021. Decolonizing the digital landscape: The role of technology in indigenous language revitalization. AlterNative: An International Journal of Indigenous Peoples, 17(3):397--405
2021
-
[23]
Bureau of Indian Affairs. 2023. https://www.bia.gov/sites/default/files/dup/inline-files/bia_native_american_revitalization_lit_review_draft_08182023.pdf Native language revitalization literature review
2023
-
[24]
Marvin K Opler. 1943. The origins of comanche and ute. American Anthropologist, 45(1):155--158
1943
-
[25]
Jhonnatan Rangel. 2019. https://hal.archives-ouvertes.fr/hal-02917830 Challenges for language technologies in critically endangered languages . In UNESCO International Conference Language Technologies for All (LT4All), Paris, France. ⟨hal-02917830⟩
2019
-
[26]
Julia Sallabank and Peter K Austin. 2023. Endangered languages. In The Routledge handbook of applied linguistics, pages 362--373. Routledge
2023
-
[27]
Lane Schwartz. 2022. https://doi.org/10.18653/v1/2022.acl-short.82 P rimum N on N ocere: B efore working with I ndigenous data, the ACL must confront ongoing colonialism . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Sho...
2022 doi
-
[28]
Schreiner
Lane Schwartz, Emily Chen, Hyunji Hayley Park, Edward Jahn, and Sylvia L.R. Schreiner. 2021. http://arxiv.org/abs/2101.10496 A digital corpus of st. lawrence island yupik . arXiv preprint
2021 arXiv
-
[29]
Saul Schwartz and Lise M Dobrin. 2016. The cultures of native north american language documentation and revitalization. Reviews in Anthropology, 45(2):88--123
2016
-
[30]
Peng Shu, Junhao Chen, Zhengliang Liu, Hui Wang, Zihao Wu, Tianyang Zhong, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, et al. 2024. Transcending language boundaries: Harnessing llms for low-resource language translation. arXiv preprint arXiv:2411.11295
2024 arXiv
-
[31]
Piyapath T Spencer and Nanthipat Kongborrirak. 2025. Can llms help create grammar?: Automating grammar creation for endangered languages with in-context learning. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10214--10227
2025
-
[32]
Shelley Tulloch. 2006. https://doi.org/10.2167/cilp098.0 Preserving dialects of an endangered language . Current Issues in Language Planning, 7(2-3):269--286
2006 doi
-
[33]
Department of the Interior
U.S. Department of the Interior . 2022. https://www.bia.gov/sites/default/files/media_document/doi_federal_indian_boarding_school_initiative_investigative_report_vii_final_508_compliant.pdf Federal Indian Boarding School Initiative Investigative Report . Accessed: 2025-03-04
2022
-
[34]
Riitta-Liisa Valij \"a rvi and Lily Kahn. 2023. The role of new media in minority-and endangered-language communities. Endangered Languages in the 21st Century. Abingdon, Oxon, England: Routledge, pages 139--157
2023
-
[35]
Chixiang Wang and Junqi Guo. 2019. A data-driven framework for learners’ cognitive load detection using ecg-ppg physiological feature fusion and xgboost classification. Procedia computer science, 147:338--348
2019
-
[36]
Yuxin Wang, Ivory Yang, Saeed Hassanpour, and Soroush Vosoughi. 2024. Mentalmanip: A dataset for fine-grained analysis of mental manipulation in conversations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...
2024
-
[37]
Yuxin Wang, Xiaomeng Zhu, Weimin Lyu, Saeed Hassanpour, and Soroush Vosoughi. 2025. Impscore: A learnable metric for quantifying the implicitness level of sentences. In The Thirteenth International Conference on Learning Representations
2025
-
[38]
Ivory Yang, Xiaobo Guo, Sean Xie, and Soroush Vosoughi. 2024. Enhanced detection of conversational mental manipulation through advanced prompting techniques. In Eighth Widening NLP Workshop (WiNLP 2024) Phase II
2024
-
[39]
u shurescue: Reviving the endangered n \
Ivory Yang, Weicheng Ma, and Soroush Vosoughi. 2025 a . N \"u shurescue: Reviving the endangered n \"u shu language with ai. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7020--7034
2025
-
[40]
Ivory Yang, Weicheng Ma, Chunhui Zhang, and Soroush Vosoughi. 2025 b . https://arxiv.org/abs/2501.15773 Is it navajo? accurate language detection in endangered athabaskan languages . arXiv preprint arXiv:2501.15773
2025 arXiv
-
[41]
Hongchuan Zeng, Senyu Han, Lu Chen, and Kai Yu. 2025. Converging to a lingua franca: Evolution of linguistic regions and semantics alignment in multilingual large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10602--10617
2025
-
[42]
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161
2021 arXiv
-
[43]
Yuhong Zhang, Shilai Yang, Gert Cauwenberghs, and Tzyy-Ping Jung. 2024. From word embedding to reading embedding using large language model, eeg and eye-tracking. In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages...
2024
-
[44]
Tianyang Zhong, Zhenyuan Yang, Zhengliang Liu, Ruidong Zhang, Yiheng Liu, Haiyang Sun, Yi Pan, Yiwei Li, Yifan Zhou, Hanqi Jiang, et al. 2024. Opportunities and challenges of large language models for low-resource languages in humanities research. arXiv preprint arXiv:2412.04497
2024 arXiv
-
[45]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[46]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.