{"id":"c3523448-10d8-452e-ad5e-4427a55c96ff","arxiv_id":"2505.18159","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper introduces a 412-phrase Comanche dataset and reports 100% few-shot language identification accuracy, but the evaluation likely shares data between the few-shot examples and the test set.","lead":"This study presents the first NLP dataset for the Comanche language, 412 curated phrases, and tests whether large language models can identify Comanche. The authors report high few-shot accuracy, but the evaluation design appears to leak the test answers into the few-shot examples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The few-shot evaluation lacks an explicit separation between training examples and test items; if they overlap, the 100% accuracy claim measures memorization rather than generalization.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing concern: the possible overlap between the few-shot examples and the test entries in Section 5, which would invalidate the near-perfect accuracy claim. My reading of the paper confirms the ambiguity: the methods text describes only 'a sample of 100 randomly selected entries' and few-shot pairs drawn 'from the dataset,' with no explicit statement that the few-shot examples were excluded from the test pool. Since the 100% accuracy result is the paper's central empirical contribution, this ambiguity must be resolved before the claim can be accepted. I do not find evidence of additional load-bearing flaws that would justify outright rejection: the zero-shot result (13.5%) is consistent with the difficulty of the task, the dataset is a tangible contribution, and the authors acknowledge limitations of the synthetic pipeline and the small dataset. The abstract/body discrepancy (five vs. three examples) and the missing error bars are real but secondary; they do not change the central risk. A clean re-evaluation with an explicit train/test split, a reproducible script, and confidence intervals would settle the concern. If the strict-split accuracy remains near 100%, the paper's central claim survives and the verdict should be acceptance with minor revisions; if it drops materially, the claim is a memorization artifact and the verdict should be reject. I therefore recommend CONDITIONAL rather than REJECT, because the paper contains enough information (dataset, pipeline, and community engagement) to be salvageable, and the decisive issue is a single verifiable experimental detail rather than an irreversible methodological error.","tokens_in":8362,"tokens_out":6033,"duration_ms":57289,"concrete_test":"Check the GitHub repository (comanchegenerate/ComancheSynthetic) for the few-shot evaluation script to see whether the 100 test entries were sampled after excluding the few-shot examples. If the script is not available, re-run the Section 5 experiment with a strict split: sample the few-shot examples from the 412-entry dataset, then sample the 100 test entries only from the remaining entries, and report one-shot and three-shot accuracy with 95% confidence intervals. If the strict-split accuracy is materially lower than 91%/100%, the current claim is unsupported; if it remains near-perfect, the central claim survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5 reports 91% one-shot and 100% three-shot accuracy for GPT-4o Comanche identification, but the experimental description never states that the 100 test entries were sampled from the dataset after removing the few-shot examples. The text says 'we used a sample of 100 randomly selected entries from our original dataset' and that 'each few-shot pair included one Comanche phrase and a randomized English entry from the dataset.' If the few-shot examples were drawn from the same 412-entry pool without exclusion, a small number of test items could be exact duplicates of the in-context examples, which would inflate the reported accuracy. With three examples, the expected number of overlapping test items is about 0.7, so overlap cannot by itself turn 97% into 100%, but the paper provides no way to determine the true held-out accuracy. The abstract/body discrepancy ('five examples' vs. 'three-shot') and the missing GPT-4o-mini results and error bars are secondary; they do not change the central risk. The dataset and synthetic pipeline are concrete contributions, but the headline empirical claim depends on a split that is not documented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a first computational case study of Comanche, an endangered Uto-Aztecan language. The authors contribute a manually curated dataset of 412 Comanche-English phrases, a synthetic data generation pipeline that uses GPT-4o with normalized Levenshtein filtering, and zero-shot and few-shot language identification experiments on GPT-4o and, per the abstract, GPT-4o-mini. The main empirical claim is that GPT-4o identifies Comanche poorly in the zero-shot setting (13.5% accuracy) but dramatically better with a few in-context examples, reaching 91% one-shot and 100% three-shot accuracy. The paper also reports a community interview and advocates for community-informed, low-cost NLP for endangered languages.","tokens_in":8520,"tokens_out":4121,"duration_ms":44498,"significance":"The dataset and the focus on Comanche are genuinely valuable: there is essentially no prior NLP resource for this endangered language, and the community-engagement framing is a positive contribution. If the few-shot result were validated on a clean held-out set, it would provide a useful data point for low-resource language identification. The authors also state that the dataset and code are publicly available, which supports reproducibility. However, the central empirical claim is currently under-supported because the few-shot evaluation does not document a separation between the in-context examples and the test entries, and the synthetic data pipeline's quality threshold is so low that it admits near-random strings.","major_comments":[{"comment":"The few-shot evaluation does not establish that the 100 test entries are disjoint from the few-shot examples. The text says the test sample consists of '100 randomly selected entries from our original dataset' and that 'each few-shot pair included one Comanche phrase and a randomized English entry from the dataset,' but it never states that the in-context examples were removed from the test pool. If the same 412-entry pool was used for both, the reported 100% three-shot accuracy may reflect the model reading the answer from the prompt rather than generalizing to unseen Comanche text. Please provide an explicit train/test split for the few-shot experiments and report accuracy on examples that were never shown in context.","section":"Section 5"},{"comment":"The synthetic data pipeline uses a normalized Levenshtein similarity threshold of 0.11, which is only marginally above the 'baseline threshold of 0.1' mentioned in the footnote. At this threshold, outputs that share almost no characters with the reference translation can be accepted, so the pipeline does not currently demonstrate that the generated Comanche is linguistically usable. The paper should report the distribution of similarity scores, show representative accepted and rejected examples, and justify why 0.11 is an appropriate quality bar.","section":"Section 4.2"},{"comment":"The abstract claims 'near-perfect accuracy with just five examples,' while Section 5 reports 91% one-shot accuracy and '100% accuracy' with a three-shot strategy. These statements are inconsistent and should be reconciled, with the reported number of examples and the exact few-shot conditions stated precisely in both places.","section":"Abstract / Section 5"},{"comment":"The abstract states that the paper contains an 'empirical evaluation of GPT-4o and GPT-4o-mini,' but Section 5 reports results only for GPT-4o. No GPT-4o-mini results appear anywhere in the manuscript. Please either report the GPT-4o-mini results or revise the abstract and any related claims.","section":"Abstract / Section 5"}],"minor_comments":[{"comment":"The text contains an unresolved 'Appendix ??' reference, which should be corrected to the appropriate appendix.","section":"Section 4.2"},{"comment":"The reference to Charney (1993) lists the title as 'A Grammer of Comanche'; this should be 'A Grammar of Comanche.'","section":"References"},{"comment":"The paper uses the word 'consistently' for the three-shot result but reports no repeated runs, variance, or confidence intervals; please provide standard deviations or run-level results across multiple random few-shot example selections.","section":"Section 5 / Figure 5"},{"comment":"The GitHub link provided for the dataset and code should include a version identifier or repository snapshot so that the exact released artifacts can be cited and verified.","section":"Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's contribution is a useful resource, not a failed method paper. I am not recommending reject because the dataset and the overall question have merit, and the few-shot evaluation could be repaired with a proper held-out split. However, if the authors cannot provide such a split and instead only clarify the current pool-based evaluation, the central claim should not be published as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dataset is the contribution. The paper's headline result—near-perfect Comanche identification with three examples—is not established, because the evaluation doesn't document a clean separation between the in-context examples and the test items. That's a fixable problem, but it has to be fixed before the number means anything.\n\nWhat's genuinely new: this is the first NLP dataset for Comanche, 412 manually curated phrase pairs from 15 public sources, with a synthetic expansion pipeline and a community interview. For a language with fewer than 50 fluent speakers, that is a real resource, and the paper is honest that it's a first step. The zero-shot result (13.5%) is believable and useful: it quantifies how invisible Comanche is to a strong LLM. The related work is appropriate, and the ethics statement takes data sovereignty seriously.\n\nSoft spots, in order of importance. First, the few-shot evaluation is under-specified. The zero-shot run uses all 412 entries while the few-shot run uses a 100-entry sample, and the text never says the few-shot examples are excluded from that sample. The stress-test math shows overlap alone wouldn't turn 97% into 100%—expected overlap with three examples is under one item—but the paper provides no way to compute the true held-out accuracy, and that's unacceptable for the central claim. Second, the abstract promises 'near-perfect accuracy with just five examples' while Section 5 reports 100% at three shots; a discrepancy this visible makes me want to check everything else. Third, the synthetic data pipeline uses a Levenshtein similarity threshold of 0.11, which is close to random for strings of that length; calling that 'quality' is generous. Fourth, the paper says it evaluates GPT-4o and GPT-4o-mini, but only GPT-4o results are reported. No error bars anywhere.\n\nNone of this kills the dataset. But the empirical story needs a clean re-run: a proper train/test split, both models, error bars, and a sanity check on what the few-shot examples actually look like. I'd send it to peer review with major-revision expectations rather than desk-reject it, because the resource matters and the flaws are methodological, not fraudulent. The right referee will take the dataset seriously and demand the evaluation be redone properly.","headline":"The Comanche dataset is a real contribution; the few-shot accuracy claim is not yet supported and needs a clean re-evaluation before it means anything.","tokens_in":9102,"tokens_out":2111,"would_cite":true,"duration_ms":20755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three examples teach GPT-4o to identify Comanche at 100 percent.","keywords":["Comanche language","endangered languages","language identification","few-shot prompting","large language models","low-resource NLP","Uto-Aztecan languages","synthetic data generation"],"falsifier":"Re-run the three-shot identification experiment with a test set built from Comanche phrases that are provably disjoint from the few-shot examples, and repeat with different random example draws. If accuracy falls well below 100 percent in any disjoint condition, the reported result reflected memorization of the prompt examples rather than generalization to Comanche.","tokens_in":8137,"feed_emoji":"🗣️","tokens_out":8733,"duration_ms":74652,"temperature":0.7,"pith_summary":"The paper introduces Comanche, an Uto-Aztecan language with fewer than fifty fluent speakers, into NLP for the first time. It builds a manually curated dataset of 412 Comanche phrases, expands it with a synthetic generation pipeline, and tests whether GPT-4o can identify the language. The central finding is that few-shot prompting with a handful of Comanche-English examples improves identification from 13.5 percent zero-shot accuracy to 91 percent with one example and a consistent 100 percent with three. The paper argues this is a minimal-cost, community-informed way to make endangered languages visible to language technology.","feed_headline":"Three examples teach GPT-4o to identify Comanche at 100 percent","feed_subtitle":"Zero-shot accuracy was 13.5 percent; a three-shot prompt closed the gap to 100 percent.","key_machinery":"The load-bearing mechanism is in-context few-shot prompting: before asking the model to classify a phrase, the prompt supplies one to three Comanche phrases paired with randomized English translations drawn from the same dataset. These pairs teach the model the orthographic and morphological shape of Comanche, letting it recognize the language without fine-tuning. The paper also uses the same few-shot prompting to generate synthetic Comanche translations, filtered by a normalized Levenshtein similarity threshold, to expand the dataset.","core_discovery":"The paper's central claim is that a small number of Comanche-English example pairs placed in the prompt is enough for GPT-4o to identify Comanche text almost perfectly, even though the model fails without such examples. In zero-shot prompting, GPT-4o correctly identified 56 of 412 entries, or 13.5 percent. Adding one Comanche example raised accuracy to 91 percent, and a three-shot prompt produced 100 percent accuracy on the 100-entry evaluation sample. English identification stayed at 97-100 percent across conditions. On the paper's terms, this shows that minimal targeted prompting can overcome the default exclusion of an endangered language from LLM-based identification.","pith_inferences":["A stress test that varies the random draw of few-shot examples and test items across repeated trials would show whether 100 percent accuracy is stable or an artifact of a favorable sample.","Comanche is polysynthetic, so the few-shot examples may teach surface spelling patterns rather than grammar; a version of the prompt that also supplies a small morpheme or pronunciation lexicon might generalize better to unseen word forms.","Because the synthetic pipeline accepts translations at a low similarity threshold (0.11), testing identification separately on synthetic versus human-curated Comanche would clarify whether the few-shot benefit comes from real linguistic signal or from prompt artifacts."],"forward_implications":["If three examples suffice, any endangered language with a few documented phrases could be made identifiable to LLMs at near-zero cost.","The same few-shot approach could extend to other Uto-Aztecan and polysynthetic languages that currently lack language-identification support.","The synthetic data pipeline offers a route to bootstrap parallel corpora for downstream tasks such as machine translation and language-learning tools.","The result suggests the main barrier to including endangered languages in NLP is data visibility, not model capability."],"supporting_citations":[{"why":"Documents that large-scale language-identification systems omit Native American languages, defining the exclusion the paper addresses.","marker":"Caswell et al., 2020"},{"why":"Provides the few-shot prompting technique that the paper applies to GPT-4o for Comanche identification.","marker":"Zhang et al., 2021"},{"why":"Prior work on accurate language detection in endangered Athabaskan languages, giving a direct comparison point for the method.","marker":"Yang et al., 2025b"},{"why":"Frames the challenges of computational linguistics for indigenous language revitalization that motivate the study.","marker":"Mager et al., 2023"},{"why":"A grammar of Comanche that is a core source for the manually curated dataset.","marker":"Charney, 1993"}],"fun_headline_variants":["Few-shot prompts make GPT-4o perfect at spotting Comanche","Three examples lift Comanche detection from 13% to 100%","GPT-4o learns Comanche identification with just three shots","Endangered language? Three examples give GPT-4o perfect ID","Zero-shot fails, three-shot nails: Comanche ID hits 100%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result assumes that the Comanche phrases used as few-shot examples are never the same as the phrases in the 100-item test set; if a phrase appears in both, the reported near-perfect accuracy could be the model repeating in-context examples rather than recognizing unseen Comanche.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot prompts make GPT-4o perfect at spotting Comanche","Three examples lift Comanche detection from 13% to 100%","GPT-4o learns Comanche identification with just three shots","Endangered language? Three examples give GPT-4o perfect ID","Zero-shot fails, three-shot nails: Comanche ID hits 100%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":2995,"prompt_tokens":839,"completion_tokens":2156,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":2063}},"tokens_in":455,"tokens_out":2156,"duration_ms":15940,"temperature":1.0,"reasoning_tokens":2063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:39:22.824976+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three-shot identification experiment with a test set built from Comanche phrases that are provably disjoint from the few-shot examples, and repeat with different random example draws. If accuracy falls well below 100 percent in any disjoint condition, the reported result reflected memorization of the prompt examples rather than generalization to Comanche.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A grammar of Comanche that is a core source for the manually curated dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames the challenges of computational linguistics for indigenous language revitalization that motivate the study."}],"review_version":1}