{"id":"acdf16ac-e77a-4b05-b0c3-65f8fb0d9224","arxiv_id":"2412.20584","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper claims LLM prompting can translate a no-resource language at BLEU 0.45-0.6, but the appendix suggests the input may have been English glosses, not Paiute, so the central claim is not established.","lead":"This paper compares three ways to translate Owens Valley Paiute, a language with fewer than 100 documented sentences: fine-tuning a small model, prompting a large language model with examples, and direct prompting. It reports that prompting works better than fine-tuning, but the paper's appendix shows the model may have been given English word glosses rather than actual Paiute text, which would invalidate the headline result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never shows Paiute input; Appendix A's 'Original Phrase' columns are English sentences, so the experiment may be English-to-English paraphrase rather than no-resource translation.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: Appendix A labels the source as 'Original Phrase' but every shown string is English-like, and the manuscript never demonstrates that actual Paiute text was given to the model. My independent reading of the full text confirms this. The methodology (§3.3) describes prompt construction without specifying the source language; the only concrete outputs in the paper are English-to-English pairs. If the model received English, then the reported BLEU scores measure paraphrase quality, not translation quality, and the central claim about no-resource translation collapses. I also note the paper provides no human baseline, no error bars, and internally inconsistent BLEU figures (e.g., direct prompting at 10 phrases is 0.605 in Table 4 while the abstract emphasizes 0.45–0.6; chain-of-reasoning at 100 phrases is 0.448 in Table 3 but 0.48 in §4.1). These secondary issues strengthen the rejection but do not replace the source-language check. The linked data and code could in principle settle the question, and I would welcome that verification; however, based on the manuscript as written, the evidence is insufficient to support the stated claim. The verdict should remain REJECT.","tokens_in":8222,"tokens_out":2563,"duration_ms":27070,"concrete_test":"Download https://github.com/kubishi/yaduha/blob/main/data/translations.csv and inspect the Kaggle notebook https://www.kaggle.com/code/standbackibyte/no-resource-tranlsation-llm-queries. Reconstruct the exact user prompt for one Appendix A row, e.g., 'He/she/it reads the squirrel.' If the prompt sent to the xAI API contains that English string or any English gloss as the source text, the central claim fails because the task is English-to-English paraphrase. If the prompt instead contains genuine OVP orthography and the Appendix entries are only display glosses, then the concern is resolved and a different review focused on missing human baselines and error bars would be needed.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that LLM in-context learning translates Owens Valley Paiute (OVP) to English with BLEU 0.45–0.60. For this claim to hold, the model inputs must be actual OVP text. Appendix A, Tables 1–6, label the source column 'Original Phrase' but every listed entry is an English sentence, e.g., 'He/she/it reads the squirrel.' and 'That lizard is going to write.' The paper nowhere presents Paiute orthography, a morpheme gloss, or an example of a constructed prompt. Section 3.3 says the system prompt includes 'known translations' and the user prompt presents 'the target phrase', but it never states the language of those strings. If the corpus used was the linked translations.csv and its source column is English, as Appendix A suggests, then the model was asked to rewrite English into similar English, and BLEU was computed against an English reference. Such a setup would not test translation of a no-resource language at all; it would test paraphrase. The abstract's claim that in-context learning 'enable[s] no-resource language translation' would therefore be unsupported. The absence of a human baseline is a second serious gap, but it becomes decisive only after the input-language question is resolved. The internal inconsistencies in reported BLEU values (e.g., direct prompting at 10 phrases is reported as 0.605 and chain-of-reasoning as 0.199, while the abstract highlights 0.45–0.6) reinforce the need to check the raw inputs, but the source-language issue is the most load-bearing because it determines whether any translation was attempted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines no-resource translation as translation of languages with fewer than 100 documented phrases, and compares three neural workflows on Owens Valley Paiute-to-English translation: QLoRA fine-tuning of PaLM, chain-of-reasoning prompting through the xAI API, and direct prompting without reasoning. It reports that fine-tuning yields near-zero BLEU scores, chain-of-reasoning prompting improves with corpus size, and direct prompting performs well on very small corpora but degrades as the corpus grows. The abstract concludes that in-context learning enables no-resource translation that outperforms low-resource approaches and rivals human translations (BLEU 0.45-0.6), establishing no-resource translation as a distinct paradigm.","tokens_in":8690,"tokens_out":3906,"duration_ms":37976,"significance":"If the central claim were established, the result would be practically and theoretically important: a purely in-context-learning approach that outperforms low-resource methods on a genuinely no-resource language would offer a concrete route for language preservation without expert linguistic input. The paper also takes a useful step by formalizing the \"no-resource\" regime and by releasing code and data links. However, the evidence as presented does not establish that the model ever received Owens Valley Paiute text: the appendix's \"Original Phrase\" columns contain only English sentences, and no Paiute orthography or example prompt is shown. The human-rivaling claim is also unsupported because no human translation baseline is reported.","major_comments":[{"comment":"The source language of the model inputs is never established. Every \"Original Phrase\" entry in the appendix is an English sentence (e.g., \"That lizard is going to write.\"), and the paper contains no Paiute orthography, morpheme gloss, or actual example of the constructed prompt. Section 3.3 says the system prompt includes \"known translations\" and the user prompt presents \"the target phrase,\" but it does not state the language of those strings. If the model received the English strings shown in Appendix A, the experiment is an English-to-English paraphrase task, not translation from a no-resource language, and the central claim of the paper collapses. This issue is load-bearing and must be resolved with direct evidence of the actual inputs.","section":"Section 3.3 and Appendix A, Tables 1-6"},{"comment":"The BLEU metric is reported with inconsistent conventions. The paper alternates among \"BLEU 0.48,\" \"BLEU: 0.448,\" \"BLEU (49%),\" and \"BLEU scores exceeding 20%\" (Section 5.1). Standard BLEU is commonly reported on a 0-100 scale, while the appendix values such as 0.199, 0.605, and 0.448 are on a 0-1 scale. Without a stated convention, the abstract's \"0.45-0.6\" range cannot be compared with the \"20%\" figure for low-resource systems, and the claim of rivaling human translations is uninterpretable.","section":"Sections 4.1-4.3, 5.1, and Tables 1-6"},{"comment":"The empirical comparisons lack the baselines needed to support the paper's headline claims. There is no human translation baseline, no comparison with the rule-based baseline from [4], and no multiple runs, error bars, or statistical tests. Since Section 4.2 itself attributes direct prompting's small-corpus advantage to \"simple copying of the format and vocabulary of seen data,\" and since the same corpus supplies both the few-shot examples and the reference translations, the reported scores may reflect memorization rather than generalization. The scaling conclusions (chain-of-reasoning improves with size; direct prompting degrades) rest on single observations and are not statistically supported.","section":"Sections 4 and 5"}],"minor_comments":[{"comment":"The column header \"T ranslation\" contains an erroneous space; it should read \"Translation.\"","section":"Appendix A"},{"comment":"There is a typo in \"enabled by a the in-context learning abilities,\" and the phrase \"chain-of reasoning\" is used inconsistently with \"chain-of-reasoning\" elsewhere.","section":"Section 5.1"},{"comment":"The paper uses \"corpuses\" where \"corpora\" is the standard plural form.","section":"Section 5.2"},{"comment":"The model name \"PaLM Flan-T5 small V2\" conflates two different model families; please specify the exact model and checkpoint used for fine-tuning.","section":"Section 4.3"},{"comment":"The data and code links are listed without details about their contents; please state explicitly whether the corpus file contains the Paiute source strings, and provide one representative constructed prompt so readers can verify the input language.","section":"Section 7"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the absence of any evidence that the model inputs were actually Owens Valley Paiute. If the authors can show the raw inputs and example prompts, and add a human baseline plus uncertainty estimates, the work might be reconsidered. As presented, the central claim is not established, and the apparent English-to-English nature of the appendix examples makes the flaw appear fundamental rather than merely a presentation problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I agree with you: the paper's central claim doesn't hold as written. The most useful thing here is the problem framing — no-resource translation as distinct from low-resource — and the attempt to compare fine-tuning, chain-of-reasoning prompting, and direct prompting on a genuinely under-documented language. That's a reasonable way to organize an experiment, and the authors should get credit for making their data and code links public, and for noting that direct prompting at small sizes is partly copying. But the evidence doesn't support what the abstract says.\n\nAppendix A is the problem: every 'Original Phrase' is an English sentence. If those are the model inputs, then the task is English-to-English paraphrase, not translation from Owens Valley Paiute. The paper never shows Paiute orthography or an example constructed prompt, so we have no way to verify the model ever saw the target language. That's a load-bearing issue, not a presentation quibble. The absence of a human baseline and the failure to compare against the rule-based system in [4] are also real gaps, but they only matter once we know the input language. The reported numbers are also internally inconsistent: the appendix gives BLEU 0.199 for chain-of-reasoning at 10 phrases, while the abstract highlights 0.45–0.6 as the general range. That doesn't inspire confidence in the rest of the quantitative claims.\n\nWhat's missing isn't a tweak; it's a transparent example of the actual prompt and the actual source text. Until that's shown, the 'rivals human translations' claim is an assertion. If the authors fix that, and add even a single human-translation baseline or the [4] comparison, the paper could be a useful empirical note. As it stands, it should not be peer reviewed in this form. I'd desk-reject and invite a revision that shows the source language and the prompt structure.","headline":"The paper's framing is reasonable but its central evidence is missing: Appendix A shows English inputs, not Paiute, so the translation claim collapses.","tokens_in":9065,"tokens_out":2394,"would_cite":false,"duration_ms":23615,"reading_group":"no","serious_thinker":"unclear","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that general-purpose LLMs, prompted with a small corpus of known translations, can translate no-resource languages, while fine-tuning traditional translation models fails.","keywords":["no-resource machine translation","in-context learning","chain-of-reasoning prompting","Owens Valley Paiute","low-resource translation","large language models","language preservation"],"falsifier":"Open the linked translations.csv and the logged prompt strings: if the source column contains English sentences such as 'He/she/it reads the squirrel.' rather than Owens Valley Paiute orthography, the central claim fails. Alternatively, run the documented chain-of-reasoning pipeline with genuine Paiute text as input and check whether BLEU remains above 0.45.","tokens_in":8023,"feed_emoji":"🌐","tokens_out":3340,"duration_ms":31971,"temperature":0.7,"pith_summary":"The paper is trying to establish that no-resource languages, defined as having fewer than 100 documented sentences, are a distinct machine translation problem that cannot be solved with low-resource techniques such as fine-tuning. Using Owens Valley Paiute as a case study, it argues that the in-context learning abilities of large language models enable translation performance that rivals human output, with BLEU scores around 0.45 to 0.60. Chain-of-reasoning prompting is reported to improve as the provided corpus grows, while direct prompting works better on very small corpora. If correct, this would make LLM prompting a practical, language-agnostic route to translating and documenting endangered languages without expert-built rules.","feed_headline":"LLM prompts beat fine-tuning for no-resource translation","feed_subtitle":"Chain-of-reasoning prompts reach BLEU 0.45–0.60 on Owens Valley Paiute; fine-tuning near zero.","key_machinery":"The central mechanism is chain-of-reasoning prompting for in-context learning: a system prompt contains a small set of known phrase translations, the target phrase is removed from that set, and the model is asked to infer the translation of the target phrase. This setup is designed to force generalization rather than memorization, and it is compared against direct prompting and against QLoRA fine-tuning of a translation-specific model. The corpus used is Owens Valley Paiute, with subsets of 10, 50, and 100 phrases, and the quality of translations is measured with BLEU, ROUGE, METEOR, and TER.","core_discovery":"The central claim is that no-resource translation is fundamentally different from low-resource translation: methods that rely on even a small corpus, such as fine-tuning with QLoRA, collapse when the corpus has fewer than about 100 sentences, whereas in-context learning in a general-purpose LLM does not. Specifically, chain-of-reasoning prompting, where the model is given a set of known translations and asked to infer a new one, reached an average BLEU of 0.48 on the 99-phrase Owens Valley Paiute reference corpus. Direct prompting reached BLEU 0.60 on a 10-phrase corpus but declined to 0.47 at 99 phrases, while fine-tuned PaLM produced near-zero BLEU scores and output unrelated languages. The paper therefore proposes that emergent reasoning in LLMs, not parameter updates, is the mechanism that makes no-resource translation work.","pith_inferences":["An implication the author leaves implicit is that this workflow could be operationalized as: collect 50 to 100 glossed sentences from a speaker, prompt a general-purpose LLM to translate the remaining field recordings, and then have a speaker verify the output, greatly reducing the cost of documentation.","The same in-context mechanism may apply to other extreme-data linguistic tasks, such as dictionary induction, glossing, or morpheme segmentation, where only a handful of examples exist.","The appendix shows the source column as English-like sentences rather than Owens Valley Paiute orthography, so the reported experiment may actually be measuring English-to-English paraphrase; a rerun with genuine Paiute text as input would test whether the claimed no-resource translation result holds."],"forward_implications":["Fine-tuning translation-specific models on fewer than 100 sentences is reported to fail completely, so low-resource methods should not be expected to transfer to no-resource settings.","Chain-of-reasoning prompting improves with additional in-context phrases, suggesting that a small bilingual glossary could be enough to bootstrap translation of an entire corpus.","Direct prompting is strongest when the corpus is tiny, indicating that substitution and format copying can succeed where inference is unnecessary.","Because the prompting methods do not depend on the target language's grammar, the approach could generalize to other no-resource languages without expert input.","The reported BLEU range of 0.45 to 0.60 is close enough to human-quality output to make LLM prompting a viable tool for language documentation and preservation."],"supporting_citations":[{"why":"Defines no-resource languages as having fewer than 100 documented phrases and provides the rule-based LLM-assisted baseline that this paper contrasts with.","marker":"[4]"},{"why":"Establishes the in-context learning capabilities of large language models that the prompting methods rely on.","marker":"[2]"},{"why":"Supplies the chain-of-thought prompting technique that the chain-of-reasoning workflow adapts for translation.","marker":"[23]"},{"why":"Describes the PaLM model that is fine-tuned with QLoRA in the low-resource comparison arm.","marker":"[3]"},{"why":"Represents the low-resource translation approach, using monolingual data, that the paper argues fails for no-resource languages.","marker":"[18]"},{"why":"Provides the Transformer architecture background that motivates neural machine translation and the LLMs used in the experiments.","marker":"[22]"}],"fun_headline_variants":["Prompting LLMs outperforms fine-tuning on tiny corpora","Chain-of-reasoning prompts hit BLEU 0.48 on Paiute","Fine-tuning fails on <100 sentences; prompting succeeds","LLM in-context learning translates languages with no corpus","No-resource translation via LLM reasoning, not fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's input was actual Owens Valley Paiute text, not an English word-by-word gloss; the released appendix shows only English-like source phrases, so if the inputs were English, the experiment measured paraphrase rather than no-resource translation.","fun_headline_variants_meta":{"raw":{"variants":["Prompting LLMs outperforms fine-tuning on tiny corpora","Chain-of-reasoning prompts hit BLEU 0.48 on Paiute","Fine-tuning fails on <100 sentences; prompting succeeds","LLM in-context learning translates languages with no corpus","No-resource translation via LLM reasoning, not fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3312,"prompt_tokens":987,"completion_tokens":2325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2250}},"tokens_in":603,"tokens_out":2325,"duration_ms":16093,"temperature":1.0,"reasoning_tokens":2250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:16:40.548840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the linked translations.csv and the logged prompt strings: if the source column contains English sentences such as 'He/she/it reads the squirrel.' rather than Owens Valley Paiute orthography, the central claim fails. Alternatively, run the documented chain-of-reasoning pipeline with genuine Paiute text as input and check whether BLEU remains above 0.45.","supporting_citations":[{"cited_title":"Language models are few-shot learners.Advances in neural informa- tion processing systems, 33:1877–1901, 2020","cited_arxiv_id":null,"evidence_quote":"Establishes the in-context learning capabilities of large language models that the prompting methods rely on."},{"cited_title":"Attention is all you need.Advances in neural information processing systems, 30:5998–6008, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer architecture background that motivates neural machine translation and the LLMs used in the experiments."}],"review_version":1}