{"id":"a31e9dad-5768-419b-bdf1-3f3eb8734e41","arxiv_id":"2505.13173","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Larger LLMs outperform smaller ones on zero-shot NLP tasks in Sanskrit, Ancient Greek, and Latin, and adding retrieved context improves Sanskrit question answering.","lead":"This paper tests how well large language models handle three ancient languages, Sanskrit, Ancient Greek, and Latin, in zero-shot tasks such as named-entity recognition, translation, and question answering. It introduces a new Sanskrit question-answer dataset and finds that larger models generalize better, especially when given retrieved context.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot attribution rests on an untested contamination assumption; the paper's own Limitations admit possible exposure for Ancient Greek MT, so 'solely cross-lingual generalization' is not established.","rationale":"The strongest claim is the causal attribution of observed performance to cross-lingual generalization rather than to exposure. I looked for the place where that claim is least secure and found it in the contamination assumption, which the reader also identified. The paper's own Limitations section admits that some datasets fall within the models' knowledge cutoffs and specifically flags Ancient Greek MT as anomalously high, so the threat is admitted in the manuscript, not merely hypothetical. Section 2's release-date argument is weaker than it appears: release after cutoff is neither necessary nor sufficient for absence from pretraining, and several datasets predate or sit at the cutoff. Section 3.1's English-prompts-beat-Sanskrit-prompts evidence is not a control for training exposure; it can be explained by instruction-following ability in the prompt language. Therefore the conclusion in Section 4 that zero-shot performance is attributable 'solely' to cross-lingual generalization is not supported, though the descriptive findings likely survive. Test-set tuning of the RAG k and the scale/family confound are real but secondary: they affect the size of the RAG gain and the scale interpretation, not the central zero-shot attribution. The new Sanskrit QA dataset and the reproducible code are genuine contributions; a contamination screen and a softened causal claim would make the acceptance condition satisfied.","tokens_in":17229,"tokens_out":8999,"duration_ms":86596,"concrete_test":"Perform a single contamination screen: for every test item in the six datasets, search for exact and near-exact (13-gram, edit distance <= 2) matches in pretraining-accessible corpora (Common Crawl, Pile, Wikipedia) and in the Ramayana and Ayurveda source texts. Exclude all matched items and recompute the zero-shot NER, MT, and QA scores in Tables 2, 3, and Figure 3. If Ancient Greek MT BLEU falls to the level of other language pairs or the main trends disappear, the 'solely cross-lingual generalization' claim is falsified; if scores are stable, the contamination objection is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4) that zero-shot performance is 'attributable solely to cross-lingual generalization' depends on the evaluation instances being absent from pretraining and on no task-specific instruction tuning in Sanskrit, Latin, or Ancient Greek. The paper's support is release dates relative to the knowledge cutoff (Section 2) and the English-prompt advantage for Sanskrit (Section 3.1). Neither checks the assumption. Latin NER (2019) and Latin MT (2023) predate the cutoff; Ancient Greek MT (Palladino et al., 2023) is at the boundary; and the Limitations section explicitly concedes that 'a few datasets we experimented were released within the models' knowledge cut-off dates raising the issue of data contamination,' singling out Ancient Greek MT as anomalously high. The prompt-language asymmetry is not a test of exposure: a model can have seen task data or source texts while still following English instructions better, so it cannot establish the absence of instruction tuning. The paper's own admitted contamination risk is thus an unresolved threat to the causal attribution in the abstract and conclusion, even though the descriptive benchmark results may survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies zero-shot cross-lingual generalization of GPT-4o, GPT-4o-mini, Llama-3.1-405b-instruct, and Llama-3.1-8b-instruct on named entity recognition and machine translation into English for Sanskrit, Ancient Greek, and Latin, and on factoid question answering for Sanskrit using a newly introduced 1501-pair dataset. The authors report that larger models match or outperform fine-tuned baselines on the out-of-domain settings they test, that English prompts are usually better for smaller models, that RAG with BM25 and a custom lemmatizer improves Sanskrit QA, and that smaller models struggle on niche entity types and in using retrieved context. The paper attributes the observed behavior to cross-lingual generalization from pretraining, arguing that the models were not instruction tuned on these classical languages.","tokens_in":17381,"tokens_out":5701,"duration_ms":59235,"significance":"If the attribution to pure zero-shot cross-lingual generalization were established, this would be a useful contribution to multilingual LLM evaluation and digital humanities: it contributes a new Sanskrit QA dataset, evaluates several open and proprietary models under a uniform protocol, and reports informative context effects through RAG and KG-based prompting. The out-of-domain comparisons and confusion-matrix analyses are also useful empirical material. However, the central significance depends on the contamination assumption, which the paper itself concedes is questionable for Ancient Greek MT. Until that is addressed, the results are best read as a descriptive benchmark under a zero-shot prompting protocol rather than as proof that performance is solely due to cross-lingual generalization.","major_comments":[{"comment":"The central claim in Section 4 that zero-shot performance is 'attributable solely to cross-lingual generalization' is not supported by the evidence. The only support offered in Sections 1 and 2 is that datasets are recent relative to knowledge cut-offs and that documentation does not mention instruction tuning. The Limitations section then concedes that 'a few datasets we experimented were released within the models' knowledge cut-off dates raising the issue of data contamination,' and identifies Ancient Greek MT as 'anomalously high,' suggesting possible exposure. This is exactly the load-bearing condition for the paper's causal claim, and it is not tested. Release dates do not rule out contamination from source texts either, since Ramayana, Ayurvedic texts, Greek and Latin corpora are likely present in large-scale pretraining. The English-prompt advantage for Sanskrit only shows that instruction following is better in English; it does not show that Sanskrit task data or instruction examples were absent. The conclusion should either be reworded to report benchmark performance under a zero-shot prompting protocol, or the authors should add a direct contamination analysis (e.g., overlap checks, memorization probes, or held-out re-annotation) before claiming sole cross-lingual generalization.","section":null},{"comment":"The conclusion that 'model scale is an important factor' (Abstract and Section 4) is weakened by the design: the large/small comparisons are between different model families and instruction-tuned variants (gpt-4o vs. gpt-4o-mini; llama-3.1-405b-instruct vs. llama-3.1-8b-instruct). Observed differences could reflect family-specific training data, alignment, decoding, or API behavior rather than parameter count alone. For example, in Table 7 the smaller models are absent from the LLM-KG comparison, and in Table 2 the smaller models show qualitatively different RAG gains from the larger models. The paper should either compare models matched on family and training procedure, or explicitly frame 'scale' as a confounded variable and support the conclusion with per-family analyses.","section":null},{"comment":"The abstract states that LLMs perform 'equal to or better than fine-tuned baselines on out-of-domain data,' but the supporting evidence in Appendix E covers only two comparisons: one Sanskrit MT benchmark and one Latin NER benchmark. These are useful indicative results, but they do not support a general claim across all three languages and both tasks. The wording should be narrowed to the specific datasets and settings evaluated, or additional out-of-domain comparisons should be reported.","section":null}],"minor_comments":[{"comment":"There is a typo in Appendix F.1: 'Python packaage' should be 'Python package'.","section":null},{"comment":"In Table 10, 'occurances' should be 'occurrences,' and in Table 11, 'Occuring' should be 'Occurring'.","section":null},{"comment":"The phrase 'With regards to how-questions' should be 'With regard to how-questions' for grammatical correctness.","section":null},{"comment":"The sentence 'This does not imply instruction tuning in these languages, since larger and smaller models likely saw comparable amounts of tuning data' is speculative; it should be presented as an untested assumption rather than an established fact.","section":null},{"comment":"The QA evaluation relies on manually curated acceptable answers and manual evaluation for a subset of reasoning questions; the authors should clarify in the main text how these resources will be released so that the evaluation is fully reproducible.","section":null}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the paper's headline claim of 'sole cross-lingual generalization' is contradicted by its own Limitations section for Ancient Greek MT and is not supported by the prompt-language asymmetry argument. The descriptive results and the new Sanskrit QA dataset are genuine contributions; if the authors either add a credible contamination analysis or substantially weaken the causal wording, the paper could be suitable. The scale conclusion should also be framed more carefully, as the model pairs differ in more than parameter count."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a useful empirical package: a new 1,501-question Sanskrit QA dataset with released code and data, and a systematic zero-shot evaluation of NER, MT, and QA in Sanskrit, Latin, and Ancient Greek. The main red flag is that the concluding claim — that the models' performance is 'attributable solely to cross-lingual generalization' — is not supported by the evidence, and the paper's own Limitations section weakens it by conceding possible data contamination for Ancient Greek MT. The stress-test note is on target here.\n\nWhat is actually new: the QA dataset (1501 pairs, drawn from Ramayana and Ayurveda MCQ books, expert-translated into Sanskrit, with category tables in the appendix) is a concrete resource. The RAG pipeline—BM25 with a T5-based lemmatizer—is a sensible, reproducible recipe. The comparisons against fine-tuned baselines on out-of-domain data are informative: gpt-4o and llama-3.1-405b beat LatinBERT on Ars Amatoria NER, and llama-3.1-405b outperforms Google/IndicTrans on the Mann ki Baat MT set. The script experiment (Devanagari vs IAST) and the confusion-matrix analysis for NER entity types are useful descriptive observations.\n\nSoft spots, in proportion. The central attribution rests on release dates and a prompt-language asymmetry. Release dates do not establish absence from pretraining, and the fact that English prompts work better for Sanskrit does not establish that the models were never instruction-tuned on the language — a model can have seen task data and still follow English instructions better. So 'solely' in Section 4 should go, and the conclusion should be reframed as a descriptive evaluation with a plausible but unverified cross-lingual transfer story. That is the main fix. Scale claims are confounded with model family and instruction-tuning recipe: comparing GPT-4o vs GPT-4o-mini and Llama-3.1-405b vs 8b is not an isolation of scale. The RAG k is tuned on the test set, and QA tables mostly lack uncertainty estimates. Minor but worth noting: the Sanskrit NER test set has only 139 sentences, so those scores are noisy.\n\nNone of this invalidates the dataset or the descriptive findings; it means the causal framing needs revision, not the experiments. This is a paper for digital humanists and low-resource NLP folks. It deserves a serious referee: with a rewritten narrative and ideally a contamination probe or at least consistent hedging, it is a solid contribution. I would cite it for the dataset alone.","headline":"Useful benchmark and evaluation for classical-language NLP, but the paper's strong zero-shot attribution claims are undercut by its own contamination caveat.","tokens_in":18012,"tokens_out":4503,"would_cite":true,"duration_ms":38588,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large off-the-shelf LLMs generalize zero-shot to Sanskrit, Latin, and Ancient Greek across named-entity recognition, machine translation, and question answering, matching or beating fine-tuned baselines.","keywords":["cross-lingual zero-shot generalization","classical languages","Sanskrit","Ancient Greek","Latin","named entity recognition","machine translation","retrieval-augmented generation"],"falsifier":"Take the published Sanskrit QA pairs and the Ancient Greek MT test sentences, build matched items written only after the models' knowledge cut-off (same style, same sources, never posted online), and run the same zero-shot prompts on those fresh items; if accuracy on fresh items drops well below the published scores, the published results are contaminated and the cross-lingual claim collapses.","tokens_in":17022,"feed_emoji":"📜","tokens_out":9160,"duration_ms":85599,"temperature":0.7,"pith_summary":"This paper tests whether general-purpose large language models can handle three classical languages—Sanskrit, Ancient Greek, and Latin—without any fine-tuning on those languages. Across named-entity recognition, translation into English, and factoid question answering in Sanskrit, the largest models match or beat fine-tuned baselines on out-of-domain data, while smaller models fall behind on rare or abstract entity types. The paper argues that this performance is genuine cross-lingual generalization: the models were not instruction-tuned on these languages, and English prompts work better than native-language prompts, especially for Sanskrit. It also contributes a new 1,501-pair Sanskrit question-answering dataset and shows that retrieving relevant passages with BM25 plus a lemmatizer substantially improves exact-match answers. If the attribution holds, digital-humanities researchers can use existing LLMs directly for classical-language work instead of building specialized models.","feed_headline":"Large LLMs work zero-shot on Sanskrit, Latin, and Ancient Greek","feed_subtitle":"Largest models match fine-tuned baselines; retrieved context boosts Sanskrit question answering by a wide margin.","key_machinery":"The operative mechanism is zero-shot prompting itself: no task-specific examples are supplied, so any success must come from the model's general linguistic competence. The paper manipulates four levers around that mechanism: model scale, prompt language, script, and retrieval context. The RAG pipeline is the empirical star: top-$k=4$ passages retrieved by BM25 from lemmatized source texts, using a Seq2Seq Sanskrit lemmatizer trained on the DCS corpus, raise exact-match QA scores sharply. The new 1,501-pair Sanskrit QA dataset, built from translated Hindi MCQ books on the Rāmāyaṇa and Bhāvaprakāśa nighaṇṭu, is the benchmark that makes the QA comparison possible. A knowledge-graph variant is explored but yields only modest gains, attributed to KG incompleteness.","core_discovery":"On the paper's own terms, the discovery is that high-capacity LLMs exhibit zero-shot cross-lingual generalization to three classical languages. GPT-4o and Llama-3.1-405B-instruct perform named entity recognition, translation into English, and factoid question answering in Sanskrit, Latin, and Ancient Greek at levels comparable to or above fine-tuned baselines on out-of-domain data, while their smaller counterparts degrade sharply on niche entity types and retrieval-based QA. Sanskrit QA scores improve markedly when BM25-retrieved context is provided, with a lemmatization step closing the gap caused by high inflection. English prompts outperform Sanskrit prompts for Sanskrit, which the paper reads as evidence that the models were not instruction-tuned on these languages; for Latin and Ancient Greek, larger models do equally well or better with native-language prompts. Devanagari input also beats IAST transliteration, consistent with transfer through shared script.","pith_inferences":["A direct testable extension of the orthographic argument: if script sharing drives transfer, models pretrained on Devanagari-rich corpora should show a larger Devanagari-over-IAST gap for Sanskrit than models with little Devanagari exposure.","Because the QA pairs were translated from Hindi MCQ books, some measured 'comprehension' may be inherited from the translation process; generating questions natively in Sanskrit from primary texts would separate language competence from source-question artifacts.","The contamination caveat for Ancient Greek MT suggests a natural control: compare scores on the published test set against a fresh, post-cutoff set of aligned Ancient Greek sentences; a large drop would confirm memorization.","If cross-lingual transfer flows through lexical influence—Latin and Greek into English, Devanagari through Hindi and Marathi—then transfer strength should correlate with measurable vocabulary overlap, which could help predict which other under-resourced languages LLMs will handle zero-shot."],"forward_implications":["If the attribution holds, researchers studying classical texts can run NER, translation, and factoid QA with current large LLMs out of the box, without fine-tuning task-specific models.","Sanskrit QA, a nearly empty benchmark space, now has a 1,501-pair dataset that future classical-language evaluations can use as a common reference point.","Retrieval-augmented generation with lemmatized lexical retrieval is a practical recipe for improving LLM answers in highly inflected languages, so classical-language NLP systems should include a lemmatizer before retrieval.","Model scale is the main lever: large models separate confusable mythological and abstract entity classes and use context effectively, while small models do not.","The Devanagari-versus-IAST gap implies that transliteration choice changes measured performance, so classical-language benchmarks should report script explicitly."],"supporting_citations":[{"why":"Supplies gpt-4o and gpt-4o-mini, the proprietary models whose zero-shot outputs are measured.","marker":"OpenAI, 2024"},{"why":"Supplies llama-3.1-405b-instruct and llama-3.1-8b-instruct, the open-source models under test.","marker":"Dubey et al., 2024"},{"why":"Provides the 139-sentence Sanskrit NER test set used in the evaluation.","marker":"Terdalkar, 2023"},{"why":"Provides the Latin NER test set used in the evaluation.","marker":"Erdmann et al., 2019"},{"why":"Provides the Ancient Greek NER corpus used in the evaluation.","marker":"Myerston, 2025"},{"why":"Supplies the Sanskrit-to-English MT benchmark and fine-tuned baselines the LLMs are compared against.","marker":"Maheshwari et al., 2024"},{"why":"Supplies the Ancient Greek MT gold standard, the dataset the paper flags as possibly contaminated.","marker":"Palladino et al., 2023"},{"why":"Defines the retrieval-augmented generation paradigm used to add context in Sanskrit QA.","marker":"Lewis et al., 2020"},{"why":"Provides the DCS corpus used to train the Seq2Seq Sanskrit lemmatizer that makes BM25 retrieval effective on inflected forms.","marker":"Hellwig, 2010-2024"},{"why":"Provides the fine-tuned LatinBERT baselines whose out-of-domain NER scores the zero-shot LLMs match or exceed.","marker":"Beersmans et al., 2023"}],"fun_headline_variants":["Zero-shot LLM mastery of Sanskrit, Latin, Ancient Greek","Scale drives zero-shot wins on classical languages","Big LLMs ace zero-shot on ancient languages","Retrieval boosts zero-shot Sanskrit QA in LLMs","LLM scale unlocks dead languages zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim collapses if models memorized any of the test items during pretraining or instruction tuning, and the paper's own Limitations section concedes that Ancient Greek MT shows anomalously high performance consistent with possible data contamination.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM mastery of Sanskrit, Latin, Ancient Greek","Scale drives zero-shot wins on classical languages","Big LLMs ace zero-shot on ancient languages","Retrieval boosts zero-shot Sanskrit QA in LLMs","LLM scale unlocks dead languages zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2438,"prompt_tokens":925,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1439}},"tokens_in":541,"tokens_out":1513,"duration_ms":12007,"temperature":1.0,"reasoning_tokens":1439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:57.656470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the published Sanskrit QA pairs and the Ancient Greek MT test sentences, build matched items written only after the models' knowledge cut-off (same style, same sources, never posted online), and run the same zero-shot prompts on those fresh items; if accuracy on fresh items drops well below the published scores, the published results are contaminated and the cross-lingual claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies gpt-4o and gpt-4o-mini, the proprietary models whose zero-shot outputs are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Latin NER test set used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Ancient Greek NER corpus used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Sanskrit-to-English MT benchmark and fine-tuned baselines the LLMs are compared against."},{"cited_title":"Wright, Anise d’Orange Ferreira, and Michel Ferreira dos Reis","cited_arxiv_id":null,"evidence_quote":"Supplies the Ancient Greek MT gold standard, the dataset the paper flags as possibly contaminated."},{"cited_title":"2010-2024","cited_arxiv_id":null,"evidence_quote":"Provides the DCS corpus used to train the Seq2Seq Sanskrit lemmatizer that makes BM25 retrieval effective on inflected forms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuned LatinBERT baselines whose out-of-domain NER scores the zero-shot LLMs match or exceed."}],"review_version":1}