{"id":"de43d6d7-fbdf-4cde-abe6-604fbc988d95","arxiv_id":"2607.17173","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KyrgyzLLM-Bench evaluates 26 LLMs on native and translated Kyrgyz tasks, showing rankings transfer only partially and translated HellaSwag is unreliable.","lead":"Researchers built the first large-scale Kyrgyz-language test suite for AI models, mixing native school-exam questions, reading comprehension, and translated English benchmarks. They evaluated 26 models and found that rankings mostly carry over from English on some tasks, but translated sentence-completion tests are unreliable in Kyrgyz.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Kyrgyz WinoGrande results are all at chance, contradicting the claimed English-to-Kyrgyz rank transfer on WinoGrande.","rationale":"The reader's weakest_assumption focuses on item correctness and the absence of inter-annotator agreement for KyrgyzMMLU/KyrgyzRC. That is a legitimate quality concern, but the more directly load-bearing problem is visible in the reported results themselves: the Kyrgyz WinoGrande column is essentially flat at chance level, while the English column has meaningful variation. The abstract's central claim explicitly says rankings transfer on WinoGrande; this is contradicted by the table. This is an internal inconsistency, not an external debate about consensus. The benchmark as a resource may still be valuable, but the cross-lingual transfer conclusion—one of the two main empirical contributions—cannot stand as written. The paper needs a substantive revision: either a re-analysis of WinoGrande using a corrected or natively authored version, a clear statement that Kyrgyz WinoGrande is at chance and excluded from rank-transfer comparisons, or a substantial weakening of the abstract claim. Because the resource itself remains useful and the issue is fixable, CONDITIONAL is the appropriate verdict, but the condition is mandatory and should include the concrete verification described above.","tokens_in":23317,"tokens_out":7735,"duration_ms":72089,"concrete_test":"Compute the Spearman rank correlation between Table 6 and Table 7 zero-shot WinoGrande accuracies across the 14 open models, with a 95% bootstrap confidence interval. Also compute binomial 95% confidence intervals for each Kyrgyz WinoGrande score. If the correlation CI includes 0 (or is negative) and the Kyrgyz intervals all overlap 50%, then the 'rankings transfer on WinoGrande' claim is unsupported and the abstract/Section 6 text must be amended to remove or re-scope that claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and Section 6 make the load-bearing claim that 'model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ.' The paper's own data contradict this for WinoGrande. In Table 7 (zero-shot), all 14 open models score between 48.3% and 51.5% on Kyrgyz WinoGrande, a two-choice task where chance is 50%. With roughly 1000 items, the binomial standard error at p=0.5 is about 1.6 percentage points, so the entire spread across models is within sampling noise. The few-shot column is similarly flat (roughly 48–52%). Proprietary results in Table 8 are also narrow (48–58%). Meanwhile, English WinoGrande scores in Table 6 range from 49.6% to 71.3%, carrying real signal. The rank order does not transfer either: the top English model, Llama-3.1-8B-Instruct (71.3%), is mid-pack in Kyrgyz (50.6%), while several much weaker English models match or exceed it. A Spearman correlation computed from these columns would be near zero. Thus the central claim that rankings transfer on WinoGrande is not merely under-supported; it is contradicted by the reported numbers. Since this is one of the two tasks named in the headline cross-lingual claim, the conclusion as stated is unreliable. The paper should either revise the claim, or characterize Kyrgyz WinoGrande as a non-discriminative translated task whose scores are at chance and not usable for cross-lingual ranking.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KyrgyzLLM-Bench, a benchmark suite for Kyrgyz consisting of two natively authored datasets (KyrgyzMMLU with 7,977 items from the national school exam and KyrgyzRC with 400 reading-comprehension items) and four translated benchmarks (WinoGrande, HellaSwag, BoolQ, TruthfulQA) that were machine-translated, post-edited by native speakers, and reviewed. The authors evaluate 26 open and proprietary LLMs under zero-shot and few-shot settings using Lighteval, report English baselines for 14 open models, and analyze cross-lingual transfer, few-shot effects, and translation artifacts. The central claims are that model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ, to a lesser extent on MMLU, and that the large English–Kyrgyz gap on HellaSwag is consistent with translation-induced plausibility shifts. The paper also releases datasets, code, and per-model results and integrates tasks into Lighteval.","tokens_in":23692,"tokens_out":2051,"duration_ms":21101,"significance":"If the benchmark and conclusions are validated, the contribution is useful: KyrgyzMMLU and KyrgyzRC are scarce native-resource additions for a low-resource Turkic language, and the transparent documentation of construction and quality-control steps, plus public release, would support reproducibility. The paper also ships machine-checkable artifacts (datasets, code, per-model results) and an explicit falsifiable prediction about HellaSwag being translation-sensitive. However, the headline cross-lingual ranking claim is only partially supported by the reported tables, and the absence of uncertainty quantification or repeated-seed evaluation weakens some numerical comparisons. The benchmark resource itself is valuable even if the cross-lingual transfer claim requires revision.","major_comments":[{"comment":"The claim that 'model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ' is not supported for WinoGrande. In Table 7, all 14 open models score between 48.3% and 51.5% zero-shot on Kyrgyz WinoGrande, a two-choice task: this spread is within binomial sampling noise (approximately ±1.6 points for ~1000 items at p=0.5). Few-shot scores are similarly flat (48–52%). English WinoGrande scores in Table 6 range from 49.6% to 71.3%, so the English signal does not transfer. The few-shot English ranking is also not mirrored: Llama-3.1-8B-Instruct, top on English WinoGrande (71.3% zero-shot, 74.0% few-shot), is mid-pack on Kyrgyz (50.6%/51.6%). The conclusion as stated should be revised: the data indicate that Kyrgyz WinoGrande is non-discriminative at chance level, not that rankings transfer. A Spearman correlation computed from the tables would likely be near zero. The paper","section":"Abstract and Section 6 (Table 7)"},{"comment":"The evaluation uses temperature=0.6 and top-p=0.9, a stochastic decoding regime, but each number is reported as a single point estimate with no repeated seeds, no variance, and no confidence intervals. This is particularly problematic for the cross-lingual comparison, since many differences in Tables 6–7 are a few percentage points; with stochastic decoding and ~1000-item tasks, the standard error alone is ~1.6 points, and seed-to-seed variation at nonzero temperature can be several points. The paper should either report mean and standard deviation over multiple runs, or use greedy decoding with a justification. This is needed to support claims such as 'few-shot gains' and 'proprietary models exhibit less consistent few-shot behavior' (Section 6).","section":"Section 5 (decoding) and Tables 6–10"},{"comment":"The validity of Myanmar/English aside, the KyrgyzRC and KyrgyzMMLU items are load-bearing for every model score, but the annotation process is sequential (one author, one domain supervisor, one linguist) with no inter-annotator agreement. The paper itself notes Cohen's kappa was not computed and that items were not independently dual-annotated. This is an acknowledged limitation, but its consequences for the central MMLU/RC claims are not quantified. For a benchmark paper, a small held-out parallel-annotation study or a post-hoc item-error audit would substantially increase confidence that systematic item errors do not drive the results. As it stands, the reader cannot rule out that some aggregate differences—particularly the low KyrgyzMMLU scores—reflect item ambiguity or formatting issues rather than model capability.","section":"Section 4.2 and Limitations"},{"comment":"The WinoGrande example in Table 5 is a red flag for the translated benchmark's validity. The original English item uses 'home' vs 'house'; the Kyrgyz translation renders both as 'үй' (home) and 'турак жай' (dwelling), which are near-synonyms in this context. The sentence 'Ал менин үйүмө эч качан келбейт, бирок мен ар дайым анын турак жайына барам' does not preserve the pragmatic distinction that makes the original Winograd schema solvable. If this is representative, Kyrgyz WinoGrande may be unsolvable by design, which would explain the chance-level performance in Table 7. The authors should provide a per-item translation-quality audit or at least report the proportion of items where the two options are not semantically distinguished in Kyrgyz. This is directly relevant to the paper's own recommendation to audit translated HellaSwag, but the same scrutiny should apply to WinoGrande.","section":"Section 4.3 / Table 5"},{"comment":"The proprietary-model results are based on the condensed KyrgyzLLM Tiny Bench (100 randomly selected questions per subject for KyrgyzMMLU sub-parts, and tasks like BoolQ/HellaSwag/WinoGrande on a 100-item subset). With 100 binary items, the binomial standard error is ~5 points, so many differences in Table 8 (e.g., 48% vs 53% on WinoGrande) are within noise. The paper presents these numbers as task-level comparisons without acknowledging the reduced statistical power. Moreover, Gemini 2.5 Flash's scores are flagged as affected by safety refusals, but the table still includes those numbers; the paper should state how many refusals occurred and whether those answers were scored as incorrect, since this materially changes interpretation.","section":"Section 6 and Table 8"},{"comment":"The claim that rankings are 'broadly preserved on WinoGrande/BoolQ' is made without reporting any correlation coefficient or rank-order statistic. Given the chance-level WinoGrande data and the narrow BoolQ range in Table 7 (zero-shot: 37.9–58.3; few-shot: 57.4–76.9), a quantitative measure (e.g., Spearman rho with a confidence interval) is needed. Without it, the phrase 'broadly preserved' is not verifiable. The paper already computes English baselines; the additional analysis is straightforward.","section":"Section 6 cross-lingual consistency paragraph"}],"minor_comments":[{"comment":"The sample KyrgyzMMLU question is a simple arithmetic word problem; it would help to show a more linguistically or culturally specific item to support the claim of cultural grounding.","section":"Table 3"},{"comment":"Appendix C shows the same Wikipedia passage as Table 4; consider using a different example to illustrate the metadata schema and avoid redundancy.","section":"Section 4.2 / Appendix C"},{"comment":"The zero-shot and few-shot blocks in Tables 6–7 would be easier to read if the few-shot delta columns were aligned with the corresponding zero-shot columns; cell colors help but a numeric delta would be clearer.","section":"Tables 6 and 7"},{"comment":"The claim that tokenizer fragmentation 'increases effective context length' should be phrased as 'increases the number of tokens (and thus context consumption) for the same content', to avoid ambiguity about what 'effective' means.","section":"Section 3"},{"comment":"Reference [14] on the Kyrgyz Seed dataset is cited for a quality analysis of NLLB OPUS; the citation is plausible but the sentence could be more specific about which proportion and which language pairs were analyzed.","section":"References"},{"comment":"The GSMT8K exclusion rationale is clear, but the paper should mention the size of the translated GSM8K set and whether the translation will be released separately, since the current statement leaves ambiguity.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The central resource contribution—two native datasets plus a translated set with quality control—is solid and worth publishing after revision. The main problem is that the headline cross-lingual transfer claim includes WinoGrande, which the paper's own tables show to be at chance level in Kyrgyz; this is a load-bearing error that can be fixed by changing the claim and by explicitly characterizing WinoGrande as non-discriminative. I would also ask for uncertainty quantification or a switch to greedy decoding, and for a small inter-annotator agreement check on KyrgyzRC/KyrgyzMMLU. The paper is not fatally flawed; the fixes are within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful, honest paper that deserves a serious referee, but one of its headline claims doesn't survive contact with its own tables. The abstract and Section 6 say model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ. Table 7 shows all fourteen open models scoring between 48.3% and 51.5% on Kyrgyz WinoGrande, a two-choice task; chance is 50%, and even with ~1000 items the binomial standard error is around 1.6 points. The few-shot column is equally flat. Proprietary scores in Table 8 span 48–58%, mostly 48–56%. English WinoGrande, by contrast, ranges from about 50% to 71%. So Kyrgyz WinoGrande carries no discriminative signal, and the rank ordering in English cannot transfer because there is nothing to transfer. The paper should either drop WinoGrande from that claim or explicitly describe it as a non-discriminative translated task.\n\nThe resource itself is genuinely valuable. KyrgyzMMLU (7,977 items from the national testing system) and KyrgyzRC (400 native reading-comprehension questions) fill a real gap, and the translated tasks are described with a reasonable quality-control pipeline: two machine translations, native-speaker post-editing, back-translation checks. The authors also integrate into Lighteval and release data, code, and per-model results. That is the kind of evidence a benchmark paper should ship. The HellaSwag plausibility-shift hypothesis is plausible and appropriately hedged as a hypothesis.\n\nOther soft spots are minor relative to the WinoGrande issue but worth fixing. No inter-annotator agreement was computed, and the paper says so. Evaluation used stochastic decoding with no repeated seeds, so no confidence intervals. The proprietary results use a 100-item subset, which puts standard errors near 5 points per subject. A few reported cells (e.g., Gemma-3-4b-it BoolQ dropping to 0.0) smell like parsing failures rather than model behavior. None of these are fatal for the benchmark's usefulness, but the quantitative claims should be bounded.\n\nThis is a paper for people building or using low-resource and Turkic benchmarks, and for anyone studying cross-lingual transfer. I'd send it to peer review with a request for revision: fix the WinoGrande claim, add uncertainty estimates, and report parsing-failure rates.","headline":"A real benchmark resource, but the WinoGrande transfer claim is contradicted by their own numbers — all Kyrgyz WinoGrande results sit at chance.","tokens_in":24157,"tokens_out":2776,"would_cite":true,"duration_ms":26689,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds the first large-scale, natively authored Kyrgyz evaluation suite and argues that English-to-Kyrgyz model rankings transfer on reasoning and reading tasks, while a large HellaSwag gap reflects translation artifacts rather t","keywords":["Kyrgyz NLP","LLM evaluation","multilingual benchmarks","low-resource languages","cross-lingual transfer","translation artifacts","reading comprehension","Turkic languages"],"falsifier":"Re-score a random sample of KyrgyzMMLU and KyrgyzRC with an independent panel of native-speaker annotators unconnected to the authors, and build a natively authored Kyrgyz HellaSwag-style continuation task; then rerun the same 26 models on it. Widespread item disagreements, or near-English accuracy on the native continuation task, would overturn the paper's data foundation or its translation-artifact explanation respectively.","tokens_in":23248,"feed_emoji":"🇰🇬","tokens_out":9834,"duration_ms":80268,"temperature":0.7,"pith_summary":"This paper aims to make Kyrgyz, a low-resource Turkic language, a place where LLM ability can be measured rather than guessed. It assembles KyrgyzLLM-Bench — 7,977 natively written multiple-choice exam questions, 400 native reading-comprehension items, and carefully post-edited translations of four standard English tasks — and scores 26 open and proprietary models on it. Its central claim is that relative model quality transfers from English to Kyrgyz on reading and pronoun-resolution tasks (WinoGrande, BoolQ) and partly on knowledge questions (MMLU), but collapses on the event-continuation task HellaSwag; the authors attribute that collapse to translation-induced plausibility shifts, not to a genuine commonsense deficit. If the measurements hold, they give researchers the first reliable way to compare models in Kyrgyz and a caution about trusting translated benchmarks for low-resource languages. A careful reader would care because almost all multilingual evaluation currently relies on translated data that can silently mislead.","feed_headline":"LLM rankings transfer to Kyrgyz — except one translated test","feed_subtitle":"The suite's reading and reasoning scores track English; its translated commonsense test does not.","key_machinery":"The carrying object is the benchmark suite itself, built from three parts: KyrgyzMMLU, 7,977 multiple-choice items sourced from the official national school test and reviewed by domain experts; KyrgyzRC, 400 reading-comprehension questions over encyclopedic, news, literary, and math passages authored by native-speaking students and checked by supervisors and a professional linguist; and Kyrgyz translations of WinoGrande, HellaSwag, BoolQ, and TruthfulQA produced by two independent machine translations, ensemble-checked, then post-edited by native speakers. The explanatory mechanism doing the argumentative work is the plausibility-shift hypothesis — translated continuations lose the naturalne","core_discovery":"On the paper's own terms, the discovery is a measurable pattern: model rankings across families and scales are broadly preserved between English and Kyrgyz on BoolQ and WinoGrande, and to a lesser degree on MMLU, which the authors read as evidence that core reasoning and comprehension capabilities partially transfer across the language boundary. The same measurements show a large and irregular English–Kyrgyz gap on HellaSwag, and the paper explains this gap through plausibility shifts: translation disrupts the morphological cohesion, discourse flow, and idiomatic naturalness that event-continuation judgments depend on, so the Kyrgyz items no longer measure the same competence. The corollary","pith_inferences":["Editorial extension: The plausibility-shift account makes a testable prediction the paper does not run — a natively authored Kyrgyz event-continuation task should largely erase the English–Kyrgyz gap; building one would confirm or refute the central explanation.","Editorial extension: Because item quality rests on a sequential review with no inter-annotator agreement, an independent audit of a few hundred released KyrgyzMMLU and KyrgyzRC items would directly test the data foundation; the public release makes this the cheapest possible check.","Editorial extension: The pattern plausibly generalizes to other under-resourced agglutinative languages, where translated event-continuation benchmarks may systematically understate model ability for the same morphological reasons.","Editorial extension: If ranking transfer is as broad as reported, the benchmark could also serve as a lightweight multilingual probe — a model's Kyrgyz score on BoolQ and WinoGrande could be used as a low-cost monitor of language-coverage drift during fine-tuning."],"forward_implications":["If the benchmark is sound, it becomes the reference yardstick for Kyrgyz LLM capability: all datasets, code, and per-model results are released, so future model releases can be checked against it directly.","The cross-lingual transfer finding implies that English performance on BoolQ- and WinoGrande-style tasks is a useful predictor of Kyrgyz performance, which can guide model choice for Kyrgyz applications before expensive native evaluation.","The HellaSwag result implies that translated event-continuation benchmarks should be abandoned or rewritten natively for agglutinative languages, since their scores conflate translation fidelity with reasoning.","The few-shot results imply that in-context learning is not a guaranteed win for low-resource languages: gains on native reading comprehension are real for open models, but translated-task scores can drop, so few-shot comparisons should always be read against zero-shot baselines.","The overall English–Kyrgyz accuracy gap on native tasks implies that pretraining corpus imbalance, not task difficulty, is the binding constraint, pointing to native Kyrgyz data collection as the highest-leverage next step."],"fun_headline_variants":["KyrgyzLLM-Bench: Rankings transfer, but HellaSwag gap emerges","Translation artifacts skew LLM performance on Kyrgyz commonsense","LLM ranking transfer fails on translated HellaSwag for Kyrgyz","Cross-lingual LLM transfer holds, but translated HellaSwag doesn't","Kyrgyz benchmark reveals translation-induced plausibility shifts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every score and ranking in the paper rests on the assumption that the released KyrgyzMMLU and KyrgyzRC items are correct, unambiguous, and culturally sound — the review was sequential rather than parallel and no inter-annotator agreement was computed (the paper states Cohen's κ was not measured), so systematic item errors would quietly invalidate all conclusions built on these datasets.","fun_headline_variants_meta":{"raw":{"variants":["KyrgyzLLM-Bench: Rankings transfer, but HellaSwag gap emerges","Translation artifacts skew LLM performance on Kyrgyz commonsense","LLM ranking transfer fails on translated HellaSwag for Kyrgyz","Cross-lingual LLM transfer holds, but translated HellaSwag doesn't","Kyrgyz benchmark reveals translation-induced plausibility shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":3883,"prompt_tokens":782,"completion_tokens":3101,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":3017}},"tokens_in":526,"tokens_out":3101,"duration_ms":21888,"temperature":1.0,"reasoning_tokens":3017,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:47:45.336089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score a random sample of KyrgyzMMLU and KyrgyzRC with an independent panel of native-speaker annotators unconnected to the authors, and build a natively authored Kyrgyz HellaSwag-style continuation task; then rerun the same 26 models on it. Widespread item disagreements, or near-English accuracy on the native continuation task, would overturn the paper's data foundation or its translation-artifact explanation respectively.","supporting_citations":[],"review_version":1}