{"id":"b69ece17-4b50-4d2c-b79f-7c2483a42314","arxiv_id":"2505.13498","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"IRLBench provides 1,700 parallel English-Irish exam questions and shows state-of-the-art LLMs perform markedly worse in Irish than in English.","lead":"A new benchmark, IRLBench, uses 2024 Irish Leaving Certificate exams with parallel questions in English and Irish to test how well AI models reason in a low-resource language. Results show even the best models score much lower in Irish than English, and often fail to answer in Irish at all.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline English-Irish gap rests on an unvalidated LLM-as-a-judge, with no human agreement data and unexplained arithmetic inconsistencies in the reported averages.","rationale":"The reader's weakest assumption correctly identifies the unvalidated LLM-as-a-judge as the most load-bearing point: all reported accuracies, including the headline 76.2% versus 55.8% gap, flow through gemini-2.5-flash's binary correctness decisions and the FastText language-fidelity threshold. No human agreement or judge-validity study is provided, and the paper itself flags the robustness concern. My independent reading of Section 3.3 and Section 5 confirms that the evaluation methodology is the least-secure link in the central claim. I also noticed additional internal arithmetic issues in Table 3 where the reported group averages do not match the weighted combination of per-group percentages; this is secondary but reinforces the need for a careful numerical audit. The benchmark construction pipeline and the qualitative direction of the gap are plausible, and the authors do release the dataset and code, which is real evidence of good faith. However, the exact numerical claims should be conditional on judge validation. Since the reader already returned CONDITIONAL, my assessment does not change the verdict, hence UNCHANGED.","tokens_in":9958,"tokens_out":2568,"duration_ms":26398,"concrete_test":"Take a stratified random sample of roughly 200 model responses (100 English, 100 Irish) across subjects and models including o4-mini and gpt-4.1. Have two independent human graders who are proficient in Irish and familiar with Leaving Certificate marking schemes grade each response as correct or incorrect using the same binary protocol given to gemini-2.5-flash. Compute Cohen's kappa between the human graders and between each human grader and the judge. Then compare the human-graded English-Irish accuracy gap with the reported 20.4% gap for o4-mini. If the human-judge kappa is below 0.6, or if the human-graded gap differs from the LLM-judged gap by more than about five percentage points, the headline performance-gap claim should be revised or presented with explicit uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is quantitative: o4-mini scores 76.2% in English versus 55.8% in Irish, and models produce valid Irish responses less than 80% of the time. Both numbers depend entirely on gemini-2.5-flash as a binary judge of open-ended Irish and English responses against official marking schemes (Section 3.3). The paper provides no calibration study, no human agreement analysis, and no judge-variance analysis. The Limitations section itself concedes that the LLM-as-a-judge paradigm 'may present challenges in scalability and robustness' (Section 5). If the judge is systematically stricter, more confused, or less reliable on Irish than on English, the reported 20.4% gap could be substantially inflated or deflated. The language-fidelity measure also relies on a FastText sentence-level classifier with an arbitrary 50% threshold, and the paper does not validate this threshold against human judgment or analyze how the judge handles code-switched or partially Irish responses. In addition, Table 3 contains internal arithmetic inconsistencies: the Irish group averages do not match the weighted combination of the per-group rows (e.g., aya-vision-8b is reported as 0.36 but the weighted row values give 0.31; o4-mini is reported as 55.82 but the weighted rows give 53.92). These inconsistencies do not by themselves overturn the qualitative finding, but they reduce confidence in the exact headline numbers. The benchmark itself is a valuable resource, and the direction of the gap is plausible, but the specific quantitative claims are not yet trustworthy without an independent check of the judge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IRLBench, a parallel English–Irish, multimodal benchmark of 1,700 open-ended questions and official marking schemes derived from the 2024 Irish Leaving Certificate higher-level exams across 12 subjects. The authors propose an LLM-as-a-judge protocol (gemini-2.5-flash) for binary correctness grading against official marking schemes, plus a FastText sentence-level language identification rule for scoring Irish-language fidelity. They evaluate six closed- and open-source models and report a persistent performance gap between English and Irish, with the best model (o4-mini) scoring 76.2% in English versus 55.8% in Irish, and most models producing valid Irish responses less than 80% of the time. The dataset and evaluation code are released.","tokens_in":10247,"tokens_out":2608,"duration_ms":24461,"significance":"If the evaluation protocol is trustworthy, IRLBench is a valuable and timely contribution to low-resource multilingual evaluation. Its grounding in official examinations and marking schemes gives it external validity, the parallel English–Irish design permits controlled cross-lingual comparisons, and the inclusion of multimodal items and open-ended generation goes beyond the multiple-choice format of most existing multilingual benchmarks. The release of the dataset and codebase is a concrete asset to the community. The reported finding that even strong reasoning models exhibit a large English–Irish gap, and the accompanying confidence miscalibration analysis, are potentially important. However, the central quantitative claims rest on an unvalidated LLM-as-a-judge, with no human agreement data, and the exact reported averages contain internal inconsistencies; these issues must be addressed before the headline numbers can be relied upon.","major_comments":[{"comment":"The central quantitative claims—the 76.2% versus 55.8% English–Irish gap for o4-mini and the language-fidelity rates below 80%—depend entirely on the reliability of gemini-2.5-flash as a binary judge grading open-ended Irish and English responses against official marking schemes. The paper reports no human agreement study, no judge calibration analysis, no judge-variance analysis, and no comparison with a second judge model. The paper's own Limitations section concedes that the LLM-as-a-judge paradigm 'may present challenges in scalability and robustness' (Section 5). If the judge is systematically stricter or more confused on Irish responses, or if it is biased by the language of the response, the reported gap could be substantially inflated or deflated. The authors should provide a sample-based human agreement study (e.g., Cohen's kappa or percentage agreement) for both English and Irish splits, report judge consistency across repeated runs, and ideally compare the primary judge against at least one alternative judge to estimate judge-induced variance.","section":"Section 3.3 / Section 4.1 / Section 5"},{"comment":"Table 3 contains internal arithmetic inconsistencies that reduce confidence in the reported averages. For example, the reported Irish average for aya-vision-8b is 0.36, but the weighted combination of the per-group rows (0.22×582 + 0.00×590 + 0.62×326 + 1.00×202)/1700 gives 0.31; similarly, the reported Irish average for o4-mini is 55.82, but the weighted combination of the rows gives approximately 53.91, and the reported English average for aya-vision-8b is 38.55 while the weighted rows give approximately 38.90. These are not rounding errors of the magnitude shown. The authors should recompute all averages with the correct group weights, or clearly explain the aggregation formula if it is not a simple weighted mean of the subject-group rows.","section":"Table 3"},{"comment":"The language-fidelity criterion is defined by an arbitrary threshold: a response is classified as non-Irish if more than 50% of its sentences are detected as English by a FastText sentence-level classifier. This threshold and the underlying classifier are not validated against human judgments, and no sensitivity analysis is reported. Code-switched or partially Irish responses may be misclassified by a sentence-level rule, and the FastText model's accuracy on Irish—especially on academic exam responses containing domain terms and proper nouns—is not assessed. The authors should report classifier accuracy for Irish (or a validation subset), and show how the reported language-fidelity percentages change under alternative thresholds (e.g., 30%, 50%, 70%) to establish that the 'less than 80%' claim is robust rather than an artifact of the chosen cutoff.","section":"Section 3.3 / Section 4.3"}],"minor_comments":[{"comment":"The text reads 'Science follows closely with 5982 samples'—this is a typo; it should be 582 samples, consistent with Table 2 and the stated total of 1,700.","section":"Section 3.2"},{"comment":"The word 'Noteably' should be 'Notably'; also consider avoiding the colloquial phrase \"failing almost entirely\" in favor of a more precise quantitative description of the near-zero Irish accuracy of aya-vision models.","section":"Section 4.2"},{"comment":"The analysis excludes models with extremely low accuracy but does not specify which models are excluded or the number of remaining valid responses per model; the 'less than 80%' claim would be easier to verify if per-model sample sizes and exclusion criteria were stated.","section":"Section 4.3 / Figure 6"},{"comment":"The binary-judge simplification is described as 'consistent and comparable,' but the paper does not discuss the degree to which binary correctness agrees with the official marking schemes' granular scoring; a brief analysis of score distribution or a calibration check would strengthen the claim.","section":"Section 3.3"},{"comment":"Figure 1 compares IRLBench with prior Irish benchmarks, including the authors' own IrishQA dataset ([12]); to avoid any appearance of self-serving comparison, the authors should note in the caption or text that IrishQA is from their prior work and clarify the saturation claim's independence from that baseline.","section":"Figure 1 / References [10], [11], [12]"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource itself is a solid contribution and well aligned with the journal's scope. The main barrier to acceptance is methodological: the central empirical claims are not yet supported without judge validation and corrected aggregation. I would be willing to review a revision that adds human-agreement evidence, judge robustness checks, sensitivity analysis for the language-fidelity threshold, and corrected Table 3 averages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. IRLBench is a genuinely useful resource: 1,700 parallel Irish-English questions drawn from the 2024 Leaving Certificate, open-ended, multimodal, with official marking schemes, human-verified, and released on Hugging Face and GitHub. That fills a real gap—the first open-ended parallel benchmark for an extremely low-resource language. The qualitative finding, that models do substantially worse in Irish than English, is plausible. But the specific numbers are not yet trustworthy. Correctness is scored by gemini-2.5-flash as a binary judge with no human agreement study, no calibration against exam grades, and no analysis of how the judge handles Irish. The authors' own Limitations section concedes the judge 'may present challenges in scalability and robustness.' If the judge is stricter or more confused on Irish, the reported 20-point gap for o4-mini is inflated. Table 3 also has arithmetic inconsistencies: aya-vision-8b's Irish average is given as 0.36 but the subgroup rows give 0.31; o4-mini's 55.82 vs. about 53.9. These don't overturn the qualitative story, but they make the exact claims untenable.\n\nWhat is genuinely new: the combination of parallel English-Irish, open-ended generation, multimodal questions, and official marking schemes. Existing Irish datasets are classification or reading comprehension; existing multilingual benchmarks are multiple-choice. The extraction pipeline—VLM-based question–marking-scheme pairing with human verification—is sensible and reusable. The language-fidelity metric (FastText with a 50% threshold) is simple, though the threshold is arbitrary and there is no sensitivity analysis.\n\nSoft spots, in proportion: the judge is load-bearing; a benchmark that claims to measure performance needs human agreement data on a sample of responses, and judge consistency across languages. The threshold should be probed. Typos and arithmetic should be fixed (e.g., '5982 samples' in Section 3.2 is clearly 582).\n\nWho this is for: people evaluating LLMs in low-resource or endangered languages, and builders of multilingual benchmarks. The dataset and code are the takeaway.\n\nRecommendation: send to peer review. The benchmark deserves publication; the evaluation numbers should be treated as provisional until the judge is validated and tables corrected.","headline":"A genuinely useful new benchmark for an extremely low-resource language, but the headline performance gap rests on an unvalidated LLM judge and some inconsistent arithmetic; referee it, but require human agreement data and fixed tables.","tokens_in":10763,"tokens_out":4050,"would_cite":true,"duration_ms":35489,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current LLMs are substantially weaker at open-ended reasoning in Irish than in English, with the best model dropping from 76.2% to 55.8% on parallel Leaving Certificate exam questions.","keywords":["Irish language","multilingual benchmark","low-resource languages","open-ended evaluation","LLM-as-a-judge","Leaving Certificate","language fidelity","multimodal reasoning"],"falsifier":"Have fluent Irish examiners double-score a random sample of model responses on the Irish split and compare their labels with gemini-2.5-flash's; if human-judge agreement is low or the gap shrinks under human scoring, the headline performance numbers fail.","tokens_in":9774,"feed_emoji":"🍀","tokens_out":5088,"duration_ms":48068,"temperature":0.7,"pith_summary":"IRLBench is a parallel English–Irish benchmark built from the 2024 Irish Leaving Certificate exams, containing 1,700 question-and-marking-scheme pairs across twelve subjects. The paper tries to show that open-ended generative evaluation in an extremely low-resource language exposes a persistent performance gap: the best model, o4-mini, scores 76.2% in English but only 55.8% in Irish, and most models produce valid Irish responses less than 80% of the time. The motivation is that existing multilingual benchmarks are multiple-choice, text-only, or already saturated, so they miss the language-generation and cultural-grounding failures this design surfaces. If the results hold, they imply that current LLM reasoning gains do not transfer to endangered languages.","feed_headline":"LLMs lose 20 points when the exam is in Irish","feed_subtitle":"Best model scores 76.2% in English but 55.8% in Irish on identical Leaving Certificate questions.","key_machinery":"The benchmark itself is the central object: 1,700 parallel English–Irish question/marking-scheme pairs extracted from official 2024 Leaving Certificate Higher Level exams, with 18.11% of samples containing images. Evaluation runs through two instruments: gemini-2.5-flash acting as an LLM judge that compares a model's free-form answer against the official marking scheme and assigns a correct/incorrect label, and a FastText sentence-level language identifier that classifies a response as Irish only if at least half of its sentences are Irish. The official marking scheme gives the judge a fixed scoring target, while the parallel structure makes the English–Irish gap a within-item comparison rather than a comparison across different datasets.","core_discovery":"The central discovery claimed is that, given identical exam questions in parallel English and Irish, every evaluated LLM performs substantially worse in Irish, with performance gaps exceeding 10 percentage points even for the strongest models, and that generative language fidelity is itself a bottleneck: models frequently answer in English when asked in Irish. The authors attribute this to weak multilingual transfer in current models, especially reasoning-optimized ones, and present IRLBench as the first multilingual, culturally grounded, parallel benchmark designed for open-ended generative evaluation in an extremely low-resource language scenario.","pith_inferences":["A direct extension would build the same parallel-exam benchmark for other endangered languages with official bilingual examinations, such as Welsh or Basque, to test whether the gap is specific to Irish or generalizes to any low-resource language.","If human examiners double-scored a random subset, the measured gap could shrink or grow; until then, part of the 20-point gap may be judge error rather than pure model inability.","The correlation between answering correctly and answering in Irish suggests that language generation and reasoning are not independent in low-resource settings, so improving Irish fluency might also improve correctness.","Because the paper reports model confidence above 80% despite Irish accuracy below 60%, IRLBench doubles as a testbed for confidence calibration in low-resource languages."],"forward_implications":["If IRLBench measures what it claims, any deployment of current LLMs for Irish-language education or public services should expect materially lower accuracy than in English, even on identical content.","Reasoning-optimized models show a larger English–Irish gap (20.4% for o4-mini versus 12.9% for gpt-4.1), so test-time scaling has not fixed low-resource multilingual transfer.","Because most models emit valid Irish less than 80% of the time, language fidelity must be tracked as a separate metric from factual correctness when evaluating low-resource generation.","The near-90% saturation of existing Irish-capable datasets such as SIB200 and Belebele is an artifact of multiple-choice formats; open-ended evaluation reveals much weaker capabilities.","Subject-level results, such as open-source models lagging most in science and least in social studies, point to where multilingual training data and evaluation effort are most needed."],"supporting_citations":[{"why":"Supplies the precedent of using educational exam materials as benchmark sources.","marker":"[13]"},{"why":"Establishes the LLM-as-a-judge evaluation approach that IRLBench adopts.","marker":"[14]"},{"why":"Flags potential biases in LLM-as-a-judge, the method the paper relies on for scoring.","marker":"[15]"},{"why":"Represents the open-ended evaluation style IRLBench extends to low-resource languages.","marker":"[5]"},{"why":"Documents cultural bias in translated benchmarks, motivating a culturally grounded parallel design.","marker":"[17]"},{"why":"Provides the closest existing multilingual multimodal benchmark and a baseline for comparison.","marker":"[18]"},{"why":"Shows existing Irish-capable topic-classification data that state-of-the-art models saturate near 90%.","marker":"[10]"},{"why":"Supplies the parallel reading-comprehension benchmark whose saturation motivates harder open-ended Irish evaluation.","marker":"[11]"},{"why":"Provides the Gemini models used for data extraction and for judging model responses.","marker":"[26]"},{"why":"Provides the FastText language identification used to measure whether responses are actually written in Irish.","marker":"[32]"}],"fun_headline_variants":["LLMs lose 20 points when exam switches to Irish","Irish exam gap: LLMs score 76% English, 56% Irish","LLMs dodge Irish questions, reply in English","IRLBench: first benchmark for endangered-language reasoning","Same exam, Irish version: LLMs trail by 20 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported English–Irish gap depends entirely on gemini-2.5-flash grading open-ended Irish and English answers against official marking schemes as reliably as a human examiner would, and the paper provides no human-agreement check on that judge.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lose 20 points when exam switches to Irish","Irish exam gap: LLMs score 76% English, 56% Irish","LLMs dodge Irish questions, reply in English","IRLBench: first benchmark for endangered-language reasoning","Same exam, Irish version: LLMs trail by 20 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3121,"prompt_tokens":911,"completion_tokens":2210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2124}},"tokens_in":527,"tokens_out":2210,"duration_ms":15407,"temperature":1.0,"reasoning_tokens":2124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:03:46.559874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have fluent Irish examiners double-score a random sample of model responses on the Irish split and compare their labels with gemini-2.5-flash's; if human-judge agreement is low or the gap shrinks under human scoring, the headline performance numbers fail.","supporting_citations":[{"cited_title":"Survey of cultural awareness in language models: Text and beyond,","cited_arxiv_id":null,"evidence_quote":"Supplies the precedent of using educational exam materials as benchmark sources."},{"cited_title":"Judging LLM-as-a-judge with MT-bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-a-judge evaluation approach that IRLBench adopts."},{"cited_title":"Justice or prejudice? quantifying biases in LLM-as-a-judge,","cited_arxiv_id":null,"evidence_quote":"Flags potential biases in LLM-as-a-judge, the method the paper relies on for scoring."},{"cited_title":"Humanity’s last exam,","cited_arxiv_id":null,"evidence_quote":"Represents the open-ended evaluation style IRLBench extends to low-resource languages."},{"cited_title":"Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation,","cited_arxiv_id":null,"evidence_quote":"Documents cultural bias in translated benchmarks, motivating a culturally grounded parallel design."},{"cited_title":"M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the closest existing multilingual multimodal benchmark and a baseline for comparison."},{"cited_title":"SIB- 200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,","cited_arxiv_id":null,"evidence_quote":"Shows existing Irish-capable topic-classification data that state-of-the-art models saturate near 90%."},{"cited_title":"The belebele benchmark: a parallel reading comprehension dataset in 122 language variants,","cited_arxiv_id":null,"evidence_quote":"Supplies the parallel reading-comprehension benchmark whose saturation motivates harder open-ended Irish evaluation."},{"cited_title":"Introducing gemini 2.0: our new ai model for the agentic era,","cited_arxiv_id":null,"evidence_quote":"Provides the Gemini models used for data extraction and for judging model responses."}],"review_version":1}