Pith. sign in

REVIEW 3 major objections 5 minor 37 references

IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that current LLMs are substantially weaker at open-ended reasoning in Irish than in English, with the best model dropping from 76.2% to 55.8% on parallel Leaving Certificate exam questions.

desk verdict A genuinely useful new benchmark for an extremely low-resource language, but the headline performance gap rests on an unvalidated LLM judge and some inconsistent arithmetic; referee it, but require human agreement data and fixed tables. read the letter →

arxiv 2505.13498 v1 pith:DB4M4BMD submitted 2025-05-16 cs.CL cs.AI

Khanh-Tung Tran , Barry O'Sullivan , Hoang D. Nguyen This is my paper · ORCID
classification cs.CLcs.AI
keywords Irishlanguagemultilingualbenchmarklow-resourcelanguagesopen-endedevaluationLLM-as-a-judgeLeavingCertificatefidelitymultimodalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IRLBench is a parallel English–Irish benchmark built from the 2024 Irish Leaving Certificate exams, containing 1,700 question-and-marking-scheme pairs across twelve subjects. The paper tries to show that open-ended generative evaluation in an extremely low-resource language exposes a persistent performance gap: the best model, o4-mini, scores 76.2% in English but only 55.8% in Irish, and most models produce valid Irish responses less than 80% of the time. The motivation is that existing multilingual benchmarks are multiple-choice, text-only, or already saturated, so they miss the language-generation and cultural-grounding failures this design surfaces. If the results hold, they imply that current LLM reasoning gains do not transfer to endangered languages.

What carries the argument

The benchmark itself is the central object: 1,700 parallel English–Irish question/marking-scheme pairs extracted from official 2024 Leaving Certificate Higher Level exams, with 18.11% of samples containing images. Evaluation runs through two instruments: gemini-2.5-flash acting as an LLM judge that compares a model's free-form answer against the official marking scheme and assigns a correct/incorrect label, and a FastText sentence-level language identifier that classifies a response as Irish only if at least half of its sentences are Irish. The official marking scheme gives the judge a fixed scoring target, while the parallel structure makes the English–Irish gap a within-item comparison rather than a comparison across different datasets.

What would settle it

Have fluent Irish examiners double-score a random sample of model responses on the Irish split and compare their labels with gemini-2.5-flash's; if human-judge agreement is low or the gap shrinks under human scoring, the headline performance numbers fail.

Watch

Extended reading notes

Core claim

The central discovery claimed is that, given identical exam questions in parallel English and Irish, every evaluated LLM performs substantially worse in Irish, with performance gaps exceeding 10 percentage points even for the strongest models, and that generative language fidelity is itself a bottleneck: models frequently answer in English when asked in Irish. The authors attribute this to weak multilingual transfer in current models, especially reasoning-optimized ones, and present IRLBench as the first multilingual, culturally grounded, parallel benchmark designed for open-ended generative evaluation in an extremely low-resource language scenario.

Load-bearing premise

The reported English–Irish gap depends entirely on gemini-2.5-flash grading open-ended Irish and English answers against official marking schemes as reliably as a human examiner would, and the paper provides no human-agreement check on that judge.

Editorial extensions

If this is right

  • If IRLBench measures what it claims, any deployment of current LLMs for Irish-language education or public services should expect materially lower accuracy than in English, even on identical content.
  • Reasoning-optimized models show a larger English–Irish gap (20.4% for o4-mini versus 12.9% for gpt-4.1), so test-time scaling has not fixed low-resource multilingual transfer.
  • Because most models emit valid Irish less than 80% of the time, language fidelity must be tracked as a separate metric from factual correctness when evaluating low-resource generation.
  • The near-90% saturation of existing Irish-capable datasets such as SIB200 and Belebele is an artifact of multiple-choice formats; open-ended evaluation reveals much weaker capabilities.
  • Subject-level results, such as open-source models lagging most in science and least in social studies, point to where multilingual training data and evaluation effort are most needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would build the same parallel-exam benchmark for other endangered languages with official bilingual examinations, such as Welsh or Basque, to test whether the gap is specific to Irish or generalizes to any low-resource language.
  • If human examiners double-scored a random subset, the measured gap could shrink or grow; until then, part of the 20-point gap may be judge error rather than pure model inability.
  • The correlation between answering correctly and answering in Irish suggests that language generation and reasoning are not independent in low-resource settings, so improving Irish fluency might also improve correctness.
  • Because the paper reports model confidence above 80% despite Irish accuracy below 60%, IRLBench doubles as a testbed for confidence calibration in low-resource languages.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces IRLBench, a parallel English–Irish, multimodal benchmark of 1,700 open-ended questions and official marking schemes derived from the 2024 Irish Leaving Certificate higher-level exams across 12 subjects. The authors propose an LLM-as-a-judge protocol (gemini-2.5-flash) for binary correctness grading against official marking schemes, plus a FastText sentence-level language identification rule for scoring Irish-language fidelity. They evaluate six closed- and open-source models and report a persistent performance gap between English and Irish, with the best model (o4-mini) scoring 76.2% in English versus 55.8% in Irish, and most models producing valid Irish responses less than 80% of the time. The dataset and evaluation code are released.

Significance. If the evaluation protocol is trustworthy, IRLBench is a valuable and timely contribution to low-resource multilingual evaluation. Its grounding in official examinations and marking schemes gives it external validity, the parallel English–Irish design permits controlled cross-lingual comparisons, and the inclusion of multimodal items and open-ended generation goes beyond the multiple-choice format of most existing multilingual benchmarks. The release of the dataset and codebase is a concrete asset to the community. The reported finding that even strong reasoning models exhibit a large English–Irish gap, and the accompanying confidence miscalibration analysis, are potentially important. However, the central quantitative claims rest on an unvalidated LLM-as-a-judge, with no human agreement data, and the exact reported averages contain internal inconsistencies; these issues must be addressed before the headline numbers can be relied upon.

major comments (3)
  1. [Section 3.3 / Section 4.1 / Section 5] The central quantitative claims—the 76.2% versus 55.8% English–Irish gap for o4-mini and the language-fidelity rates below 80%—depend entirely on the reliability of gemini-2.5-flash as a binary judge grading open-ended Irish and English responses against official marking schemes. The paper reports no human agreement study, no judge calibration analysis, no judge-variance analysis, and no comparison with a second judge model. The paper's own Limitations section concedes that the LLM-as-a-judge paradigm 'may present challenges in scalability and robustness' (Section 5). If the judge is systematically stricter or more confused on Irish responses, or if it is biased by the language of the response, the reported gap could be substantially inflated or deflated. The authors should provide a sample-based human agreement study (e.g., Cohen's kappa or percentage agreement) for both English and Irish splits, report judge consistency across repeated runs, and ideally compare the primary judge against at least one alternative judge to estimate judge-induced variance.
  2. [Table 3] Table 3 contains internal arithmetic inconsistencies that reduce confidence in the reported averages. For example, the reported Irish average for aya-vision-8b is 0.36, but the weighted combination of the per-group rows (0.22×582 + 0.00×590 + 0.62×326 + 1.00×202)/1700 gives 0.31; similarly, the reported Irish average for o4-mini is 55.82, but the weighted combination of the rows gives approximately 53.91, and the reported English average for aya-vision-8b is 38.55 while the weighted rows give approximately 38.90. These are not rounding errors of the magnitude shown. The authors should recompute all averages with the correct group weights, or clearly explain the aggregation formula if it is not a simple weighted mean of the subject-group rows.
  3. [Section 3.3 / Section 4.3] The language-fidelity criterion is defined by an arbitrary threshold: a response is classified as non-Irish if more than 50% of its sentences are detected as English by a FastText sentence-level classifier. This threshold and the underlying classifier are not validated against human judgments, and no sensitivity analysis is reported. Code-switched or partially Irish responses may be misclassified by a sentence-level rule, and the FastText model's accuracy on Irish—especially on academic exam responses containing domain terms and proper nouns—is not assessed. The authors should report classifier accuracy for Irish (or a validation subset), and show how the reported language-fidelity percentages change under alternative thresholds (e.g., 30%, 50%, 70%) to establish that the 'less than 80%' claim is robust rather than an artifact of the chosen cutoff.
minor comments (5)
  1. [Section 3.2] The text reads 'Science follows closely with 5982 samples'—this is a typo; it should be 582 samples, consistent with Table 2 and the stated total of 1,700.
  2. [Section 4.2] The word 'Noteably' should be 'Notably'; also consider avoiding the colloquial phrase "failing almost entirely" in favor of a more precise quantitative description of the near-zero Irish accuracy of aya-vision models.
  3. [Section 4.3 / Figure 6] The analysis excludes models with extremely low accuracy but does not specify which models are excluded or the number of remaining valid responses per model; the 'less than 80%' claim would be easier to verify if per-model sample sizes and exclusion criteria were stated.
  4. [Section 3.3] The binary-judge simplification is described as 'consistent and comparable,' but the paper does not discuss the degree to which binary correctness agrees with the official marking schemes' granular scoring; a brief analysis of score distribution or a calibration check would strengthen the claim.
  5. [Figure 1 / References [10], [11], [12]] Figure 1 compares IRLBench with prior Irish benchmarks, including the authors' own IrishQA dataset ([12]); to avoid any appearance of self-serving comparison, the authors should note in the caption or text that IrishQA is from their prior work and clarify the saturation claim's independence from that baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and its reported performance gap are externally grounded in official exam materials and measured outcomes, with no fitted parameter or self-citation chain doing derivational work.

full rationale

IRLBench's construction is anchored to the 2024 Irish Leaving Certificate exam papers and their official marking schemes (Section 3.1), so correctness is measured against an external academic standard rather than derived from the paper's own model outputs. The headline English–Irish gap (e.g., o4-mini 76.2% vs 55.8%, Section 4.1) and the language-fidelity percentages are observed evaluation scores produced by gemini-2.5-flash under the LLM-as-a-judge protocol and a stated 50% FastText threshold (Section 3.3). These are decision rules for measurement, not fitted parameters renamed as predictions, and no equation in the paper defines a derived result in terms of its own input. The paper cites its own IrishQA dataset [12] as an example of a saturated Irish benchmark, but that citation is only motivational context; the central benchmark and gap findings do not depend on the saturation claim. Concerns about judge reliability or arithmetic inconsistencies in Table 3 are validity and correctness issues, not circularity. Therefore, no circular step is established.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on four domain assumptions: the parallelism of the source exams, the reliability of the LLM judge, the reliability of the language detector, and the validity of binary marking. The only explicit free parameter is the 50% language-fidelity threshold. No new physical or conceptual entities are introduced.

free parameters (1)
  • language fidelity threshold = 50% of sentences detected as English
    Responses are classified as non-Irish if more than half of their sentences are detected as English by FastText (Section 3.3). This threshold is chosen without sensitivity analysis or validation against human judgments.
assumptions (4)
  • domain assumption English and Irish versions of each Leaving Certificate exam have identical content, making language the only variable.
    Stated in Section 3.1 ('Because each exam is available in parallel English and Irish versions, with identical content') but no verification of parallelism is provided.
  • domain assumption gemini-2.5-flash as LLM-as-a-judge provides valid binary correctness judgments against official marking schemes.
    Invoked in Section 3.3 for all correctness scores. No human agreement study or judge reliability analysis is presented.
  • domain assumption FastText language identification reliably distinguishes Irish from English in LLM-generated text.
    Used in Section 3.3 to compute language fidelity. No evaluation of the detector on model outputs is reported.
  • domain assumption Official marking schemes can be reduced to a binary correct/incorrect judgment without losing validity.
    The judge assigns binary labels (Section 3.3), deviating from the granular scoring of the official schemes, but the paper does not validate that this reduction preserves measurement quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation." pith.science (2026). https://pith.science/paper/DB4M4BMD

@misc{pith2026250513498,
  author       = {Pith},
  title        = {Pith review of: IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DB4M4BMD}},
  note         = {Machine review of arXiv:2505.13498}
}
read the original abstract

Recent advances in Large Language Models (LLMs) have demonstrated promising knowledge and reasoning abilities, yet their performance in multilingual and low-resource settings remains underexplored. Existing benchmarks often exhibit cultural bias, restrict evaluation to text-only, rely on multiple-choice formats, and, more importantly, are limited for extremely low-resource languages. To address these gaps, we introduce IRLBench, presented in parallel English and Irish, which is considered definitely endangered by UNESCO. Our benchmark consists of 12 representative subjects developed from the 2024 Irish Leaving Certificate exams, enabling fine-grained analysis of model capabilities across domains. By framing the task as long-form generation and leveraging the official marking scheme, it does not only support a comprehensive evaluation of correctness but also language fidelity. Our extensive experiments of leading closed-source and open-source LLMs reveal a persistent performance gap between English and Irish, in which models produce valid Irish responses less than 80\% of the time, and answer correctly 55.8\% of the time compared to 76.2\% in English for the best-performing model. We release IRLBench (https://huggingface.co/datasets/ReliableAI/IRLBench) and an accompanying evaluation codebase (https://github.com/ReML-AI/IRLBench) to enable future research on robust, culturally aware multilingual AI development.

Figures

Figures reproduced from arXiv: 2505.13498 by the authors.

Figure 1
Figure 1. Performance of State-of-the-Art models on recent Irish benchmarks compared to IRLBench. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrative examples showcasing the Irish-English parallel nature and the diverse tasks of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline. Recent English-centric datasets, including SPIQA [25] and Humanity’s Exam [5], have transitioned towards open-ended, free-form generative evaluation. However, there is still a need for such datasets for extremely low-resource languages. Our work extends this direction by introducing IRLBench, the first multilingual, culturally grounded, parallel benchmark specifically designed for open-ended generative eva… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distributions of subject groups and subjects as percentages of total amount of samples in IRLBench. 3.3 Benchmarking Task Evaluating open-ended generation tasks poses significant challenges, as answers that are correct often vary widely in wording and detail, making tr…
Figure 5
Figure 5. Figure 5: Accuracy scores on IRLBench per model and language. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Percentage of responses generated by models that are in Irish (on Irish split of IRLBench). [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Model self-reported confidences compared to performances [37]. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 33 canonical work pages

  1. [1]

    Evaluating large vision-and-language models on children’s mathematical olympiads,

    A. Cherian, K.-C. Peng, S. Lohit, J. Matthiesen, K. A. Smith, and J. B. Tenenbaum, “Evaluating large vision-and-language models on children’s mathematical olympiads,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024

    work page 2024

  2. [2]

    Towards reasoning in large language models: A survey,

    J. Huang and K. C.-C. Chang, “Towards reasoning in large language models: A survey,” inFind- ings of the Association for Computational Linguistics: ACL 2023 (A. Rogers, J. Boyd-Graber, and N. Okazaki, eds.), (Toronto, Canada), pp. 1049–1065, Association for Computational Linguistics, July 2023

    work page 2023

  3. [3]

    The bitter lesson learned from 2,000+ multilingual benchmarks,

    M. Wu, W. Wang, S. Liu, H. Yin, X. Wang, Y . Zhao, C. Lyu, L. Wang, W. Luo, and K. Zhang, “The bitter lesson learned from 2,000+ multilingual benchmarks,” 2025

    work page 2025

  4. [4]

    Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models,

    H. Sun, Y . Min, Z. Chen, W. X. Zhao, Z. Liu, Z. Wang, L. Fang, and J.-R. Wen, “Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models,” 2025

    work page 2025

  5. [5]

    Humanity’s last exam,

    L. Phan et al., “Humanity’s last exam,” 2025

    work page 2025

  6. [6]

    The multilingual mind : A survey of multilingual reasoning in language models,

    A. Ghosh, D. Datta, S. Saha, and C. Agarwal, “The multilingual mind : A survey of multilingual reasoning in language models,” 2025

    work page 2025

  7. [7]

    Scaling test-time compute for low-resource languages: Multilingual reasoning in llms,

    K.-T. Tran, B. O’Sullivan, and H. D. Nguyen, “Scaling test-time compute for low-resource languages: Multilingual reasoning in llms,” 2025

    work page 2025

  8. [8]

    Towards measuring and modeling “culture

    M. F. Adilazuarda, S. Mukherjee, P. Lavania, S. S. Singh, A. F. Aji, J. O’Neill, A. Modi, and M. Choudhury, “Towards measuring and modeling “culture” in LLMs: A survey,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, eds.), (Miami, Florida, USA), pp. 15763–15784, Assoc...

    work page 2024

Show all 37 references
  1. [9]

    Memory of Peoples Series, UNESCO, 3 ed., Feb

    UNESCO, ed., Atlas of the world’s languages in danger . Memory of Peoples Series, UNESCO, 3 ed., Feb. 2010

    2010

  2. [10]

    SIB- 200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,

    D. I. Adelani, H. Liu, X. Shen, N. Vassilyev, J. O. Alabi, Y . Mao, H. Gao, and E.-S. A. Lee, “SIB- 200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,” in Proceedings of the 18th Conference of the European Chapter of t...

    2024

  3. [11]

    The belebele benchmark: a parallel reading comprehension dataset in 122 language variants,

    L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa, “The belebele benchmark: a parallel reading comprehension dataset in 122 language variants,” inProceedings of the 62nd Annual Meeting of the Association ...

    2024

  4. [12]

    Uccix: Irish-excellence large language model,

    K.-T. Tran, B. O’Sullivan, and H. D. Nguyen, “Uccix: Irish-excellence large language model,” ECAI 2024, 2024

    2024

  5. [13]

    Survey of cultural awareness in language models: Text and beyond,

    S. Pawar, J. Park, J. Jin, A. Arora, J. Myung, S. Yadav, F. G. Haznitrama, I. Song, A. Oh, and I. Augenstein, “Survey of cultural awareness in language models: Text and beyond,” 2024

    2024

  6. [14]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and...

    2023

  7. [15]

    Justice or prejudice? quantifying biases in LLM-as-a-judge,

    J. Ye, Y . Wang, Y . Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y . Chen, N. V . Chawla, and X. Zhang, “Justice or prejudice? quantifying biases in LLM-as-a-judge,” in The Thirteenth International Conference on Learning Representations , 2025

    2025

  8. [16]

    Language models are multilingual chain-of-thought reasoners,

    F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. V osoughi, H. W. Chung, Y . Tay, S. Ruder, D. Zhou, D. Das, and J. Wei, “Language models are multilingual chain-of-thought reasoners,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  9. [17]

    Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation,

    S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y . Susanto, R. Ng, S. Longpre, W.-Y . Ko, S. Ruder, M. Smith, A. Bosselut, A. Oh, A. F. T. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. E...

    2025

  10. [18]

    M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,

    W. Zhang, M. Aljunied, C. Gao, Y . K. Chia, and L. Bing, “M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023

    2023

  11. [19]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021

    arXiv

  12. [20]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  13. [21]

    CMMLU: Measuring massive multitask language understanding in Chinese,

    H. Li, Y . Zhang, F. Koto, Y . Yang, H. Zhao, Y . Gong, N. Duan, and T. Baldwin, “CMMLU: Measuring massive multitask language understanding in Chinese,” inFindings of the Association for Computational Linguistics: ACL 2024 (L.-W. Ku, A. Martins, and V . Srikumar, eds.), (Bangk...

    2024

  14. [22]

    KMMLU: Measuring massive multitask language understanding in Korean,

    G. Son, H. Lee, S. Kim, S. Kim, N. Muennighoff, T. Choi, C. Park, K. M. Yoo, and S. Biderman, “KMMLU: Measuring massive multitask language understanding in Korean,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu- tation...

    2025

  15. [23]

    ArabicMMLU: Assessing massive multitask language understanding in Arabic,

    F. Koto, H. Li, S. Shatnawi, J. Doughman, A. Sadallah, A. Alraeesi, K. Almubarak, Z. Alyafeai, N. Sengupta, S. Shehata, N. Habash, P. Nakov, and T. Baldwin, “ArabicMMLU: Assessing massive multitask language understanding in Arabic,” in Findings of the Association for Com- puta...

    2024

  16. [24]

    None of the others: a general technique to distinguish reasoning from memorization in multiple-choice llm evaluation benchmarks,

    E. S. Salido, J. Gonzalo, and G. Marco, “None of the others: a general technique to distinguish reasoning from memorization in multiple-choice llm evaluation benchmarks,” 2025

    2025

  17. [25]

    SPIQA: A dataset for multimodal ques- tion answering on scientific papers,

    S. Pramanick, R. Chellappa, and S. Venugopalan, “SPIQA: A dataset for multimodal ques- tion answering on scientific papers,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  18. [26]

    Introducing gemini 2.0: our new ai model for the agentic era,

    Google, “Introducing gemini 2.0: our new ai model for the agentic era,” 2024

    2024

  19. [27]

    Gemini 2.0: Flash, flash-lite and pro,

    Google, “Gemini 2.0: Flash, flash-lite and pro,” 2025

    2025

  20. [28]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics (P. Isabelle, E. Charniak, and D. Lin, eds.), (Philadelphia, Pennsylva...

    2002

  21. [29]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out, (Barcelona, Spain), pp. 74–81, Association for Computational Linguistics, July 2004

    2004

  22. [30]

    LIMA: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. YU, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy, “LIMA: Less is more for alignment,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023

    2023

  23. [31]

    Bag of tricks for efficient text classification,

    A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759, 2016

    2016 arXiv

  24. [32]

    Fasttext.zip: Com- pressing text classification models,

    A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov, “Fasttext.zip: Com- pressing text classification models,” arXiv preprint arXiv:1612.03651, 2016

    2016 arXiv

  25. [33]

    Introducing openai o3 and o4-mini,

    OpenAI, “Introducing openai o3 and o4-mini,” 2025

    2025

  26. [34]

    Introducing gpt-4.1 in the api,

    OpenAI, “Introducing gpt-4.1 in the api,” 2025

    2025

  27. [35]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,

    M. AI, “The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,” 2025

    2025

  28. [36]

    Aya vision: Advancing the frontier of multilingual multimodality,

    S. Dash, Y . Nan, J. Dang, A. Ahmadian, S. Singh, M. Smith, B. Venkitesh, V . Shmyhlo, V . Aryabumi, W. Beller-Morales, J. Pekmez, J. Ozuzu, P. Richemond, A. Locatelli, N. Frosst, P. Blunsom, A. Gomez, I. Zhang, M. Fadaee, M. Govindassamy, S. Roy, M. Gallé, B. Ermis, A. Üstün,...

    2025

  29. [37]

    Measuring short-form factuality in large language models,

    J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Measuring short-form factuality in large language models,” 2024. 12

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.