Pith. sign in

REVIEW 2 major objections 5 minor 34 references

IMPACT: Inflectional Morphology Probes Across Complex Typologies

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs consistently judge grammatical inflections better than ungrammatical ones across five morphologically rich languages, with generation in between, and chain-of-thought fails to fix the gap.

desk verdict A solid, transparent morphology benchmark whose headline JY>Gen>JN asymmetry is robust, with a localized gold-label worry in Arabic reverse gender templates. read the letter →

arxiv 2506.23929 v1 pith:ZWOMR7F4 submitted 2025-06-30 cs.CL

Mohammed J. Saeed , Tommi Vehvilainen , Evgeny Fedoseev , Sevil Caliskan , Tatiana Vodolazova This is my paper
classification cs.CL
keywords IMPACTinflectionalmorphologymultilingualLLMevaluationgrammaticalityjudgementmorphologicallyrichlanguageschain-of-thoughtpromptingLLM-as-a-judgetemplate-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces IMPACT, a synthetically generated suite of roughly 33,000 unit-test-style cases that probe inflectional morphology in Arabic, Russian, Finnish, Turkish, and Hebrew, with English as a sanity check. Its central finding is an asymmetry: across eight multilingual LLMs, models judge grammatically correct inflections well, judge ungrammatical inflections poorly, and sit in between when asked to generate the correct inflection themselves. The paper also shows that chain-of-thought prompting and a dedicated thinking mode produce inconsistent, sometimes harmful, effects rather than reliable gains. A sympathetic reader should care because the result suggests that surface fluency and even good acceptance of grammatical forms can coexist with a weak grasp of morphological rules, which matters for anyone using LLMs to evaluate or generate text in non-English languages.

What carries the argument

The carrying mechanism is a template-based generator. Each template is a blueprint with placeholders, for example '[NAME] [VERB]' for subject-verb agreement, and each unique combination of placeholder features (name gender, number, verb tense, mood, and so on) forms an evaluation unit, the smallest tested element. Inflected word forms come from the UniMorph lexicon, and negative examples are created by perturbing inflections so they break an agreement rule, a case rule, or a vowel-harmony pattern. Scores are computed per evaluation unit and aggregated with a harmonic mean over units, so a template's score is dragged down by any single weak unit; language-level and model-level aggregates use the arithmetic mean. The two test scenarios are Generation (fill in the missing inflection) and Judgement (answer Yes or No to whether the utterance is grammatical).

What would settle it

Take a human-verified sample of the generated items, ideally all items for one language and template, have native speakers annotate grammaticality, then recompute Gen, JY, and JN; if the JY-over-JN gap shrinks or flips once gold-label errors are removed, part of the asymmetry is an artifact of the gold data. A second check is to measure each model's overall Yes rate on ungrammatical items; if models answer Yes near-uniformly regardless of template, the deficit is response leniency rather than missing morphological knowledge.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLMs' morphological competence is lopsided: they are much better at recognizing well-formed inflections than at rejecting malformed ones. Across all five languages and across both shared templates (subject-verb agreement, imperative agreement, adjective-noun agreement) and language-specific templates (Arabic and Hebrew reverse gender agreement, Russian motion verbs, Turkish vowel and consonant harmony and evidentiality, Finnish lexical case and vowel harmony), judgement accuracy for grammatical items (JY) is consistently higher than judgement accuracy for ungrammatical items (JN), with generation accuracy (Gen) falling between. The paper further claims that chain-of-thought and thinking-mode prompting help on some templates but degrade performance on others, and that qualitative inspection of reasoning traces shows models often state the correct morphological rule while failing to apply it, or accept errors because they silently repair the input.

Load-bearing premise

The load-bearing premise is that the gold inflections from UniMorph and the template agreement logic are correct; the paper's limitations section concedes that UniMorph is not fully human-verified and that only some examples were manually checked.

Editorial extensions

If this is right

  • Any evaluation of multilingual LLMs that relies only on grammatical examples will overstate morphological competence; ungrammatical probes are needed to expose the gap.
  • LLM-as-a-judge pipelines for non-English text inherit this weakness: models will accept morphologically malformed output even when they can recognize the well-formed version.
  • Chain-of-thought and thinking modes should be validated per template and per language, since the paper finds both gains and losses and neither strategy reliably improves negative judgement.
  • Generation accuracy alone is not a sufficient diagnostic; the paper's three-way Gen/JY/JN split shows that generation sits between recognition and rejection, so each task reveals a different facet of grammatical knowledge.
  • The released IMPACT framework gives other researchers unit-test-style probes for Arabic, Russian, Finnish, Turkish, and Hebrew, including hard cases such as dual agreement and vowel harmony.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper leaves implicit: if a control set of ungrammatical items is built by breaking agreement in English or by purely orthographic errors and the JN deficit persists, the asymmetry is a general judgment bias rather than a morphology-specific one; if it vanishes, the deficit is tied to inflectional complexity.
  • Because the gold inflections come from UniMorph and the paper's own limitations section concedes the data is not fully human-verified, an independent native-speaker audit of the negative items would settle whether part of the JN gap is an artifact of gold-label errors.
  • The template method could be carried into derivational morphology, dialectal variants, or additional language families such as other Semitic or Uralic languages, where the same unit-test structure would make cross-linguistic comparisons of the JY/JN gap directly measurable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces IMPACT, a synthetic template-based benchmark for probing LLM competence on inflectional morphology in Arabic, Hebrew, Russian, Turkish, and Finnish, with English as a sanity check. The benchmark includes common templates (subject-verb agreement, imperatives, adjective-noun agreement) and language-specific templates (Arabic/Hebrew reverse gender agreement with numerals, Russian motion verbs, Turkish locative and evidentiality, Finnish lexical case and vowel harmony). Models are evaluated in a Generation (fill-in-the-blank) and a Judgement (grammaticality) scenario, with Judgement scored separately for grammatical (JY) and ungrammatical (JN) items. Eight multilingual LLMs are tested, and the main empirical claims are that JY scores typically exceed JN scores, that Generation performance falls in between, and that Chain-of-Thought and Thinking Mode have inconsistent effects. The framework is publicly released.

Significance. If the empirical claims hold up, IMPACT is a useful contribution: it is a transparent, publicly released benchmark with a clear pipeline, native-speaker involvement in template creation, and a harmonic-mean aggregation that penalizes the weakest evaluation units. The benchmark has no fitted parameters or model-derived labels, so the headline comparisons are not circular. The strongest and most interesting claim is the systematic gap between judging grammatical versus ungrammatical inflections, which, if real, points to a concrete weakness in LLM evaluation and training for morphologically rich languages. The paper also provides evidence for the inconsistency of CoT and Thinking Mode. However, the central JN scores rest on the correctness of template-generated negative gold labels, which the paper itself only partially verifies; the benchmark's lasting value depends on the analyses requested in the major comments.

major comments (2)
  1. [Sec. 3.3, Appendix J, Limitations] The paper's central claim that JY > Gen > JN hinges on the correctness of negative gold labels. Section 3.3 constructs ungrammatical examples by perturbing inflections according to template logic, but the paper itself fronts two caveats: Appendix J concedes that ara-1's 'reverse gender agreement' is a simplification, and the cited Alqarni (2021) argues there is no gender polarity in Arabic numeral phrases; if that is correct, many ara-1 negatives are not unambiguously ungrammatical. Section 3.3 also notes for rus-1 that native speakers 'sometimes mix both unidirectional and multidirectional forms,' which undermines the gold label for rus-1 negatives. Since the Limitations admit UniMorph is 'not fully human-verified' and only 'some examples' were manually checked, the same risk applies to all templates. Please provide native-speaker verification for a random sample of negative items (especially ara-1, heb-1, rus-1, fin-1), report agreement, and if the labels cannot be defended, exclude those templates from the cross-language JY>JN claim.
  2. [Sec. 4.1] Invalid judge responses are replaced by random sampling (Section 4.1). Valid response rates range from 0.919 (Qwen3 Non-Thinking) to 0.9999 (GFL2, Table 7), so this imputation affects a non-negligible fraction of judgments for some models. Since JN scores are near chance for several models, random imputation can materially change the reported values and the JY>Gen>JN ranking. Please report a sensitivity analysis (e.g., counting invalid responses as incorrect or excluding them) and confirm that the central pattern and the model rankings in Tables 9 and 2 are robust to the imputation rule.
minor comments (5)
  1. [Sec. 3.2, com-1] The sentence 'In Arabic and Hebrew, [NAME] and [VERB] must agree in both gender and plurality, with a dedicated inflection for the dual form' is inconsistent with Table 16, which lists no dual feature for Hebrew (noun plurality: [PL, SG]); only Arabic uses the dual in this template. Please correct.
  2. [Sec. 1] The sentence 'We introduce IMPACT, a collection of tests designed to assess whether LLMs’ grasp of inflectional morphology across languages (Figure 2)' is incomplete; it should be 'designed to assess whether LLMs grasp inflectional morphology' or similar.
  3. [Abstract / Sec. 5] The paper states that the IMPACT framework is publicly released but provides no URL, repository identifier, or license. Please add a link to the released code and data.
  4. [Limitations] The Limitations state that no repeated trials were conducted; given that the CoT and Thinking-Mode deltas in Tables 10, 11, 13, and 14 are often small (e.g., +0.010 or -0.015), the authors should either add bootstrap confidence intervals over evaluation units or explicitly caution against interpreting small deltas.
  5. [Appendix A] The prompt examples in Appendix A show truncated utterances (e.g., 'Utterance: Mohammed' for the generation prompt), which makes it impossible to follow the full input; please include complete example utterances.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IMPACT is an externally grounded benchmark; all reported quantities are direct accuracies with no fitted parameters.

full rationale

The paper is an empirical evaluation, not a derivation. Template-generated utterances are scored against gold inflections taken from UniMorph (Batsuren et al., 2022) and checked by native speakers; no model output, template score, or model score is fed back into the definition of the gold labels. The central claims — JY > Gen > JN and inconsistent CoT effects — are direct accuracy comparisons on these externally defined labels. There is no fitted parameter that is later renamed a prediction, no uniqueness theorem, and no load-bearing self-citation. The only self-reference is that IMPACT's templates both create the test items and define the scoring criterion; this is inherent to any behavioral benchmark and does not reduce a claimed derivation to its input. The acknowledged limitations (UniMorph not fully human-verified; Appendix J's simplification of Arabic numeral gender polarity per Alqarni 2021) are potential threats to gold-label validity, which would affect accuracy levels and possibly the JY/JN asymmetry, but they are correctness risks, not circularity: the benchmark's labels are sourced externally rather than derived from the models' own judgments. CoT and Thinking Mode deltas are computed from the same scoring rule before and after a prompt change, so they do not smuggle in the conclusion. Score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper fits no parameters and postulates no new entities. It rests on two domain assumptions: UniMorph gold inflections are correct enough for scoring, and random replacement of invalid judge responses is an unbiased proxy. Both are stated in the paper.

assumptions (2)
  • domain assumption UniMorph 4.0 supplies correct inflected forms for template word lists.
    Section 3.1 fills placeholders with UniMorph forms; Section 5 Limitations acknowledges the dataset is not fully human-verified.
  • domain assumption Randomly sampling a response for an invalid LLM judge output does not bias accuracy.
    Section 4.1 states invalid responses are labeled and randomly sampled following Bavaresco et al. (2024); the paper does not analyze how this affects JY/JN scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IMPACT: Inflectional Morphology Probes Across Complex Typologies." pith.science (2026). https://pith.science/paper/ZWOMR7F4

@misc{pith2026250623929,
  author       = {Pith},
  title        = {Pith review of: IMPACT: Inflectional Morphology Probes Across Complex Typologies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZWOMR7F4}},
  note         = {Machine review of arXiv:2506.23929}
}
read the original abstract

Large Language Models (LLMs) have shown significant progress on various multilingual benchmarks and are increasingly used to generate and evaluate text in non-English languages. However, while they may produce fluent outputs, it remains unclear to what extent these models truly grasp the underlying linguistic complexity of those languages, particularly in morphology. To investigate this, we introduce IMPACT, a synthetically generated evaluation framework focused on inflectional morphology, which we publicly release, designed to evaluate LLM performance across five morphologically rich languages: Arabic, Russian, Finnish, Turkish, and Hebrew. IMPACT includes unit-test-style cases covering both shared and language-specific phenomena, from basic verb inflections (e.g., tense, number, gender) to unique features like Arabic's reverse gender agreement and vowel harmony in Finnish and Turkish. We assess eight multilingual LLMs that, despite strong English performance, struggle with other languages and uncommon morphological patterns, especially when judging ungrammatical examples. We also show that Chain of Thought and Thinking Models can degrade performance. Our work exposes gaps in LLMs' handling of linguistic complexity, pointing to clear room for improvement. To support further research, we publicly release the IMPACT framework.

Figures

Figures reproduced from arXiv: 2506.23929 by the authors.

Figure 1
Figure 1. The top English examples show that (i) the adjective "big" remains unchanged for singu￾lar (SG) and plural (PL) nouns, and (ii) the number "four" inherently signals plurality. The language model (robot icon) correctly generates both. Be￾low, the same examples in Arabic illustrate com￾plex agreement with gender (MASC/FEM) and dual (DU) forms. Green cells indicate correct fea￾ture matches; red cells show errors. On th… view at source ↗
Figure 2
Figure 2. IMPACT workflow: (1) Identify tem￾plates and prepare placeholder data (Section 3.1); (2) generate utterances with correct inflections us￾ing morphological agreement logic (Section 3.2); (3) create prompts for Generation and Judgement tasks (Section 3.3), including ungrammatical ex￾amples by mismatching inflections; (4) evaluate LLM performance. While large language models (LLMs) consis￾tently show improvements on va… view at source ↗
Figure 3
Figure 3. Radar plots showing Gen, JY, and JN scores for each language. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Turkish is a highly agglutinative language, meaning that each affix usually corresponds to [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muteb Alqarni. 2021. https://doi.org/10.1162/ling_a_00385 No gender polarity in arabic numeral phrases . Linguistic Inquiry, 52(3):441--472

    work page doi:10.1162/ling_a_00385 2021

  4. [4]

    Dang Anh, Limor Raviv, and Lukas Galke. 2024. https://doi.org/10.18653/v1/2024.cmcl-1.15 Morphology matters: Probing the cross-linguistic morphological generalization abilities of large language models through a wug test . In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 177--188, Bangkok, Thailand. Association for...

    doi:10.18653/v1/2024.cmcl-1.15 2024

  5. [5]

    Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kiera \'s , G \'a bor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, and 76 others. 2022. https://aclanthology....

    work page 2022

  6. [6]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. htt...

    arXiv doi:10.48550/arxiv.2406.18403 2024

  7. [7]

    Smith, and Luke Zettlemoyer

    Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.604 Breaking the curse of multilinguality with cross-lingual expert language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10822--1...

    doi:10.18653/v1/2024.emnlp-main.604 2024

  8. [8]

    Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen

    Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.236 When is multilinguality a curse? language modeling for 250 high- and low-resource languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074--4096, Miami, Florida, USA. Association for C...

    doi:10.18653/v1/2024.emnlp-main.236 2024

Show all 34 references
  1. [9]

    Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, and Kun Han. 2024. https://api.semanticscholar.org/CorpusID:267095066 Orion-14b: Open-source multilingual large language models . ArXiv, abs/2401.12246

    2024 arXiv

  2. [10]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 156...

    2023 doi

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  4. [12]

    Avia Efrat, Or Honovich, and Omer Levy. 2023. https://doi.org/10.18653/v1/2023.findings-acl.666 LM entry: A language model benchmark of elementary language tasks . In Findings of the Association for Computational Linguistics: ACL 2023, pages 10476--10501, Toronto, Canada. Asso...

    2023 doi

  5. [13]

    Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2024. https://aclanthology.org/2024.findings-eacl.71/ Are large language model-based evaluators the solution to scaling up multilingual evaluation?...

    2024

  6. [14]

    Ester Hlavnova and Sebastian Ruder. 2023. https://doi.org/10.18653/v1/2023.acl-long.396 Empowering cross-lingual behavioral testing of NLP models with typological features . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2023 doi

  7. [15]

    Oskar Holmstr \"o m and Ehsan Doostmohammadi. 2023. https://aclanthology.org/2023.nodalida-1.62/ Making instruction finetuning accessible to non- E nglish languages: A case study on S wedish models . In Proceedings of the 24th Nordic Conference on Computational Linguistics (No...

    2023

  8. [16]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.826 Not all languages are created equal in LLM s: Improving multilingual capability by cross-lingual-thought prompting . In Findings of...

    2023 doi

  9. [17]

    Karthikeyan K, Shaily Bhatt, Pankaj Singh, Somak Aditya, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022. https://doi.org/10.18653/v1/2022.findings-aacl.27 Multilingual C heck L ist: Generation and evaluation . In Findings of the Association for Computational ...

    2022 doi

  10. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  11. [19]

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. 2025. https://arxiv.org/abs/2504.09858 Reasoning models can be effective without thinking . Preprint, arXiv:2504.09858

    2025 arXiv

  12. [20]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su \'a rez, Yoann Dupont, Laurent Romary, \'E ric de la Clergerie, Djam \'e Seddah, and Beno \^i t Sagot. 2020. https://doi.org/10.18653/v1/2020.acl-main.645 C amem BERT : a tasty F rench language model . In Proceedings of the ...

    2020 doi

  13. [21]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  14. [22]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020 doi

  15. [23]

    Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Beno \^i t Sagot, Niklas Muennighoff, Albert Villanova del Moral, and 372 others

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili'c, Daniel Hesslow, Roman Castagn'e, Alexandra Sasha Luccioni, François Yvon, Matthias Gall \'e , Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Be...

    2022 arXiv

  16. [24]

    Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. https://doi.org/10.18653/v1/2024.findings-acl.136 Multilingual instruction tuning with just a pinch of multilinguality . In Findings of the Association for Computational Linguistics:...

    2024 doi

  17. [25]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://openreview.net/forum?id=fR3wGCk-IXp Language models are multilingual chain-of-thought reasone...

    2023

  18. [26]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. https://openreview.net/forum?id=4FWAwZtd2n Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning . In Proceedings of the International Conference on Learning Rep...

    2025

  19. [27]

    Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tormo-Bañuelos, and Seungone Kim. 2025. https://arxiv.org/abs/2410.17578 Mm-eval: A multilingual meta-evaluation benchmark for llm-as-a-judge and...

    2025 arXiv

  20. [28]

    Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2025. https://openreview.net/forum?id=w6nlcS8Kkn To cot or not to cot? chain-of-thought helps mainly on math and symbolic re...

    2025

  21. [29]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2025 arXiv

  22. [30]

    A. Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. https://ap...

    2024

  23. [31]

    Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi

    Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2024. https://openreview.net/forum?id=CF8H8MS5P8 The generative AI ...

    2024

  24. [32]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  25. [33]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

    2025 arXiv

  26. [34]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.