Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

How Reliable is Multilingual LLM-as-a-Judge?

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-as-a-Judge is not yet reliable for multilingual predictions: the same parallel answer gets inconsistent verdicts across languages, with average Fleiss' Kappa around 0.3.

desk verdict Useful measurement buried under a statistically compromised headline: the all-gold design makes Fleiss' Kappa uninterpretable, and the abstract average doesn't match the table. read the letter →

arxiv 2505.12201 v1 pith:FCKUV6WW submitted 2025-05-18 cs.CL

Xiyan Fu , Wei Liu This is my paper · ORCID
classification cs.CL
keywords multilingualLLM-as-a-JudgejudgmentconsistencyFleiss'Kappacross-lingualevaluationlow-resourcelanguagespointwiseensemblemajorityvote
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a foundational assumption behind multilingual LLM-as-a-Judge: that a good judge gives the same verdict for the same content regardless of the language it is expressed in. The authors build parallel test sets in 25 languages across five tasks, run five LLM judges, and treat each language as one rater to compute Fleiss' $\kappa$. They report an average $\kappa$ near 0.3, with some model-task pairs far lower, meaning the same judge frequently disagrees with itself about semantically identical answers. Consistency is worst for low-resource languages, and neither multilingual pretraining nor larger model size fixes it. If this holds, automatic evaluation results cannot be treated as language-neutral, and multilingual benchmarks need a separate reliability check for their judges.

What carries the argument

The load-bearing device is parallel multilingual data combined with Fleiss' Kappa computed across languages: each language's judgment is treated as a rater, so that agreement measures whether the verdict depends on content or on language. Since the same underlying item appears in every language, language is the only variable. The judge prompt is held fixed in English with a placeholder naming the evaluation language, following prior practice; pointwise evaluation (one candidate judged against a rubric) is used rather than pairwise comparison because parallel incorrect candidates are hard to obtain. The result is a measurement scheme that isolates cross-lingual judgment reliability from task accuracy.

What would settle it

Build a parallel corpus of model-generated answers of deliberately mixed quality in the same 25 languages, have the same five judges score them, and recompute Fleiss' $\kappa$ across languages; if agreement rises well above the reported 0.3, the paper's reliability conclusion may not transfer to real judging situations.

Watch

Extended reading notes

Core claim

The paper's central discovery is that LLM-as-a-Judge is not language-invariant: when the same question-answer pair is translated into 25 languages and presented in a parallel format, a single judge model frequently disagrees with itself. Treating each language's verdict as one rater, the authors compute Fleiss' $\kappa$ and find average agreement around 0.3 across five models and five tasks, with some model-task combinations falling below 0.1. Good task accuracy does not rescue consistency: a judge can be mostly correct in every language while still disagreeing across languages, and Spearman correlations between accuracy and Kappa vary in sign across tasks. Low-resource languages show particularly low agreement with English judgments, and neither multilingually trained models such as Aya-Expanse nor larger parameter counts reliably close the gap. The paper also reports that an ensemble majority vote of three open-source judges improves consistency relative to the worst single judge in most settings, and that binary Yes/No judgments are more consistent than 1-5 grade judgments.

Load-bearing premise

The study assumes that judging perfectly correct, parallel reference answers behaves like judging the mixed-quality, sometimes wrong candidate outputs that LLM-as-a-Judge meets in real use, and that judging one candidate at a time captures the reliability of judging pairs of candidates as well.

Editorial extensions

If this is right

  • Multilingual evaluation results obtained by a single LLM judge in one language cannot be assumed to hold in another; reported numbers should be accompanied by a cross-language consistency check.
  • Low-resource languages are the weakest link, so evaluations reported for languages like Telugu or Swahili need the most scrutiny.
  • Since larger models and multilingual pretraining do not systematically improve consistency, simply upgrading the judge is unlikely to make multilingual evaluation trustworthy.
  • The ensemble majority vote of open-source judges is a practical remedy that usually beats the worst single judge, and binary judgments with explanatory prompts are more consistent than fine-grained grades.
  • A judge that scores well on accuracy in each language can still be inconsistent across languages, so accuracy and consistency should be reported as separate axes of judge quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Pith inference: because the prompt is written in English with a target-language placeholder, part of the measured inconsistency may come from the prompt language rather than the judged content; a direct test would rerun the experiment with fully native prompts per language.
  • Pith inference: the pointwise-only design leaves pairwise comparative judging untested; constructing parallel incorrect-candidate pairs and measuring Kappa there would reveal whether the instability is specific to absolute grading.
  • Pith inference: the near-zero or negative correlation between accuracy and Kappa suggests a judge could be uniformly wrong yet 'consistent' in appearance, so practical reliability checks should also calibrate against human labels, not just cross-language agreement.
  • Pith inference: the task-dependence of consistency suggests a certification scheme in which a judge is validated per task-language cell before its outputs are used, rather than trusted on the strength of its average multilingual competence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates whether LLM-as-a-Judge is reliable in multilingual settings by measuring cross-language consistency of judgments on parallel multilingual data. For five models, five tasks, and 25 languages, each item's ground-truth answer is judged separately in each language, and Fleiss' Kappa is computed across languages, treating each language as a rater. The authors report an average Fleiss' Kappa of about 0.3, conclude that LLMs are not yet reliable multilingual judges, analyze factors such as low-resource languages, task type, prompt design, and model scale, and propose a majority-vote ensemble of open-source models to improve consistency.

Significance. If the headline result were supported by an appropriate measure, the paper would be a useful, broad empirical caution for multilingual LLM-as-a-Judge and a practical contribution via the ensemble strategy. The study has real strengths: it uses parallel data across many languages and tasks, covers closed and open models, provides full prompt templates in the appendix, and involves no fitted parameters or circular derivations. However, the central evidence is currently compromised by a measurement-validity problem in the choice and interpretation of Fleiss' Kappa, so the significance of the conclusion is conditional on reanalysis.

major comments (3)
  1. [Section 3.4 and Table 2] The central consistency metric is not interpretable for the all-gold design. Because every judged item is a ground-truth answer (Section 3.3), the true rating is constant ('correct' or grade 5), and each language's marginal positive rate is the per-language accuracy. When the marginal rate is high, say p=0.95, two independently labeling languages agree with raw probability p^2+(1-p)^2≈0.905, yet the expected Fleiss' Kappa is 0. Thus values around 0.3 for Aya-Expanse on XQuAD, where accuracy is 96.86%, do not by themselves indicate large cross-language disagreement; they may indicate only that the few errors are not perfectly correlated across languages. The paper never reports raw pairwise agreement, percent agreement, or prevalence-adjusted coefficients such as PABAK, and it applies conventional Kappa benchmarks that assume variable true labels. This affects not only the headline absolute values but also the comparisons in Sections 4.2, 5.1, 5.4, and Table 5, because Kappa is compressed differently whenever per-language accuracy differs. The claim that LLMs are inconsistent across languages needs to be re-established with an appropriate agreement measure or raw agreement rates before it can support the paper's conclusions.
  2. [Abstract and Section 4.1] The reported average Fleiss' Kappa of about 0.3 is not consistent with Table 2. The arithmetic mean of the 25 Yes/No Kappa values is about 0.24, the mean of the 25 Grade Kappa values is about 0.17, and the overall mean of all 50 cells is about 0.21. No model or task subset in the table produces an average near 0.3 except isolated cells. The abstract and Section 4.1 should either report the actual aggregate value or explain which subset is being averaged.
  3. [Section 5.1 and Figure 4] The Cohen's Kappa analysis between English and other languages inherits the same prevalence artifact. For low-resource languages, per-language judgment accuracy is typically lower, which changes the expected chance agreement and therefore compresses or inflates Kappa regardless of pairwise agreement. The claim that consistency is particularly poor for low-resource languages (for example, the MGSM Telugu result near 0.002 in Section 5.1) is not supported without reporting raw agreement between the English and non-English judgments or using a prevalence-adjusted coefficient.
minor comments (4)
  1. [Section 1] The metric name 'BLUE' should be 'BLEU'.
  2. [Section 3.1] The phrase 'State-ot-the-art' should be 'state-of-the-art'.
  3. [Table 4] The column header 'Acc' appears to report average grades, not accuracy; the header should be aligned with the Grade setting or the values should be described consistently in the caption.
  4. [Figure 5] The legend contains 'Column2', which appears to be a placeholder, and the axis label is garbled; both should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is a direct empirical measurement of cross-language judgment consistency, with no fitted parameters or predictions defined in terms of the outcome.

full rationale

This is a pure empirical study. The central quantity, Fleiss' Kappa across languages, is computed directly from the models' binary or graded judgments on parallel gold answers (Sections 3.3-3.4, Table 2); no parameter is fitted to any subset of the data and no predicted quantity is defined in terms of the outcome it is used to explain. The ensemble strategy of Section 6 applies a majority vote post hoc to the same measured judgments; its reported Kappa improvement is a derived statistic, not an input. The claimed 'prediction' is the headline generalization that LLM-as-a-Judge is not language-consistent, which is an inductive summary of the measurements, not a quantity forced by construction. The authors themselves flag the main external-validity limitation in Section 9: pointwise judgment on gold answers was chosen because parallel multilingual incorrect candidates are hard to obtain; that is a scope limitation, not a circular step. A separate concern, raised by a skeptical reader, is that the all-gold design inflates chance agreement and makes conventional Fleiss Kappa benchmarks difficult to interpret; if valid, that undermines the strength of the empirical conclusion, but it is a measurement-validity critique and does not make the derivation circular. There are no load-bearing self-citations: citations to prior LLM-as-a-Judge work motivate the setup, but the measurements are new and independent. Score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study introduces no free parameters or invented entities. Its conclusions rest on four domain assumptions: parallel data equivalence, languages as raters, English-prompt universality, and transfer from gold answers to real candidates. The last is explicitly acknowledged as a limitation.

assumptions (4)
  • domain assumption Parallel datasets preserve semantic identity across languages, so language is the only variable.
    Used in Section 3.2 to justify that Fleiss' Kappa measures pure language bias rather than content differences. If translations shift meaning, the inconsistency could be partly content-driven.
  • domain assumption Treating each language's output as a rater yields a valid Fleiss' Kappa.
    Section 3.4 defines raters as languages. This is nonstandard but internally consistent; however, Fleiss' Kappa assumes independent raters, and languages are not independent raters in a strict sense.
  • domain assumption An English prompt with a language placeholder is a fair configuration for all models.
    Section 3.3 adopts this following Ahuja et al. (2023); if some models judge better with native-language prompts, the measured consistency may be pessimistically biased.
  • domain assumption Ground-truth answers are used as judged items, assuming judge behavior on perfect answers transfers to real candidates.
    Section 3.3 and the Limitation section: the authors acknowledge that parallel incorrect candidates are hard to obtain. This directly limits the external validity of the Kappa values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Reliable is Multilingual LLM-as-a-Judge?." pith.science (2026). https://pith.science/paper/FCKUV6WW

@misc{pith2026250512201,
  author       = {Pith},
  title        = {Pith review of: How Reliable is Multilingual LLM-as-a-Judge?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCKUV6WW}},
  note         = {Machine review of arXiv:2505.12201}
}
read the original abstract

LLM-as-a-Judge has emerged as a popular evaluation strategy, where advanced large language models assess generation results in alignment with human instructions. While these models serve as a promising alternative to human annotators, their reliability in multilingual evaluation remains uncertain. To bridge this gap, we conduct a comprehensive analysis of multilingual LLM-as-a-Judge. Specifically, we evaluate five models from different model families across five diverse tasks involving 25 languages. Our findings reveal that LLMs struggle to achieve consistent judgment results across languages, with an average Fleiss' Kappa of approximately 0.3, and some models performing even worse. To investigate the cause of inconsistency, we analyze various influencing factors. We observe that consistency varies significantly across languages, with particularly poor performance in low-resource languages. Additionally, we find that neither training on multilingual data nor increasing model scale directly improves judgment consistency. These findings suggest that LLMs are not yet reliable for evaluating multilingual predictions. We finally propose an ensemble strategy which improves the consistency of the multilingual judge in real-world applications.

Figures

Figures reproduced from arXiv: 2505.12201 by the authors.

Figure 1
Figure 1. Inconsistency in multilingual LLM-as-a￾Judge. Left part shows a multilingual Question An￾swering example. All question-answer pairs are parallel and perfectly aligned across languages. Human evalua￾tors assess the results with uniform criteria. In contrast, LLM-as-a-Judge demonstrates inconsistency in its judg￾ments, failing to maintain consistency across languages. limits their flexibility and effectiveness. With t… view at source ↗
Figure 2
Figure 2. Prompt template for using LLM-as-a￾Judge in a Question Answering task. Placeholders <eval_language>, <context>, <question>, <answer> are replaced by the input language, and its correspond￾ing context, question and answer. The text in the prompt is color-coded to represent different sections: for role definition, for evaluation rubric, for output. the consistency of the multilingual judge in real-world applications. … view at source ↗
Figure 3
Figure 3. Fleiss Kappa value gap (∆) between Yes / No and Grade evaluation criteria of various multilingual LLM-as-a-Judge models. 3 illustrates the gap across all datasets. We observe that most gap values are positive, i.e., consistency in Yes / No evaluations is consistently higher than in Grade evaluations. It indicates that grade judg￾ment is more challenging than binary judgment. This result may be due to more options in… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Consistency (Cohen’s Kappa) of LLMs’ judge results between English and other languages across four [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Variation of Fleiss Kappa for Grade judgment (KGrade) across Qwen-2.5 and Aya-Expanse in different model scale. pared to the 72-billion version. Additionally, while the 32-billion Aya-Expanse outperforms its smaller counterparts, its improvement on WMT23 remains limite…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Bias in GPT-5.2 and Gemini 2.5 Flash changes rather than transfers between English and Swahili, with GPT-5.2 refusal behavior appearing only in English.

Reference graph

Works this paper leans on

45 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.258 MEGA : Multilingual evaluation of generative AI . In Proceedings of the 2023 Conference on Empirical Methods in Natural ...

    doi:10.18653/v1/2023.emnlp-main.258 2023

  5. [5]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  8. [8]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

Show all 45 references
  1. [9]

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztaj...

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=CybBmzWBX0 Length-controlled alpacaeval: A simple debiasing of automatic evaluators . In First Conference on Language Modeling

    2024

  4. [12]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr\'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for fi...

    2023 doi

  5. [13]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.365 GPTS core: Evaluate as you desire . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2024 doi

  6. [14]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554

    2023 arXiv

  7. [15]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . arXiv preprint arXiv:2411.15594

    2024 arXiv

  8. [16]

    Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2024. https://aclanthology.org/2024.findings-eacl.71/ Are large language model-based evaluators the solution to scaling up multilingual evaluation? In...

    2024

  9. [17]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond rej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Tos...

    2023 doi

  10. [18]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19/ Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Ta...

    2023

  11. [19]

    Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.360 W iki L ingua: A new benchmark dataset for cross-lingual abstractive summarization . In Findings of the Association for Computational Linguistics: EMNLP 2...

    2020 doi

  12. [20]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  13. [21]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  14. [22]

    Zeming Liu, Ping Nie, Jie Cai, Haifeng Wang, Zheng-Yu Niu, Peng Zhang, Mrinmaya Sachan, and Kaiping Peng. 2023. https://doi.org/10.18653/v1/2023.acl-long.684 XD aily D ialog: A multilingual parallel dialogue corpus . In Proceedings of the 61st Annual Meeting of the Association...

    2023 doi

  15. [23]

    Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. https://doi.org/10.18653/v1/2023.bea-1.32 Automated evaluation of written discourse coherence using GPT -4 . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), p...

    2023 doi

  16. [24]

    OpenAI. 2024. https://openai.com/research/gpt-4o Introducing gpt-4o . OpenAI Blog. Accessed: 2024-06-17

    2024

  17. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  18. [26]

    Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.57 O ffset B ias: Leveraging debiased data for tuning evaluators . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10...

    2024 doi

  19. [27]

    Vyas Raina, Adian Liusie, and Mark Gales. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.427 Is LLM -as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024 doi

  20. [28]

    David Rau, Herv\'e D\'ejean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, St\'ephane Clinchant, and Vassilina Nikoulina. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.449 BERGEN : A benchmarking library for retrieval-augmented generation . In Findings of the Associa...

    2024 doi

  21. [29]

    Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1--39

    2022

  22. [30]

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076

    2023 arXiv

  23. [31]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://openreview.net/forum?id=RIu5lyNXjT Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . In The Twelfth International...

    2024

  24. [32]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  25. [33]

    Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.278 Large language models are not yet human-level evaluators for abstractive summarization . In Findings of the Association for Computational Linguisti...

    2023 doi

  26. [34]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://openreview.net/forum?id=fR3wGCk-IXp Language models are multilingual chain-of-thought reasone...

    2023

  27. [35]

    Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791

    2024

  28. [36]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796

    2024 arXiv

  29. [37]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. https://doi.org/10.18653/v1/2023.newsum-1.1 Is C hat GPT a good NLG evaluator? a preliminary study . In Proceedings of the 4th New Frontiers in Summarizatio...

    2023 doi

  30. [38]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.511 Large language models are not fair evaluators . In Proceedings of the 62nd Annual Meeting o...

    2024 doi

  31. [39]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024 b . Self-taught evaluators. arXiv preprint arXiv:2408.02666

    2024 arXiv

  32. [40]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594

    2024 arXiv

  33. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  34. [42]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736

    2024 arXiv

  35. [43]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://proceedings.neurips.cc/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf Bartscore: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems, volume 34, pages 27263--...

    2021

  36. [44]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Proce...

    2023

  37. [45]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.