Pith. sign in

REVIEW 2 major objections 1 minor 8 references

When in-domain data exists, fine-tuned smaller models match proprietary ones at multilingual judgment while larger models work better zero-shot out of domain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 12:29 UTC pith:G3ZSAMCG

load-bearing objection Useful empirical trade-offs on multilingual LLM judges for Spanish and Basque, but the claims rest on how well the dataset extensions hold up. the 2 major comments →

arxiv 2605.28710 v1 pith:G3ZSAMCG submitted 2026-05-27 cs.CL cs.AI

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Irune Zubiaga , Aitor Soroa , Rodrigo Agerri This is my paper
classification cs.CL cs.AI
keywords multilingual evaluationLLMs-as-a-judgefine-tuningzero-shot evaluationlow-resource languagesBasqueSpanishautomatic text evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests ways to build LLMs that judge generated text in English, Spanish, and Basque. It compares fine-tuning on available data against zero-shot use of bigger models, along with choices about translating instructions and mixing languages during training. The central finding is that the best route depends on whether matching data is on hand. Readers care because reliable automatic evaluation is needed for non-English text generation yet most existing judges are English-only and expensive. The work supplies concrete trade-offs to guide choices between model size and training approach.

Core claim

This paper establishes that fine-tuned smaller models reach performance levels comparable to proprietary models when in-domain data is available for fine-tuning, while zero-shot evaluation using larger models is more effective in out-of-domain settings; it further shows that fine-tuning on out-of-domain data harms results, based on systematic tests across high-, mid-, and low-resource languages using newly extended meta-evaluation datasets.

What carries the argument

The comparison of fine-tuning versus zero-shot strategies together with monolingual versus multilingual supervision and instruction translation, measured on extended meta-evaluation datasets for Basque and Spanish.

Load-bearing premise

The extended meta-evaluation datasets for Basque and Spanish serve as valid proxies for real-world multilingual judgment quality across resource levels.

What would settle it

New human ratings collected on actual generated text in Basque or Spanish that show fine-tuned smaller models no longer match proprietary models on the same tasks.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuning on out-of-domain data reduces model performance for multilingual judgment.
  • Smaller models become practical substitutes for proprietary ones once in-domain data is supplied.
  • Zero-shot use of larger models avoids the need for task-specific data in new domains.
  • The same trade-offs hold across English, Spanish, and Basque, covering high- to low-resource cases.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same data-availability rule may guide choices for other low-resource languages not tested here.
  • Hybrid pipelines that switch between fine-tuned and zero-shot models depending on domain could cut costs further.
  • Public release of the extended datasets and code lowers the barrier for others to replicate or extend the trade-off analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper conducts an empirical study on strategies for multilingual LLMs-as-a-judge across English (high-resource), Spanish (mid-resource), and Basque (low-resource). It extends two existing meta-evaluation datasets to Spanish and Basque, then systematically compares instruction translation, monolingual vs. multilingual supervision, model size, and in-domain vs. out-of-domain settings. Central claims are that fine-tuned smaller models achieve performance comparable to proprietary models when in-domain data is available, zero-shot larger models are more effective out-of-domain, and fine-tuning on out-of-domain data can degrade performance. Data and code are released publicly.

Significance. If the empirical trade-offs hold under scrutiny, the work supplies concrete, actionable guidance for constructing multilingual evaluation pipelines, especially for low-resource languages. It highlights the value of in-domain fine-tuning for efficiency and the risks of mismatched adaptation. The public release of extended datasets and code is a clear strength that enables direct verification and reuse. The study addresses a documented gap in non-English LLM judging.

major comments (2)
  1. [Dataset extension] Dataset extension (Section describing meta-evaluation datasets): The central trade-off claims rest entirely on performance measured on the authors' extensions of two existing meta-evaluation datasets to Basque and Spanish. The manuscript provides no detailed account of the extension process (translation method, post-editing, annotation protocol, or inter-annotator agreement), so it is impossible to assess whether systematic shifts in difficulty, label reliability, or cultural alignment were introduced—particularly for Basque. This directly undermines the validity of the in-domain vs. out-of-domain comparisons.
  2. [Results and analysis] Results and analysis (section reporting in-domain vs. out-of-domain comparisons): The claim that fine-tuned smaller models reach parity with proprietary zero-shot models in in-domain settings, and that out-of-domain fine-tuning harms performance, is presented without statistical significance tests or confidence intervals on the reported metrics. Without these, the observed differences cannot be distinguished from noise, weakening the practical guidance offered.
minor comments (1)
  1. [Introduction] Clarify the exact definition of 'in-domain' and 'out-of-domain' early in the introduction, as the distinction is central to all reported trade-offs.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thorough review and constructive comments on our manuscript. We address each major comment below and indicate the revisions we plan to make.

read point-by-point responses

  1. Referee: [Dataset extension] Dataset extension (Section describing meta-evaluation datasets): The central trade-off claims rest entirely on performance measured on the authors' extensions of two existing meta-evaluation datasets to Basque and Spanish. The manuscript provides no detailed account of the extension process (translation method, post-editing, annotation protocol, or inter-annotator agreement), so it is impossible to assess whether systematic shifts in difficulty, label reliability, or cultural alignment were introduced—particularly for Basque. This directly undermines the validity of the in-domain vs. out-of-domain comparisons.

    Authors: We agree that providing a detailed description of the dataset extension process is crucial for the validity of our findings. In the revised manuscript, we will add a dedicated subsection detailing the translation method (machine translation with human post-editing by native speakers), the annotation protocol, and inter-annotator agreement metrics for both Spanish and Basque extensions. This will help assess any potential shifts in difficulty or cultural aspects. revision: yes

  2. Referee: [Results and analysis] Results and analysis (section reporting in-domain vs. out-of-domain comparisons): The claim that fine-tuned smaller models reach parity with proprietary zero-shot models in in-domain settings, and that out-of-domain fine-tuning harms performance, is presented without statistical significance tests or confidence intervals on the reported metrics. Without these, the observed differences cannot be distinguished from noise, weakening the practical guidance offered.

    Authors: We acknowledge that statistical tests are important to support our claims. We will revise the results section to include bootstrap confidence intervals and appropriate statistical significance tests (such as McNemar's test for paired comparisons) for the key performance differences in the in-domain and out-of-domain settings. This will provide stronger evidence for the observed trade-offs. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical reporting of observed model performance

full rationale

The paper conducts an empirical study by extending two meta-evaluation datasets to Basque and Spanish, then measuring LLM-as-a-judge performance under different fine-tuning and zero-shot conditions across English, Spanish, and Basque. No equations, derivations, or first-principles predictions appear; all claims rest on direct experimental results rather than any fitted parameter being relabeled as a prediction or any self-citation chain substituting for independent justification. The central trade-off findings (fine-tuned small models vs. zero-shot large models) are therefore self-contained observations on the constructed test sets.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical study with no mathematical derivation; no free parameters, axioms, or invented entities are invoked.

pith-pipeline@v0.9.1-grok · 5732 in / 943 out tokens · 25818 ms · 2026-06-29T12:29:08.766413+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for multilingual evaluation, extending LLM-based evaluators to multilingual settings remains challenging, particularly for low-resource languages and scenarios where in-domain data is scarce. This work explores several strategies for developing multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge.

Figures

Figures reproduced from arXiv: 2605.28710 by Aitor Soroa, Irune Zubiaga, Rodrigo Agerri.

Figure 1
Figure 1. Figure 1: Density plot of predictions against gold labels [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Density plot of predictions against gold la [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Density plot showing the distribution of pre [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pearson correlations between model predictions and human judgments across [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean differences between model scores on the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA

    The zeno’s paradox of ‘low-resource’ lan- guages. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA. Associa- tion for Computational Linguistics. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation...

    work page arXiv 2024

  2. [2]

    BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

    COMET-22: Unbabel-IST 2022 submission for the metrics shared task. InProceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems.A...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    11 Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight

    Judgelm: Fine-tuned large language models are scalable judges. 11 Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neu- ral machine translation. InProceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguist...

    2016

  4. [4]

    Maintain the original meaning and intent of the instructions

  5. [5]

    Use standard {target_lang} language

  6. [6]

    Keep the technical terms that don’t have widely accepted {target_lang} translations

  7. [7]

    Preserve any code snippets, variables, or special characters exactly as they appear

  8. [8]

    Please provide accurate {target_lang} translations for all text fields

    Translate only the text content, not the JSON structure The input will be a JSON object with English text. Please provide accurate {target_lang} translations for all text fields. C Justification of the use of MT benchmarks Wilcoxon signed-rank tests (Wilcoxon, 1945), a nonparametric method for comparing two re- lated samples, was used to evaluate whether ...

    1945