Pith.

Computation and Language · arXiv:2611.14861 v1 · submitted 11 Nov 2026 · 31 references

Is Jev a Good Multilingual Judge?

Wei Liu, Yi Fan, Xiyan Fu

The paper says

“Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention.”

more of the abstract

We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Competent practical evaluation of Jev across languages: the central claim holds, novelty is incremental, and the paper deserves a serious referee. This line is wrong

1 major objection · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 31 references say.

The paper says, at §4.2, Tables 3-5

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

The paper describes Jev as 'competitive on most multilingual judging tasks' (Abstract, §1, §8). However, on MEMERAG (Table 3) Jev's BAcc (69.0) is 4-5 points below both Qwen and GPT-6.1, and on Global-MMLU-Lite it is 6.5 points below GPT-6.1. The term 'competitive' is not operationally defined (e.g., within a specified margin or statistical test).

more of objection 1

Given the cost advantage, the claim may be justified, but the paper would benefit from an explicit statement of what constitutes 'competitive' for each task, perhaps with respect to task difficulty or baseline variability. Without this, a reader could interpret the phrase as overclaiming.

6 minor objections

The paper says, at §3.2, Table 2

Pith says: minor objection 1 of 6

The cost comparison uses API prices for Jev and GPT-6.1 and an estimated API-equivalent cost for Qwen. The paper should note whether the Jev price is as of the date of the experiments and that pricing may change, as this directly affects the cost ratios.

The paper says, at §4.2, Tables 3-5

Pith says: minor objection 2 of 6

The results are reported without confidence intervals or significance tests. Given that comparisons are based on fixed benchmark samples, reporting bootstrap intervals or McNemar tests for binary tasks would strengthen the claims of differences.

show 4 more

The paper says, at §5, Figure 2

Pith says: minor objection 3 of 6

The decomposition of cross-lingual judgment changes is informative. The paper should clarify whether the error categories (false acceptances, false rejections, etc.) are computed from Jev's binary decisions (threshold 0.5) or from its probability outputs. This is implied but not explicitly stated.

The paper says, at §7.2, Table 8

Pith says: minor objection 4 of 6

Table 8 reports accuracy and coverage at fixed confidence thresholds. For reproducibility, the paper should specify the exact set of thresholds used and whether confidence is defined as max(p,1-p) for binary tasks (as in Appendix G.2). The current text is clear but could cross-reference the appendix more directly.

The paper says, at Appendix F

Pith says: minor objection 5 of 6

The cost calculation for Qwen uses Alibaba API pricing, but the paper does not state the date of query or whether it reflects batch vs. on-demand pricing. Adding a note would improve replicability.

The paper says, at General

Pith says: minor objection 6 of 6

There are a few minor typographical errors (e.g., 'judgment' vs. 'judgement' is inconsistent with the style; the use of 'evaluation cost' in the abstract is split across lines in the preprint version). These do not affect comprehension but should be corrected in the final version.

What would settle it

Recompute the accuracy and cost figures for Global-MMLU-Lite and MENLO using the same API versions and prompts but on a held-out subset of samples not used in the paper (e.g., 10% random sample from each benchmark) to verify that the relative ordering and cost ratios replicate within statistical error. This would confirm that the reported results are not artifacts of a particular sample draw or API transient behavior.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 26 s · 19,203 tokens read, 3,228 written
  3. Editor wrote the verdict above 13 s · 19,969 tokens read, 1,581 written
  4. Referee minor revision: 1 major objection, 6 minor 21 s · 19,519 tokens read, 2,661 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper gives a clear, honest answer to a practical question: how well does Jev work as a multilingual judge compared to expensive generative models? The answer is “pretty well, with caveats,” and the evidence supports it.

What’s new: this is the first systematic multilingual evaluation of Jev. Prior work focused on English. The paper covers 47 languages, three judgment formats, includes cross-lingual consistency analysis and confidence-based error detection, and provides per-language results that let you see where the model works and where it doesn’t. The experimental design is thorough, the cost comparison is transparent ($2.64 total for Jev vs. $97.45 for GPT-6.1), and the authors are upfront about the limitations: Jev falls behind on knowledge-intensive answer verification and degrades in low-resource languages, mostly through new false acceptances. The practical recommendations (use English instructions, raise confidence thresholds when accuracy matters, route low-confidence samples to a stronger judge) are well-supported.

Soft spots: the novelty is incremental. The paper applies existing evaluation methodology to a new model and language set—no new method or analytical tool. It relies entirely on benchmark annotations without independent quality checks; this is standard practice but limits depth. The confidence-based routing gains are modest (0.4–3.2 percentage points), which the authors acknowledge. The cost figures depend on current API pricing and may shift, but the relative ratios are what matter. Also, readers unfamiliar with Jev’s architecture will need to go elsewhere for background, which is fine for a short empirical report but worth noting.

The paper is for practitioners who need a cost-effective multilingual judge and want guidance on when Jev works and when it does not. Researchers will appreciate the per-language breakdowns. It is not a theoretical breakthrough, but it is a well-executed, honest evaluation of an important practical question.

Recommendation: accept for peer review. The paper is solid, the claims are supported, and the limitations are transparent. Minor revision could add a short note on annotation quality or a sensitivity check on cost assumptions. No fatal flaws.

The strongest objection

After careful review, I find no load-bearing concern that undermines the central claim that Jev achieves competitive multilingual judging performance at substantially lower cost than GPT-6.1. The paper conducts a systematic evaluation across four benchmarks, three judgment formats, and up to 47 language varieties. Cost comparisons are clearly stated with recorded token usage and API prices. Performance differences are reported in full per-language detail, and the authors explicitly note where Jev falls behind (knowledge-intensive answer verification) and where gains are modest (confidence routing on most tasks). The reader flagged annotation quality as a weakest assumption, but this is a standard limitation of benchmark evaluation that the paper acknowledges implicitly by using established human-annotated datasets with prior validation. The analysis of cross-lingual inconsistency includes a decomposition into new errors vs. corrections of English errors, addressing the natural concern that disagreement with English might reflect improved judgments. The paper provides reproducible prompt templates and inference settings. I see no internal inconsistency, hidden assumption that would flip the conclusions, or omitted validation that is clearly required. The claim holds as stated.

The reader's verdict is ACCEPT with high confidence. My stress-test finds no significant objection. The paper's methodology is sound: it compares Jev against two generative judges on four established multilingual benchmarks, with clear metrics and cost accounting. Limitations are explicitly discussed, including variability across tasks, lower cross-lingual consistency in low-resource languages, and limited gains from confidence routing. The central claim that Jev is competitive on most multilingual judging tasks at 1/31 to 1/61 of GPT-6.1's cost is directly supported by Tables 2–5 and the per-language breakdowns. The reader's weakest assumption (annotation quality) is a general benchmark limitation, not a paper-specific flaw; the paper does not rely on any novel annotations. The analysis of cross-lingual inconsistency includes a decomposition that addresses potential confounds. Thus I concur with the ACCEPT verdict.

The referee's report

If the results hold, the paper provides strong empirical evidence that a decision model can serve as a cost-effective multilingual judge for many evaluation tasks, potentially reducing the computational expense of automated evaluation. The thoroughness across 47 language varieties and multiple judgment formats, combined with explicit cost accounting and practical recommendations, makes this a valuable resource for practitioners. The analysis of cross-lingual inconsistency and confidence-based error detection is methodologically sound and offers actionable insights.

Pith's reading of the paper (a machine's summary, not the authors')

The paper asks whether Jev, a decision model that directly outputs judgment probabilities, can serve as a multilingual judge instead of generative LLMs like GPT-6.1. Across four benchmarks covering up to 47 language varieties and three evaluation formats, Jev achieves competitive performance on most tasks at roughly 1/31 to 1/61 of GPT-6.1's cost. However, it falls behind on knowledge-intensive answer verification, particularly in low-resource languages where it tends to accept incorrect answers that it correctly rejects in English. The analysis also shows that Jev's probability outputs help distinguish correct from incorrect judgments, especially for binary tasks, offering a practical confidence signal for filtering or routing decisions.

What carries the argument. The central object is Jev, a decision model that outputs probability distributions over predefined judgment categories (e.g., Yes/No, rating levels) rather than generating free-form text. This probability output enables confidence-based error detection and makes inference cheap, as it avoids autoregressive token generation. The paper compares Jev against generative models using the same prompts and benchmarks, leveraging its structured API and probability outputs for downstream analysis.

Core claim. Jev, a decision model, is competitive with generative LLM judges (Qwen, GPT-6.1) on most multilingual judging tasks at approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, but trails GPT-6.1 on knowledge-intensive answer verification. Across binary decisions, pairwise comparisons, and graded ratings in up to 47 languages, Jev achieves the highest balanced accuracy on METAL hallucination detection (85.9%) and the highest macro-F1 on MENLO ratings (33.3%), while its cross-lingual consistency (93.4% agreement with English) exceeds Qwen but is lower than GPT-6.1, especially in low-resource languages. The primary driver of inconsistency in low-resource languages is false acceptance of incorrect answers that Jev correctly rejects in English.

The premise it rests on. The human annotations used as ground truth across all four benchmarks are assumed to be correct and representative, but the paper does not audit annotation quality, inter-annotator agreement, or cultural representativeness beyond the benchmarks' original validation, so any noise or bias in gold labels directly propagates into all reported accuracy metrics.

If the paper is right (Pith's inferences, not the authors')

If Jev is used as a multilingual judge, organizations can perform large-scale evaluation across many languages at a fraction of the cost of generative LLMs, making automated quality assessment accessible for lower-budget settings.

On tasks where Jev competes with GPT-6.1 (e.g., pairwise preference, hallucination detection), it can serve as a drop-in replacement, freeing resources for more complex evaluations.

Confidence-based filtering can improve accuracy on binary verification tasks (e.g., Global-MMLU-Lite accuracy rises from 89.57% to 99.18% when retaining only samples with confidence >= 0.95), though coverage drops substantially.

Routing low-confidence Jev judgments to a stronger model like GPT-6.1 yields modest gains on most tasks, suggesting limited room for improvement when errors are shared across models.

The finding that Jev accepts incorrect answers more often in low-resource languages implies that multilingual robustness requires attention to language-specific biases, even in models that are otherwise consistent.

The paper's results suggest a practical cascade: use Jev for low-cost initial judgments and escalate only low-confidence samples to a generative model. This could preserve cost savings while mitigating the knowledge-intensive weakness, but the limited gains reported indicate that the models share similar failure patterns, reducing the benefit.

The analysis of cross-lingual judgment changes reveals that Jev's errors in low-resource languages are predominantly false acceptances, hinting that the model's calibration or prior probabilities may be skewed toward acceptance in unfamiliar language contexts. Testing this hypothesis with synthetic data could inform targeted fine-tuning.

The confidence signal is weaker for graded ratings (AUROC 0.550–0.599) than for binary tasks, suggesting that probability-based error detection may require reformulation (e.g., using distributional distance metrics) for multi-class judgments.

The paper's recommendation to use English instructions by default, combined with the observed cross-lingual inconsistency, implies that the model's understanding of evaluation criteria is language-independent but its application to content varies; this could be leveraged to design language-agnostic evaluation templates.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately 1/31 to 1/61 of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.97

“At approximately $1/31$ to $1/61$ of GPT-6.1's evaluation cost, the classifier is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.93 · human participants 0.42 · controlled comparison 0.85 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.96 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. gap 0.86 0.05 0.03 asserted 0.99
We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. method 0.73 0.96 0.05 asserted 1.00
At approximately $1/31$ to $1/61$ of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. result 1.00 0.94 0.55 asserted 1.00
Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. result 0.96 0.95 0.16 asserted 0.60
Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge. future work 0.72 0.72 0.33 asserted 0.92

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 4,387 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Agreement with English judgments across 22
Figure 2 of the paper
Figure 2. Changes in judgments from English to other languages on Global-MMLU-Lite. False acceptances and
Figure 3 of the paper
Figure 3. A Global-MMLU-Lite example of judgment changes from English to Welsh. Jev introduces a false
Figure 4 of the paper
Figure 4. How well Jev’s confidence distinguishes cor
Figure 5 of the paper
Figure 5. Prompts for Global-MMLU-Lite answer verification (top) and MEMERAG sentence-level faithfulness
Figure 6 of the paper
Figure 6. MENLO prompts for pairwise comparison and response rating, shown for cultural relevance.
Figure 7 of the paper
Figure 7. METAL summary-rating prompts for linguistic acceptability and output content quality.
Figure 8 of the paper
Figure 8. METAL prompts for task-quality rating and hallucination detection.

Figures are the authors', reproduced from the paper.

References (31)
  1. [1] Jev Documentation , year = on Pith
  2. [2] Cruz Bland. Proceedings of the 63rd Annual Meeting of the. 2025 , pages = on Pith
  3. [3] Singh, Shivalika and Romanou, Angelika and Fourrier, Cl. Global. Proceedings of the 63rd Annual Meeting of the. 2025 , pages = on Pith
  4. [4] Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations , year = on Pith
  5. [5] Zhang, Xinyu and Thakur, Nandan and Ogundepo, Odunayo and Kamalloo, Ehsan and Alfonso-Hermelo, David and Li, Xiaoguang and Liu, Qun and Rezagholizadeh, Mehdi and Lin, Jimmy , journal =. Making a on Pith
  6. [6] Fu, Xiyan and Liu, Wei , booktitle =. How Reliable is Multilingual. 2025 , pages = on Pith
  7. [7] Li, Yubo and Miao, Yidi and Krishnan, Ramayya and Padman, Rema , journal = on Pith
  8. [8] Rao, Delip and Callison-Burch, Chris , journal =. Decision Models vs on Pith
  9. [9] Deu. Evaluating and Benchmarking the System One Model. arXiv preprint arXiv:2609.37647 , year = on PithPith's reviewarXiv
  10. [10] Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It , author =. arXiv preprint arXiv:2609.26758 , year = on PithPith's reviewarXiv
  11. [11] Li, Xing and Chang, Qingcheng and Ning, Jinzhong and Xu, Changfeng and Zhang, Shenlong and Zhang, Yijia and Luo, Ling and Lin, Hongfei , journal =. Specialized Decision Models vs. General-Purpose on Pith
  12. [12] Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging. 2023 , url = on Pith
  13. [13] Zhou, Ej and Resck, Lucas and Hui, Zheng and Korhonen, Anna , booktitle =. Lower-Resource, Higher Scores: Language Bias in. 2026 , url = on Pith
  14. [14] Zhang, Hongbin and Chen, Kehai and Bai, Xuefen and Pan, Youcheng and Xiang, Yang and Wang, Jinpeng and Zhang, Min , journal =. Mitigating Translationese Bias in Multilingual on Pith
  15. [15] Son, Guijin and Yoon, Dongkeun and Suk, Juyoung and Aula-Blasco, Javier and Aslan, Mano and Kim, Vu Trong and Islam, Shayekh Bin and Prats-Cristi. arXiv preprint arXiv:2410.17578 , year = on PithPith's reviewarXiv
  16. [16] Do. Challenges and Recommendations for. arXiv preprint arXiv:2607.02235 , year = on PithPith's reviewarXiv
  17. [17] Gureja, Srishti and Miranda, Lester James V. and Islam, Shayekh Bin and Maheshwary, Rishabh and Sharma, Drishti and Winata, Gusti and Lambert, Nathan and Ruder, Sebastian and Hooker, Sara and Fadaee, Marzieh , booktitle =. 2025 , pages = on Pith
  18. [18] Doddapaneni, Sumanth and Khan, Mohammed Safi Ur Rahman and Venkatesh, Dilip and Dabre, Raj and Kunchukuttan, Anoop and Khapra, Mitesh M. , booktitle =. Cross-Lingual Auto Evaluation for Assessing Multilingual. 2025 , pages = on Pith
  19. [19] Pombal, Jos. Conference on Language Modeling , year = on Pith
  20. [20] Whitehouse, Chenxi and Ruder, Sebastian and Lin, Tony and Kurylo, Oksana and Takagi, Haruka and Lam, Janice and Busetto, Nicol. International Conference on Learning Representations , year = on Pith
show 11 more
  1. [21] Hada, Rishav and Gumma, Varun and Ahmed, Mohamed and Bali, Kalika and Sitaram, Sunayana , editor =. Findings of the. 2024 , address =. doi:10.18653/v1/2024.findings-naacl.148 , pages = on Pithdoi:10.18653/v1/2024.findings-naacl.148
  2. [22] Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Mubasshir, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat , editor =. Findings of the. 2021 , address =. doi:10.18653/v1/2021.findings-acl.413 , pages = on Pithdoi:10.18653/v1/2021.findings-acl.413
  3. [23] Large Language Models are not Fair Evaluators , author =. Proceedings of the 62nd Annual Meeting of the. 2024 , pages = on Pith
  4. [24] Chen, Lingjiao and Zaharia, Matei and Zou, James , journal =. 2024 , url = on Pith
  5. [25] Jung, Jaehun and Brahman, Faeze and Choi, Yejin , booktitle =. Trust or Escalate:. 2025 , url = on Pith
  6. [26] On the Calibration of Massively Multilingual Language Models , author =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year = on Pith
  7. [27] On Calibration of Modern Neural Networks , author =. Proceedings of the 34th International Conference on Machine Learning , year = on Pith
  8. [28] Dror, Rotem and Baumer, Gili and Shlomov, Segev and Reichart, Roi , editor =. The Hitchhiker. Proceedings of the 56th Annual Meeting of the. 2018 , address =. doi:10.18653/v1/P18-1128 , pages = on Pithdoi:10.18653/v1/p18-1128
  9. [29] Measuring Nominal Scale Agreement among Many Raters , author =. Psychological Bulletin , volume = on Pith
  10. [30] The Combination of Estimates from Different Experiments , author =. Biometrics , volume = on Pith
  11. [31] A Practitioner's Guide to Cluster-Robust Inference , author =. Journal of Human Resources , volume = on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment