Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper establishes that translating a model's non-English answer back to English and running English-only evaluators measures cross-lingual consistency cheaply, and that by this measure popular LLMs are markedly less consistent for…

desk verdict Useful framework and a broad, honest empirical sweep; the information-consistency numbers mostly hold up, but the empathy-consistency scores rest on an unvalidated translation-fidelity assumption and the paper needs error bars before the ranking claims are taken at face value. read the letter →

arxiv 2505.21999 v1 pith:2TNBA5VS submitted 2025-05-28 cs.CL

classification cs.CL
keywords cross-lingualconsistencymultilingualevaluationtranslatethenevaluateinformationempathylargelanguagemodelsFActScoreEPITOME
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that the consistency of a large language model across languages can be measured without expensive multilingual annotations: translate the model's answer in the other language back to English, then run standard English evaluators on the pair. Using that recipe on twelve models and thirty languages, the authors find that consistency is far from guaranteed: answers to the same question in Indic and Dravidian languages carry substantially less of the information and empathy of the English answer, with open-weight models lagging proprietary ones. The finding matters because users increasingly rely on these models in healthcare, legal, and therapeutic settings, where getting a shorter or less empathetic answer because of the language you asked in is a real harm. The paper positions the framework as a cheap, scalable barometer for multilingual capability.

What carries the argument

The load-bearing object is the consistency score $C_{M,E}(\ell) = \frac{1}{|X|}\sum_{x\in X} E(r_{\text{en}}(x), \hat r_\ell(x))$, the average compatibility between the English answer and the NLLB-translated answer to the same prompt. What carries the argument is the translation model $T_{\ell\to\text{en}}$ and the choice of English evaluator $E$: NLLB (54B MoE) does the language bridging, and the two instantiations are FActScore's atomic-claim precision and recall (harmonically averaged) for information, and EPITOME-derived binary classifiers over emotional reaction, interpretation, and exploration (exact profile match) for empathy. The framework is evaluator-agnostic; swapping $E$ would measure other properties such as toxicity or sentiment.

What would settle it

Back-translate a set of English empathetic responses into the thirty languages and back into English, then run the same empathy classifiers; if the self-consistency scores fall materially below 1.0, the empathy consistency numbers in Table 4 are confounded by translation loss and do not measure the models' own consistency.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurement strategy rather than a new model: cross-lingual consistency can be quantified by a two-step pipeline, Translate then Evaluate. For a question asked in English and in another language, the model's non-English response is machine-translated back into English with NLLB, and an English-only evaluator scores how well that translation matches the model's English response. The paper instantiates the pipeline twice. Information consistency uses FActScore, which decomposes a response into atomic claims; the authors compute both precision (facts in the translated response supported by the English response) and a swapped recall variant, and report their harmonic mean. Empathy consistency uses three binary classifiers trained on the EPITOME framework for emotional reaction, interpretation, and exploration, and requires the three-bit empathy profile of the translated response to exactly match the English one. Applied to twelve models and thirty languages, the measure shows that every model is at best partially consistent: gpt-4o reaches 0.68 information and 0.82 empathy consistency, Llama-3.1-8B scores 0.32 and 0.45, and response consistency drops sharply for Indic scripts and Dravidian language families, with scores as low as 0.06 in the information dimension.

Load-bearing premise

That the NLLB translation preserves exactly the facts and empathy cues the English evaluators are looking for; the paper validates this for factual information on ten languages but explicitly leaves empathy translation fidelity to future work.

Editorial extensions

If this is right

  • Multilingual benchmarks can now include dozens of languages at the cost of translation only: no human annotations or per-language task formulation are required.
  • Manufacturers should report cross-lingual consistency alongside task accuracy; the numbers here show that model cards claiming multilingual support do not guarantee equal service.
  • A user asking in Hindi, Tamil, or Malayalam can receive materially less complete and less empathetic answers than the same user asking in English, which is a fairness and safety concern in clinical and legal settings.
  • Larger models are usually, but not always, more consistent, so scaling alone does not solve the problem; consistency needs to be benchmarked directly.
  • A model can be consistent on information and inconsistent on empathy (and vice versa), so single-dimension evaluations miss important failure modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: part of the measured deficit may live in the translation step rather than the model; rerunning the pipeline with human translations or a stronger translator for low-resource languages would separate translation loss from model inconsistency.
  • Editorial extension: because English is the reference, the score measures agreement with English answers, not correctness; a model could be consistently wrong in both languages and still score well, so consistency should complement, not replace, factual evaluation.
  • Editorial extension: the same scaffold could be turned into a regularizer or training signal: penalizing cross-lingual inconsistency at decoding time might directly improve the reported numbers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a 'Translate then Evaluate' framework for measuring cross-lingual consistency of LLM responses without parallel annotated data: non-English outputs are translated to English and scored against the English response with an English-only evaluator. The framework is instantiated with an information consistency metric based on FActScore and an empathy consistency metric based on three EPITOME classifiers, and applied to 12 models across 30 languages. The headline results are low information and empathy consistency in many languages, particularly Indic and Dravidian, with proprietary models generally more consistent than open-weight models. The authors provide a human fidelity study for the information dimension, release code and data, and report detailed experimental setup.

Significance. If the measurements were valid, the paper would make a useful and low-cost contribution to multilingual LLM evaluation: the framework is evaluator-agnostic, avoids expensive annotation, and covers a broad language set. The transparency in reporting the information-fidelity human study and the code/data release are strengths. However, the empathy-consistency results rest on an explicitly unvalidated assumption, so the paper's stronger claim of measuring cross-lingual empathy consistency is not yet supported; the information-consistency findings are better supported but still rest on partial validation. I agree with the stress-test concern that the empathy scores in Table 4 are not established as measurements of model empathy consistency.

major comments (3)
  1. [§5.1, Limitations] Section 5.1 explicitly leaves an empathy translation-fidelity study to future work, and the Limitations section repeats that no such study was run, yet Table 4's empathy consistency scores are presented as measurements of model empathy consistency. The framework's validity depends on the §2 desideratum that the translation model preserves the pragmatic properties the evaluator measures, and the EPITOME classifiers were trained on original English Reddit text rather than machine-translated input. Consequently, the low empathy scores for Dravidian and other language groups may reflect translation-induced changes in indirect emotional reactions, interpretations, or exploratory questions rather than model inconsistency. Please add a fidelity check that compares EPITOME profiles of original non-English responses with profiles of their English translations, or human annotations of preserved empathy mechanisms, or explicitly restrict the empathy claims to 'translation-mediated consistency' rather than model empathy consistency.
  2. [§4.1, Table 3, Appendix E.3] Even for information consistency, the validation of the approximation E(ren, r_l) ≈ E(ren, T(r_l)) is indirect. The human fidelity study in Table 3 rates content preservation of short snippets of 1-4 sentences for ten languages, not whether the FActScore evaluator produces the same scores on original versus translated responses; the automatic check in Table 1 uses back-translated English responses, not the imperfect, lower-resource outputs that the framework is designed to score. The paper should provide a direct check of evaluator score agreement and either extend the human study to more languages and full responses or state the extrapolation more cautiously.
  3. [Appendix A] Appendix A reports that Gemini API recitation errors were handled by substituting the default string 'No answer,' but it does not report the frequency of such events or a sensitivity analysis. If this substitution occurred non-negligibly, it would lower the Gemini consistency scores in Table 4 and could affect the comparison between proprietary and open-weight models. Please quantify the substitution rate and show that the conclusions are unchanged when those cases are excluded.
minor comments (6)
  1. [§4.1] The choice of Qwen-2-14B as the FActScore claim extraction and verification model is justified only by 'we find performs both steps effectively,' with no quantitative comparison; please report agreement with the original GPT-4-based implementation on a sample.
  2. [Table 7] The header and caption of Table 7 are confusing: the columns for F1 and Accuracy are not clearly separated, and the claim that GPT-4o significantly underperforms the bi-encoder is not straightforwardly supported by the values as printed.
  3. [Appendix E.2] The heading of Appendix E.2 says 'Language Grouping by Scripts' but the table lists language families; this heading should be corrected.
  4. [§5.2] The phrase 'significantly more consistent' is used without significance tests or confidence intervals; given 150 prompts and 100 utterances, please add interval estimates or bootstrap confidence intervals, at least for the headline model comparisons.
  5. [§3.1] The paper does not state how the FActScore harmonic mean handles a response with no extracted claims, which is relevant for abstentions such as the 'No answer' substitutions mentioned in Appendix A; please specify the convention.
  6. [Appendix E.4] There are several typographical errors in the text (e.g., 'mdoels', 'comnparison', and 'sed' for 'used' in Appendix E.4) that should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Translate then Evaluate consistency metric is an explicit definition, the evaluators and translation model are external artifacts, and the admitted empathy-fidelity gap is a validity threat, not a circular reduction.

full rationale

The paper's central quantity, CM,E(ℓ), is defined in Eq. (1) as the empirical mean of an evaluator score between the English response and the NLLB-translated non-English response. This is an explicit measurement definition, not a derived prediction: no fitted parameter is later renamed as a consistency result, and no equation reduces the reported scores to the framework's inputs by construction. The translation model (NLLB) and the two evaluators (Qwen-based FActScore and the EPITOME-based empathy classifiers) are external artifacts; none is trained on or tuned to the paper's cross-lingual consistency scores. The self-citations in the paper (Li et al. 2019 and Mehta and Srikumar 2023) appear only in related work and in the limitations discussion, and neither is load-bearing for the main derivation. The human fidelity study for information consistency (Table 3) is an independent sanity check of the translation assumption, and the automatic back-translation check (Table 1) is also external evidence, not circular validation. The paper explicitly acknowledges that a comparable translation-fidelity study for empathy consistency is not performed (Section 5.1: “such evaluations require domain expertise in therapeutic communication… we leave it to future work”; repeated in the Limitations). This is a serious validity caveat for the empathy results, but it is not circularity: it is an unvalidated assumption about translation preserving empathy mechanisms, not a case where the reported consistency score is equivalent to an input by definition or by fitted constants. Therefore, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unproved assumptions: that prompts have equivalent translations, that translation preserves the measured properties (only partially validated for information, not for empathy), and that English is an appropriate reference. There are no free parameters fitted to produce the consistency scores, because the evaluators are pre-trained external components.

assumptions (3)
  • domain assumption All prompts in X have valid and semantically equivalent translations in each language ℓ in L.
    Stated in Section 2 as a formal assumption of the framework; if translations are not semantically equivalent, the consistency comparison is invalid.
  • domain assumption The translation model preserves the properties that the evaluator measures, so that E produces similar scores before and after translation.
    Stated in Section 2 as an assumption. For information consistency, a human fidelity study covers only ten languages; for empathy consistency, the paper explicitly leaves this validation to future work in Section 5.1.
  • domain assumption The English response is an appropriate reference for information completeness.
    The consistency definition uses the English response as the reference for both precision and recall, which biases low-resource languages toward lower scores; the authors acknowledge this in Section 5.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate." pith.science (2026). https://pith.science/paper/2TNBA5VS

@misc{pith2026250521999,
  author       = {Pith},
  title        = {Pith review of: Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TNBA5VS}},
  note         = {Machine review of arXiv:2505.21999}
}
read the original abstract

Large language models (LLMs) provide detailed and impressive responses to queries in English. However, are they really consistent at responding to the same query in other languages? The popular way of evaluating for multilingual performance of LLMs requires expensive-to-collect annotated datasets. Further, evaluating for tasks like open-ended generation, where multiple correct answers may exist, is nontrivial. Instead, we propose to evaluate the predictability of model response across different languages. In this work, we propose a framework to evaluate LLM's cross-lingual consistency based on a simple Translate then Evaluate strategy. We instantiate this evaluation framework along two dimensions of consistency: information and empathy. Our results reveal pronounced inconsistencies in popular LLM responses across thirty languages, with severe performance deficits in certain language families and scripts, underscoring critical weaknesses in their multilingual capabilities. These findings necessitate cross-lingual evaluations that are consistent along multiple dimensions. We invite practitioners to use our framework for future multilingual LLM benchmarking.

Figures

Figures reproduced from arXiv: 2505.21999 by the authors.

Figure 1
Figure 1. Llama-3 8B ( ) response to an English (en) prompt versus its response to the translation of the same prompt in a different language like Hindi (hi) . We observe a clear gap in information generated between the two languages. The responses in Hindi has been translated for the reader. The English response is truncated for brevity. late LLM responses to non-English queries into English using a high-fidelity translation… view at source ↗
Figure 2
Figure 2. Annotation screen used in the user study. The annotator is asked to rate the translation quality on a Likert [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Cross-lingual information consistency scores. The languages are grouped roughly by what they script they [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Cross-lingual empathy consistency scores. The languages are grouped roughly by what they script they [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Zoom In Disparities in Healthcare LLM Q&A

    cs.CL 2025-10 conditional novelty 5.0 of 10

    Health Q&A chatbots align their answers with English Wikipedia even for non-English prompts, and injecting a non-English Wikipedia excerpt at query time shifts answers toward local references.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Therefore, we translate sentence-by-sentence instead of all generated text as one input

    As mention in the NLLB paper (Costa-Jussà et al., 2022), we observe that translation qual- ity with NLLB mdoels significantly degrades with increase in input length. Therefore, we translate sentence-by-sentence instead of all generated text as one input. We find that for splitting sentences properly across languages, we need to use language specific sente...

  2. [2]

    We use the largest MoE model instead of the smaller dense 3.3B model because anecdo- tally, it produced fewer degenerate transla- tions (although the reported numbers are com- parable – only 1.1% drop in chrf++)

  3. [3]

    We also experimented with using sampling instead of greedy and found that it had little to no affect on degener- ation

    We set the translation model to generate greedy generations as the original code base uses greedy decoding7. We also experimented with using sampling instead of greedy and found that it had little to no affect on degener- ation

  4. [4]

    What were the roles of women in political leadership across pre-colonial African societies?

    We also handle degeneration wherever we can detect. Given a translation, we see if there are any repeated phrases or repeated patterns. If they do, we generate a translation by sampling for that particular example with temperature as 0.7, and top_p as 0.95. C Experimental Details for Measuring Discrepancy in Empathy This section deals with the experimenta...

  5. [5]

    The **Dahomey Amazons** (Fon people of present-day Benin) were an all-female military regiment under the king’s command, demonstrating institutionalized female power in warfare

    **Military Leadership:** * In some societies, women served as military commanders or warriors. The **Dahomey Amazons** (Fon people of present-day Benin) were an all-female military regiment under the king’s command, demonstrating institutionalized female power in warfare

  6. [6]

    **Village and Clan Leadership:** * At the local level, women frequently served as heads of lineages, clan leaders, or representatives in councils, particularly in societies with decentralized politi- cal structures like the Igbo and Yoruba. In sum, pre-colonial African societies often recognized women’s authority in both formal and informal governance str...

  7. [9]

    For example: * **Nzinga Mbande** of Ndongo and Matamba (Angola) was a warrior queen who led resistance against Portuguese colonization in the 17th century

    **Queens and Queen Mothers (Matriarchs):** * In several African societies, women held formal authority as sovereigns or co-rulers. For example: * **Nzinga Mbande** of Ndongo and Matamba (Angola) was a warrior queen who led resistance against Portuguese colonization in the 17th century. * The **Queen Mothers of the Asante** (modern-day Ghana), such as Yaa ...

  8. [10]

    In many cases, Queen Mothers (e.g., in the Oyo and Benin Kingdoms) had substantial influence in court affairs and succession decisions

    **Advisors and Regents:** * Women often acted as advisors to kings or served as regents during the minority of a male heir. In many cases, Queen Mothers (e.g., in the Oyo and Benin Kingdoms) had substantial influence in court affairs and succession decisions

Show all 22 references
  1. [11]

    **Leaders in Matrilineal Societies:** * In matrilineal systems, such as among the Akan and Bemba, lineage and inheritance passed through the mother’s line, giving women and maternal uncles significant political influence in succession and property decisions

  2. [12]

    For example, female priestesses and diviners (e.g., the Igbo *dibia*) mediated between the spiritual and political realms

    **Spiritual and Religious Leaders:** * Women often held high-ranking spiritual positions that conferred political influence. For example, female priestesses and diviners (e.g., the Igbo *dibia*) mediated between the spiritual and political realms

  3. [15]

    **Pemimpin politik dan militer** – Beberapa wanita memegang kekuasaan tertinggi, seperti *Yaa Asantewaa* dari Kekaisaran Asante (Ghana) yang memimpin perang melawan kolonial Inggris, atau *Amina dari Zazzau* (Nige- ria) yang dikenal sebagai ratu pejuang dan penakluk

  4. [16]

    **Ratu ibu (Queen Mothers)** – Di banyak kerajaan seperti Ashanti dan Buganda, ratu ibu memiliki otoritas signifikan, termasuk pengaruh dalam pe- nunjukan raja, nasihat politik, dan pengelolaan wilayah atau istana kerajaan

  5. [17]

    **Kepala klan atau suku** – Dalam masyarakat matrilineal seperti suku Akan, wanita tua dalam klan (biasanya nenek atau ibu kepala) sering menjadi penentu garis keturunan, warisan, dan keputusan penting komunitas

  6. [18]

    Peran ini menunjukkan bahwa kepemimpinan perempuan bukanlah hal yang asing atau sekunder, melainkan integral dalam struktur sosial dan politik pra- kolonial Afrika

    **Mediator dan penasihat** – Wanita sering memainkan peran sebagai penasihat raja atau dewan, serta mediator dalam konflik antar-kelompok atau keluarga. Peran ini menunjukkan bahwa kepemimpinan perempuan bukanlah hal yang asing atau sekunder, melainkan integral dalam struktur ...

  7. [19]

    **Political and military leaders** – Some women held supreme power, such as *Yaa Asantewaa* of the Asante Empire (Ghana) who led the war against British colonial rule, or *Amina of Zazzau* (Nigeria) who was known as a warrior queen and conqueror

  8. [20]

    **Queen Mothers** – In many kingdoms such as Ashanti and Buganda, queen mothers wielded significant authority, including influence over king ap- pointments, political advice, and the management of the royal court or territory

  9. [21]

    **Clan or tribal chiefs** – In matrilineal societies such as the Akan, the elder women in the clan (usually the grandmother or matriarch) often determined lineages, inheritance, and important community decisions

  10. [22]

    What were the roles of women in political leadership across pre-colonial African societies?

    **Mediators and advisors** – Women often played roles as advisors to the king or council, as well as mediators in conflicts between groups or families. This role shows that female leadership was not an extraneous or secondary thing, but rather integral to the social and politi...

  11. [2018]

    Annotation Artifacts in Natural Language In- ference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Assoc...

  12. [2019]

    arXiv preprint arXiv:1906.09833

    Translationese in machine translation eval- uation. arXiv preprint arXiv:1906.09833. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 h...

  13. [2020]

    No answer

    Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer, and Diyi Yang. 2025. Helping the helper: Supporting peer counselors via ai-empowered practice an...

  14. [2022]

    Computational Linguistics, 48(3):673–732

    Survey of Low-Resource Machine Translation. Computational Linguistics, 48(3):673–732. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.