Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

This paper argues that the multilingual performance gap on the MGSM math benchmark mostly disappears when translation errors are corrected and answer extraction respects language-specific number formats.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Correcting translation errors and answer extraction in MGSM eliminates most of the apparent cross-lingual math performance gap in strong LLMs.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A useful and mostly convincing demonstration that MGSM's cross-lingual gap is inflated by translation errors and sloppy answer extraction, but the model-guided correction loop means the dataset deserves independent validation before the strong conclusions are taken at face value. the 3 major comments →

arxiv 2511.05162 v2 pith:JR2CQQEY submitted 2025-11-07 cs.CL

Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Jan-Thorsten Peter , David Vilar , Tobias Domhan , Dan Malkin , Markus Freitag This is my paper
classification cs.CL
keywords multilingual evaluationMGSMtranslation errorsanswer extractionmath word problemsbenchmark data qualitycross-lingual gapLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a widely reported gap—large language models doing consistently worse on math word problems in non-English languages—is largely an artifact of flawed test data and evaluation details. On the standard MGSM benchmark, human translations contain subtle errors that change the math (e.g., 'every second glass' becomes 'each additional glass'), and the common regex-based answer extraction mishandles decimal and thousands separators and Bengali numerals. After semi-automatically retranslating and rephrasing problematic questions and using a more robust last-number extractor, the gap mostly disappears; for the strongest models, maximum accuracy differences drop to 1.2–2.0 percentage points. The paper's point is that benchmark conclusions should be checked against data quality rather than taken at face value, and it releases a corrected version of the dataset.

Core claim

Using math as a testbed, the authors first replicate the standard finding: every tested model shows a consistent cross-lingual gap on MGSM, including high-resource languages like French. They then show that the gap is driven by two fixable problems: translation errors in the released test set (e.g., 'not in the girl scout' rendered as 'in the girl scout', 'round to nearest dollar' as 'round up', 'Tuesday' as 'Thursday') and answer extraction that does not account for regional number formats. They propose a semi-automatic QA pipeline: when a majority of strong models fail a translated question, retranslate it, verify fact preservation, and rephrase ambiguities. Combined with a more permissive

What carries the argument

The central mechanism is a two-part correction loop plus an evaluation fix. First, a semi-automatic quality-assurance loop: questions that a majority of strong models fail get retranslated, verified for fact preservation, and rephrased if ambiguous; this addresses data-level errors. Second, a relaxed answer extractor that takes the last number in the response and strips commas, dots, and spaces while normalizing Bengali numerals; this addresses evaluation-level errors. The MGSM benchmark itself is the central object: because the same 250 questions exist in 11 languages, language gaps can be measured; the paper uses that symmetry to show that the gap is an artifact.

Load-bearing premise

The load-bearing premise is that when strong models fail a translated question, the translation is at fault, and that the retranslations used as ground truth are genuine fixes rather than easier rewrites.

What would settle it

A concrete test: take the corrected questions and run a blind human evaluation that swaps original and corrected translations for matched questions. If human judges rate the corrected versions as no more faithful to the English source, or if models trained only on the original English questions show no larger gap on corrected versus original translations, the central claim weakens. Independently retranslating the same questions through a different pipeline and checking whether the cross-lingual gap reappears would also settle it.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Prior MGSM-based comparisons across languages should be re-examined; the corrected dataset gives a different ranking and much smaller gaps.
  • For benchmarks where models are already near saturation, small remaining errors dominate the metric, so data cleaning and standardized answer extraction matter more as accuracy approaches 100%.
  • Answer extraction must be language- and script-aware; ignoring decimal/thousands separators or non-Arabic numerals can shift scores by double digits.
  • The corrected dataset provides a cleaner testbed for future multilingual reasoning evaluation.
  • A meaningful remaining gap persists for smaller models and low-resource languages, so the conclusion is not that all multilingual math gaps vanish.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The cleaning method has a circularity risk: the data is corrected toward what strong models can solve, so the same models' improved scores on corrected data are partly self-confirming. A fair test would hold out a portion of corrections and check with models not used in the loop.
  • If the paper's diagnosis generalizes, other multilingual benchmarks that are near saturation may have hidden translation and parsing artifacts; re-auditing them could change reported language rankings.
  • The relaxed last-number extractor is not universal; for tasks with non-numeric answers or multiple numbers, it could silently pick the wrong value. A testable extension is to build a standardized multilingual answer-extraction protocol.
  • The paper's conclusion is deliberately narrow—the gap disappears for strong models on a saturated benchmark—so it does not rule out real cross-lingual deficits on harder or less contaminated tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies cross-lingual math performance of LLMs on the MGSM benchmark across eleven languages. Initial experiments (Table 2) show a consistent performance gap relative to English, including high-resource languages like French. The authors then inspect the test data and find translation errors, ambiguous source questions, and answer-extraction problems (e.g., decimal/thousands separators, Bengali numerals). They propose a semi-automatic, LLM-guided correction loop that retranslates and reformulates questions, and an improved answer-extraction regex. Combined, these changes produce Table 4, where the cross-lingual gap mostly disappears for strong models. The corrected dataset is released as MGSM-Rev2.

Significance. If established, the central claim is important: it would show that prior cross-lingual math gaps on MGSM were largely artifacts of test-data quality and evaluation details rather than genuine capability differences. The paper gives concrete, convincing examples of real translation errors (Table 3) and demonstrates clear effects of answer-extraction choices (e.g., French number formatting and Bengali numerals). Releasing the corrected dataset and specifying the evaluation fixes are valuable contributions. However, the headline conclusion—that the gap 'mostly disappears'—is weakened by a methodological circularity: corrections are selected based on model failures and verified by the same model family, so the revised data is conditioned on model solvability. The paper itself acknowledges that the procedure 'guarantees solvability' (§5.1). The evidence therefore supports the existence of data and evaluation issues, but not the stronger claim that the original translations were the primary cause of the measured gap.

major comments (3)
  1. [5.1] The correction procedure conditions the revised test set on model solvability. The retranslation loop is triggered by 'a majority of strong models failed to answer a translated question', the retranslations are verified by Gemini, and the text states 'While this guarantees solvability...'. Consequently, Tables 4 and 5 measure whether items can be rephrased to be solvable by this model class, not whether the original translations were independently wrong. Without human-validated ground truth or a held-out set annotated before seeing model failures, the central claim that the gap is an artifact of translation/evaluation is not established.
  2. [Appendix B / Figure 1] Many modifications are not translation corrections but ambiguity resolutions in the English source that change the expected answer (e.g., B.4 minutes vs seconds, B.11 whether John is among the 20 friends, B.8/B.9 'times more'). Figure 1 shows 'clarity & precision' is the largest modification category. The paper conflates source-question ambiguity with translation errors. The analysis should disaggregate these categories and show that the gap reduction is driven by genuine mistranslations rather than re-annotations of ambiguous items.
  3. [Table 5] The delta rows combine the data correction and regex change, but the data-correction component is not independent. Since the same model family (Gemini/Claude/GPT) drove the corrections, evaluating these models on the resulting set is circular. Please provide a per-sample analysis of which corrections were genuine translation errors (with human judgement) and report accuracy restricted to items untouched by the correction loop, or on an independent human-verified set.
minor comments (5)
  1. [Section 1] Typo: 'language languages' appears in the second paragraph; 'grad school math problems' should be 'grade school math problems'.
  2. [5.1] Typo: 'newlt translated questions' should be 'newly translated questions'.
  3. [Appendix B] The text refers to 'Gemini 3.5 Pro' in the opening paragraph, but the rest of the paper uses 'Gemini 2.5 Pro'. Please make consistent.
  4. [5.3.1] Typo: 'it's numerals' should be 'its numerals'.
  5. [7] The sentence 'there wasno changeto the setup' is missing spaces; also clarify that the system prompt and decoding setup were unchanged, but data and answer extraction necessarily changed.

Circularity Check

2 steps flagged

Correction loop retranslates until strong LLMs can solve items, then evaluates those LLMs on the revised data — the gap reduction is partly by construction.

specific steps
  1. self definitional [Section 5.1 (A Closer Look — Wrongly Translated Questions)]
    "Whenever a majority of strong models failed to answer a translated question, we retranslated it using Gemini 2.5 Pro. We verified each translation through using Gemini ensuring that all relevant facts were preserved. We then prompted the LLMs with the newlt translated questions to check if it was now solvable. If this verification failed, we repeated the process or we reformulated the original question if we considered the question to be ambiguous (Section 5.2). While this guarantees solvability, it does not account for instances where the model can infer the correct answer from context."

    The paper's 'translation errors' are operationally defined as items that a majority of strong models fail. Those items are then retranslated/reformulated and iteratively re-checked by LLMs until they are solvable ('While this guarantees solvability'). Evaluating the same classes of strong LLMs (including Gemini) on the corrected set therefore measures whether items can be rewritten until those models solve them, not whether the original translations were wrong. The disappearance of the language gap in Table 4 is forced for the flagged items by construction, so it is not an independent confirmation of the data-corruption hypothesis.

  2. self definitional [Section 5.2 (Wrong and Ambiguous Questions)]
    "The majority of ambiguous questions were identified through an analysis of frequent post-translation validation failures. In collaboration with Gemini, these questions were manually rephrased to improve clarity while maintaining their original complexity."

    Ambiguity is detected by failures of models from the same families later evaluated; the fix is performed 'in collaboration with Gemini', one of the evaluated systems. Consequently the corrected questions are not independent of the models whose gap is being measured.

full rationale

The central correction procedure (Section 5.1) is not an independent data audit: it uses exactly the quantity that the later evaluation re-measures. 'Whenever a majority of strong models failed... we retranslated... If this verification failed, we repeated... While this guarantees solvability.' The revised set is therefore constructed so that the strongest models can solve the flagged items; Table 4 then reports that those models' language gap mostly disappears. This portion of the claimed reduction is an artifact of the selection loop. The paper is not wholly circular: Table 3 and Appendix B give concrete, human-verifiable translation errors and genuine ambiguities, and the improved number extraction (Section 5.3, Table 5) is an independent evaluation improvement that does not depend on model solvability. Also, no load-bearing self-citation or uniqueness-import is used. But because the aggregate headline result — the 'gap mostly disappears' — is partly manufactured by fitting the benchmark to the same models that are evaluated, the circularity score is 6 rather than 0-2.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new free parameters or entities. Its load-bearing assumptions are about the reliability of LLM-guided data correction and the adequacy of the answer-extraction heuristic. The most significant circularity risk is that model failures drive data corrections, which are then verified using the same class of models.

axioms (4)
  • domain assumption A majority of strong models failing a translated question indicates the translation is erroneous rather than the models being weak in that language.
    Section 5.1: 'Whenever a majority of strong models failed to answer a translated question, we retranslated it using Gemini 2.5 Pro.' This is the selection trigger for corrections and is load-bearing for the gap-reduction result.
  • domain assumption Gemini 2.5 Pro retranslations, verified by Gemini, preserve the intended math and are valid ground truth.
    Section 5.1: 'We verified each translation through using Gemini ensuring that all relevant facts were preserved.' This assumes the correcting model is reliable enough to fix the data.
  • domain assumption If a model can solve a retranslated question, the question is 'solvable' and therefore correct.
    Section 5.1: 'We then prompted the LLMs with the newly translated questions to check if it was now solvable.' This conflates solvability by LLMs with correctness.
  • domain assumption The last-number extraction heuristic and the rule to drop trailing .00/,00 recover the intended numeric answer in all 11 languages.
    Section 5.3 and Figure 3: the heuristic is stated as working well in practice, but no full validation is given across all languages and edge cases.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results." pith.science (2026). https://pith.science/paper/JR2CQQEY

@misc{pith2026251105162,
  author       = {Pith},
  title        = {Pith review of: Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JR2CQQEY}},
  note         = {Machine review of arXiv:2511.05162}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Most current large language models (LLMs) support a wide variety of languages in addition to English, including high-resource languages (e.g. German, Chinese, French), as well as low-resource ones (e.g. Swahili, Telugu). In addition, they have shown impressive capabilities in different domains, like coding, science and math. In this paper, taking math as an example domain, we study the performance of different LLMs across languages. Experimental results show that there exists a non-negligible and consistent gap in the performance of the models across languages. Interestingly, and somewhat against expectations, the gap exists for both high- and low-resource languages. These results should impact further research into cross-lingual capability generalization for next generation LLMs. Or they would, if it weren't for the fact that they are distorted by data quality issues. By analyzing one of the standard multilingual math benchmarks (MGSM), we determine that several translation errors are present in the data. Furthermore, the lack of standardized answer extraction from LLM outputs further influences the final results. We propose a method for semi-automatic quality assurance to address the first issue at scale, and give recommendations to address the second one. Combining these two approaches we show that the aforementioned language gap mostly disappears, leading to completely different conclusions from our research. We additionally release the corrected dataset to the community (https://github.com/google-research-datasets/MGSM-Rev2).

Figures

Figures reproduced from arXiv: 2511.05162 by Dan Malkin, David Vilar, Jan-Thorsten Peter, Markus Freitag, Tobias Domhan.

Figure 1
Figure 1. Figure 1: Number of modified problems and categorization of changes. The total number of ques [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Function to extract answer by Kaggle 1 # The answer prefix is taken from this array 2 # The remove of zeros after the decimal points is originally done 3 # in the scoring function 4 # https://www.kaggle.com/code/kerneler/mgsm-benchmark-starter-code 5 6 def parse_kaggle_answer(prediction: str, answer_prefix: str) -> str: 7 if answer_prefix not in prediction: return "" 8 answer_text = prediction.split(answer… view at source ↗
Figure 3
Figure 3. Figure 3: Function to last number in response 1 def extract_last_number(text, lang): 2 # Searching the last sequence of digits, commas, dots, 3 # or spaces followed by digits and then non-numeric 4 # characters or the end of the string. 5 match = re2.search(r’([\d., ]+)(?:\D*$|$)’, text) 6 if match: 7 # Remove commas and spaces before returning the number 8 prediction = match.group(1).replace(’ ’, ’’) 9 if predictio… view at source ↗
Figure 4
Figure 4. Figure 4: The prompt used to classify modifications to MGSM questions. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval

    cs.CL 2026-06 unverdicted novelty 6.0

    ParaEval reduces false performance gaps in MCQA benchmarks from over 2 points to below 1 point by scoring models on multiple paraphrases per answer option instead of single surface forms.

  2. The Effect of Scripts and Formats on LLM Numeracy

    cs.CL 2026-01 conditional novelty 6.0

    LLM arithmetic accuracy falls sharply when numerals leave the familiar Hindu–Arabic format, and few-shot prompting with examples narrows most of that gap.

Reference graph

Works this paper leans on

6 extracted references · cited by 2 Pith papers

  1. [1]

    ‘‘of the total’’)

    **Clarity & Precision: ** Resolving ambiguous phrasing, specifying exact definitions of terms or relationships, clarifying temporal context, and detailing the scope of percentages or quantities (e.g., ‘‘of the remainder’’ vs. ‘‘of the total’’)

  2. [2]

    $20’’ to

    **Correction of Numerical/Factual Errors: ** Rectifying incorrect values, multipliers, initial conditions, or the specific item being asked about (e.g., changing "$20’’ to "$0.20’’ cashback, ‘‘3 times more’’ to ‘‘twice the amount’’, asking for ‘‘not in’’ a group instead of ‘‘in’’ it)

  3. [3]

    **Conciseness & Directness: ** Rephrasing the problem or the question to be more straightforward, removing unnecessary words or complex sentence structures

  4. [4]

    **Unit Specification: ** Explicitly stating the required unit for the answer (e.g., ‘‘how many minutes’’ instead of ‘‘how long’’, ‘‘how many dozen’’ instead of ‘‘how many eggs’’) or clarifying units within the problem (e.g., ‘‘miles’’ vs ‘‘kilometers’’)

  5. [5]

    **Grammatical/Syntactic Improvement: ** Fixing typos, awkward phrasing, or grammatical errors that improve readability without necessarily changing the core mathematical logic

  6. [6]

    **No meaningful change ** Question v1: {v1} Question v2: {v2} Answer: {answer} 16

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.