Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

Jieyu Zhao; Meng Jiang; Xiangliang Zhang; Xiuying Chen; Yanbo Wang; Yue Huang; Zixiang Xu

REVIEW 4 major objections 6 minor 13 references

Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An automated beam-search pipeline generates bilingual question pairs that make multilingual LLMs drop from near-perfect English accuracy to below 50% in target languages.

desk verdict A useful dataset for probing cross-lingual gaps, but the headline drop is partly a selection artifact and the semantic check needs more scrutiny before the numbers are trusted. read the letter →

arxiv 2505.18673 v1 pith:KFVOTV3C submitted 2025-05-24 cs.CL

Zixiang Xu , Yanbo Wang , Yue Huang , Xiuying Chen , Jieyu Zhao , Meng Jiang , Xiangliang Zhang This is my paper

classification cs.CL

keywords cross-lingualevaluationmultilinguallargelanguagemodelsbeamsearchLLM-basedsimulationlinguisticsimilarityquestionperturbationfine-tuningtransferbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multilingual large language models can be probed for cross-lingual weakness automatically: a beam search over perturbations, guided by an LLM-based simulation score, generates bilingual question pairs where the model gives the right answer in English but the wrong one in the target language. The method produced a dataset of 6,713 pairs across 16 languages and, across ten models, consistently revealed average accuracy drops over 50% in target languages while English accuracy stayed near 100%. If the claim holds, the contribution is a cost-effective diagnostic that pinpoints where multilingual models are fragile, a reusable evaluation benchmark, and evidence that linguistically related languages share similar weaknesses and improve together under target-language fine-tuning.

What carries the argument

The load-bearing object is the beam-search pipeline driven by the simulation score $V(q^{E'}, q^{T'}) = (\bar{\beta}^{E'})^{\gamma} - \bar{\beta}^{T'}$, where $\bar{\beta}^{E'}$ and $\bar{\beta}^{T'}$ are the average accuracies of a panel of LLMs on the perturbed English and target-language questions and the exponent $\gamma > 1$ rewards pairs that keep English accuracy high. The perturbation function $\varphi$ takes an English question and one of the incorrect answer choices and generates a semantically irrelevant but contextually plausible insertion, which is then translated into the target language; a semantic-similarity constraint with thresholds $\theta$ and $\theta'$ is meant to guarantee meaning preservation. The search uses an inclusion threshold $\theta_{\text{inc}}$, an early-stopping potential threshold $\theta_{\text{pot}}$, and a redundancy control. A second named object, the Relative Affinity Score $D_{x,y}$, measures how much weakness observed in language $y$ transfers to language $x$, with lower values indicating stronger affinity; this is what carries the linguistic-similarity findings.

What would settle it

Take the question pairs with the largest English-to-target-language accuracy drops, have several independent fluent speakers (blind to the models' outputs) re-check each pair for semantic equivalence and answer consistency, and recompute the average accuracy gap on only the pairs that pass; if the recomputed gap shrinks well below the reported 50%, the headline weakness is largely a translation artifact.

Watch

Extended reading notes

Core claim

The central claim is that cross-lingual weaknesses in LLMs are discoverable by optimizing for the accuracy gap between English and a target language rather than for absolute accuracy. The method starts with English multiple-choice questions from existing benchmarks, translates them, and then applies a beam search: at each step a proxy LLM adds a contextually plausible but semantically neutral distraction linked to an incorrect answer option, the distraction is translated into the target language, and a panel of LLM 'simulators' scores each pair by $V(q^{E'}, q^{T'}) = (\bar{\beta}^{E'})^{\gamma} - \bar{\beta}^{T'}$ with $\gamma > 1$. Only pairs that satisfy a semantic-similarity check and exceed an inclusion threshold enter the candidate list. Evaluated on ten models, the resulting pairs leave English accuracy near perfect while target-language accuracy falls by over 50% on average, with GPT-4o and Claude-3.5-Sonnet still dropping more than 30% on Chinese. The paper further claims that weaknesses transfer along linguistic family lines, quantified by a Relative Affinity Score, and that fine-tuning on Chinese or French preferentially improves related languages such as Japanese/Korean or German/Spanish.

Load-bearing premise

The measurements are only valid if every translated and perturbed question means the same thing in both languages, so that a wrong answer in the target language is a real model weakness rather than a translation artifact.

Editorial extensions

If this is right

Running the same pipeline on a different model's or dataset's English questions produces a language-specific probe set without manual prompting.
The released 6,713-pair, 16-language dataset can serve as a diagnostic benchmark: models scoring near 100% in English and far lower in a target language reveal exactly where multilingual alignment fails.
Fine-tuning on one language transfers most to linguistically similar languages, so targeted post-training should prioritize those pairs.
Because the simulation panel can be changed, the method can identify cross-lingual weaknesses of any given LLM by including that model in the simulator set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same search-and-simulate machinery could be applied to languages outside QA, such as probing cross-lingual instruction following or safety refusal behavior, since the perturbation generator only needs a question and an answer set.
The reported cost asymmetry — weaknesses cost more to find in French, Spanish, Italian, and German than in lower-resource languages — could mean these models genuinely transfer better to languages close to English, or that the perturbation search space is less expressive there; a follow-up could distinguish the two.
Because human evaluation measured semantic equivalence at 83–98% per language, a small fraction of pairs may carry semantic drift; if drift concentrates in the largest-gap pairs, the dataset as a benchmark should be filtered by stricter human verification.
The behavioral Relative Affinity Score could be used to build empirically derived language-family trees of model competence, complementing typological measures and possibly revealing where model representations diverge from human linguistic classifications.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The genuinely new thing is the pipeline: beam search over LLM-generated perturbations, guided by a simulation score, to produce bilingual question pairs that separate English and target-language accuracy. That is a reasonable way to mine for cross-lingual weaknesses, and the resulting dataset—6,713 pairs across 16 languages, with code and evaluation scripts—is a real artifact the community can use. The conversion-rate comparison in Table 1 (NP vs DP vs search) is a fair piece of evidence that the search itself is doing work, not just the translation.

The strongest part of the paper is that held-out models also show large drops. When the authors replace two simulation models with GPT-4o in the loop, GPT-4o's Chinese accuracy falls sharply, and models not in the simulation set (Llama-3.1-70B, o1-mini, Yi-Lightning, Claude-3.5-Sonnet) generally show substantial gaps. That tells me the effect is not entirely manufactured by ranking on the same models.

The soft spots are real, though. First, the aggregate "over 50% drop" claim in Figure 3 mixes the five simulation models into the evaluation. For those models, high drops are partly guaranteed because candidates were selected to maximize the gap for exactly those models. The held-out numbers are the honest ones, and they are weaker than the headline. Second, the semantic-equivalence thresholds θ and θ′ in Eq. (4) are never reported. The human evaluation samples 100 random pairs per language. That is the wrong conditioning: the search selects for high English accuracy and low target accuracy, so any translation or perturbation that subtly changes meaning or flips the answer is exactly the kind of pair that lands in the high-drop subset. With Amharic at 83% semantic equivalence and 88% answer consistency on a random sample, the residual risk is not negligible. The authors do acknowledge this in the Limitations, but they don't test whether the bad pairs concentrate where the gaps are largest. Third, "inherent" is too strong a word for weaknesses defined relative to current model families and this generation pipeline.

My bottom line: a serious referee should get this. The method is useful, the dataset is reusable, and the main issues are disclosure and framing rather than a demonstrated failure. The authors should separate simulation and evaluation models in the headline numbers, report θ/θ′, and add a semantic-drift analysis conditioned on the actual selected pairs. If those checks survive, the dataset will be a standard reference. I'd bring it to reading group; I'd cite the dataset with a caveat.

Referee Report

4 major / 6 minor

Summary. The paper proposes an automatic pipeline for generating bilingual multiple-choice question pairs that expose cross-lingual weaknesses in LLMs. The pipeline translates English questions, iteratively perturbs them via beam search, and uses a set of five LLMs to score each pair by the simulated English-versus-target-language accuracy gap. This yields a 16-language dataset of 6,713 pairs. The authors evaluate ten models on this dataset, report large average accuracy drops in target languages, analyze how weaknesses cluster by linguistic family using a proposed Relative Affinity Score and embedding visualizations, and show that fine-tuning on one language improves performance more on linguistically related languages.

Significance. If the headline results are trustworthy, the paper contributes a low-cost diagnostic method and a reusable multilingual evaluation dataset, both of which are genuinely useful for studying cross-lingual robustness. The paper also attempts to connect observed weaknesses to linguistic similarity and demonstrates a possible targeted fine-tuning application. The authors are transparent about some limitations, and the release of code and data is a plus. However, the central quantitative claim ('over 50% accuracy drops') is currently supported by a procedure in which the models that guide the search also appear in the main evaluation, and the semantic-equivalence safeguard is not reported or validated on the items that matter most. These issues must be resolved before the main claims can be accepted.

major comments (4)

[Section 2.3 (Eq. 8) and Section 3.2] The simulation model set M is used to rank candidate pairs via the score V(qE', qT') = (β̄_E')^γ − β̄_T', and the same five models (Llama-3.1-8B, Gemma-2-9B, Gemma-2-27B, GPT-4o-mini, Qwen2.5-72B) then appear in the final accuracy evaluations in Figure 5 and Figures 11–25. For these models, a large portion of the reported drop is guaranteed by the selection criterion rather than being an independent discovery. The aggregate 'over 50% drop' figure therefore conflates in-search and held-out models. Please report results separately for the simulation models and for held-out models, recompute the aggregate without the in-search models, and state per-model average drops for each group.
[Section 2.2 (Eq. 4) and Appendix C] The semantic-equivalence constraints S(qE, qE') ≥ θ and S(qE', qT') ≥ θ′ are central to the claim that target-language failures are model weaknesses rather than translation artifacts, but θ and θ′ are never specified anywhere, including Appendix B.1. In addition, the human evaluation in Appendix C samples 100 random pairs per language, which is not the right conditioning: the search objective preferentially retains pairs with large English-target gaps, so even a small overall semantic-failure rate could be concentrated in exactly the high-drop subset that drives the headline. Please report the threshold values and the semantic-check prompt, and perform a targeted human evaluation on the top-drop pairs per language (e.g., the top decile by simulated gap) to quantify semantic-equivalence and answer-consistency rates on the pairs that actually determine the reported numbers.
[Abstract and Section 3.2 (Figure 5)] The abstract claims that the method 'consistently reveal[s] over 50% accuracy drops in target languages across a wide range of models,' but the per-model results shown for Chinese indicate that held-out top models such as GPT-4o and Claude-3.5-Sonnet exhibit roughly 30% drops, with the larger drops concentrated among the five simulation models. The aggregate claim is misleading unless it is explicitly restricted to the simulation models or recomputed on a held-out-only basis. Please restate the claim to match the evidence, or provide new held-out results that support the original wording.
[Section 3.3 (RAS, Figure 7)] The Relative Affinity Score Dx,y is defined directly from model accuracy on expanded pairs and is then used to conclude that 'languages with closer linguistic ties tend to share cross-lingual weaknesses.' Because the same accuracy data define both the affinity measure and the outcome it is supposed to explain, the reasoning is at risk of circularity. Please validate RAS against an independent linguistic distance measure (e.g., phylogenetic distance, Levenshtein distance, or a published language-family taxonomy), or run a permutation test that shuffles language labels to show that the observed clustering is stronger than chance.

minor comments (6)

[Appendix C] The human evaluation does not report inter-annotator agreement, and the four annotators are described as 'proficient in English and various translation tools' rather than native or near-native speakers of all 16 target languages; this is especially relevant for Amharic, Yoruba, Zulu, and other lower-resource languages.
[Section 2.2 (Eq. 2)] The wording that φ 'preserv[es] its original semantics and embedding patterns influenced by the incorrect answer' is confusing, since the perturbation is explicitly generated from the incorrect answer; please clarify whether the perturbation is semantically neutral, misleading, or merely contextually plausible.
[Table 1] The No Perturbation (NP) baseline reports a conversion rate of 0.000 for Chinese, yet Figure 1 shows a direct English-Chinese pair (apparently without perturbation) that causes GPT-4o to fail; please clarify the exact definition of 'conversion rate' and state which models are used for the baseline comparison.
[Appendix B.1] The hyperparameter list omits the semantic-similarity thresholds θ and θ′ from Eq. (4), and the semantic-check prompt is not included in Appendix E; these should be documented for reproducibility.
[Section 3.4 (Tables 2 and 5)] The phrase 'simulated DPO' is used without explanation; please describe how the DPO preference pairs were constructed and whether this is a standard DPO setup.
[Section 3.3 (RAS definition)] The text uses both 'C' and 'c' for the same constant, and Appendix B.1 states 'C was set to -1' while the formula and prose use lowercase c; please unify the notation.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported >50% drops for the five simulation models are enforced by the inclusion threshold; held-out models provide partial independent support.

fitted input called prediction [Section 2.3 Eq. (8); Section 2.4 'Inclusion Threshold Strategy'; Appendix B.1 hyperparameters]
"V(q^{E'}, q^{T'}) = (β̄^{E'})^γ − β̄^{T'} ... A bilingual pair is immediately included in the candidate list if its simulation score exceeds a predefined inclusion threshold θinc ... θinc was set to 0.8 ... the models used for simulation typically exhibit greater accuracy degradation."

The candidate list is defined by thresholding the same simulation score V that is later reported as the cross-lingual accuracy drop. With γ=2 and θinc=0.8, every accepted pair satisfies (β̄^{E'})^2 − β̄^{T'} ≥ 0.8; since β̄^{E'} ≤ 1, this forces β̄^{T'} ≤ 0.2 for the average of the five simulation models (Llama-3.1-8B, Gemma-2-9B, Gemma-2-27B, GPT-4o-mini, Qwen2.5-72B). Thus a ≥80% target-language failure for these models is guaranteed by the construction of the dataset, not discovered by evaluation. Including the same five models in the 10-model evaluations of Figures 3 and 5 means part of the headline 'over 50% accuracy drops across a wide range of models' is the optimization objective restated as an empirical finding.

full rationale

The paper's core pipeline is an optimization: beam search drives V = (β̄^{E'})^γ − β̄^{T'} (Eq. 8) upward, and a bilingual pair is admitted to the candidate list once V ≥ θinc, with θinc set to 0.8 (Appendix B.1). For the five simulation models, this inclusion rule mathematically forces average target-language accuracy to ≤0.2, i.e., an accuracy drop of at least 80% relative to English, independent of any genuine cross-lingual weakness. Reporting those five models in the aggregate 'over 50% accuracy drop' figures is therefore the selection criterion evaluated on the same models that defined it. This is a genuine partial circularity. It is mitigated by the paper's explicit acknowledgment that 'the models used for simulation typically exhibit greater accuracy degradation,' and more importantly by the held-out models: GPT-4o, Claude-3.5-sonnet, o1-mini, Yi-Lightning, and Llama-3.1-70B were not in the initial simulation set, yet they also show large drops, so the strongest SOTA-model claim has independent content. The missing threshold values θ and θ′ and the random human-evaluation sampling are threats to validity, but they are not circularity under the stated criteria, so I do not count them here. I found no load-bearing self-citation or imported uniqueness theorem; the citation to Huang et al. (2025b) is only motivational. Overall the central method is not equivalent to its inputs—the held-out results break the loop—but a substantial portion of the reported aggregate gap is forced by construction, warranting a partial-circularity score of 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its load-bearing assumptions are semantic preservation across translation and perturbation, the representativeness of the simulation model set, and the use of a performance-derived RAS as a linguistic-proximity measure. The free parameters are search and scoring hyperparameters that shape which pairs are labeled weaknesses, plus the unreported semantic thresholds in Eq. (4).

free parameters (6)

Simulation score exponent gamma = 2
Equation (8) uses (beta^E')^gamma - beta^T' to weight English accuracy; gamma is set to 2 in Appendix B.1 and is not justified by sensitivity analysis.
Inclusion and potential thresholds theta_inc, theta_pot = 0.8 and 0.6
Appendix B.1 sets theta_inc=0.8 for adding a pair to the candidate list and theta_pot=0.6 for expanding search depth; these directly control which weaknesses are reported.
Beam search width and depths = w=12, d1=4, d2=6, r=3, W=4
Appendix B.1 reports these hyperparameters. They define the size and diversity of the search, and no ablation study is provided.
RAS constant c = -1
The Relative Affinity Score D_x,y in Section 3.3 uses exp(c * |A_y - A_x|) with c=-1. The constant controls the sensitivity of the linguistic-affinity claim and appears to be chosen by hand.
Semantic similarity thresholds theta and theta' = Not reported
Equation (4) imposes S(qE, qE') >= theta and S(qE', qT') >= theta', but the paper never states values or the semantic similarity function S. These are load-bearing free parameters for the claim that translations preserve meaning.
Simulation model set M = Llama-3.1-8B, Gemma-2-9B, Gemma-2-27B, GPT-4o-mini, Qwen2.5-72B
The choice of which five models guide the search determines which weaknesses are found. Appendix B.1 selects these as 'cost-effective' but does not analyze how the results change with a different simulation set.

assumptions (5)

domain assumption Cross-lingual weakness is defined as correct in English but incorrect in at least one other language, with English treated as the model's strongest language.
Section 1 defines the target quantity this way and explicitly excludes questions that are wrong in English too. If English is not the strongest language for a particular model, the definition would misclassify knowledge gaps as cross-lingual weaknesses.
domain assumption LLM-based translation plus LLM semantic checks preserve meaning and answer consistency across languages.
Section 2.2 and the Discussion rely on this assumption for all 16 languages. Human evaluation in Appendix C supports it for 100 sampled pairs per language, but not perfectly, and the semantic thresholds in Eq. (4) are unspecified.
domain assumption The five-model simulation set M is a reliable proxy for the behavior of other LLMs.
Section 2.3 scores candidates with M and later uses held-out models to validate. If M is idiosyncratic, the search will preferentially find weaknesses that are specific to those models rather than general cross-lingual weaknesses.
domain assumption Inserting 2-3 context sentences around a question increases cognitive difficulty without changing the correct answer.
Section 2.1 and the perturbation prompt in Appendix E assume the inserted text does not alter the question's meaning. The case studies show many examples where this holds, but no mechanism guarantees it for every pair.
domain assumption The Relative Affinity Score D_x,y, computed from model accuracies, is a valid measure of linguistic proximity.
Section 3.3 defines D_x,y from model accuracies and then interprets low values as evidence of linguistic and cultural proximity. This risks being partly circular, since the measure is derived from the very model behavior it is used to explain.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models." pith.science (2026). https://pith.science/paper/KFVOTV3C

@misc{pith2026250518673,
  author       = {Pith},
  title        = {Pith review of: Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFVOTV3C}},
  note         = {Machine review of arXiv:2505.18673}
}

read the original abstract

Large Language Models (LLMs) have achieved remarkable success in Natural Language Processing (NLP), yet their cross-lingual performance consistency remains a significant challenge. This paper introduces a novel methodology for efficiently identifying inherent cross-lingual weaknesses in LLMs. Our approach leverages beam search and LLM-based simulation to generate bilingual question pairs that expose performance discrepancies between English and target languages. We construct a new dataset of over 6,000 bilingual pairs across 16 languages using this methodology, demonstrating its effectiveness in revealing weaknesses even in state-of-the-art models. The extensive experiments demonstrate that our method precisely and cost-effectively pinpoints cross-lingual weaknesses, consistently revealing over 50\% accuracy drops in target languages across a wide range of models. Moreover, further experiments investigate the relationship between linguistic similarity and cross-lingual weaknesses, revealing that linguistically related languages share similar performance patterns and benefit from targeted post-training. Code is available at https://github.com/xzx34/Cross-Lingual-Pitfalls.

Figures

Figures reproduced from arXiv: 2505.18673 by the authors.

**Figure 1.** An example of an English-Chinese question pair discovered by our search methodology (where the Chinese question is semantically equivalent to the English) highlights the cross-lingual performance gap: even GPT-4o, despite its strong multilingual capabilities, provides the correct answer in English but gives an incorrect response in Chinese. 2025d), improving their effectiveness in addressing real-world problems w… view at source ↗

**Figure 2.** The overview of the proposed methodology for generating questions that precisely challenge the cross [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Evaluation of 10 models on our generated 6,600 bilingual pairs across 16 languages. While all models [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (23 more)

**Figure 4.** Figure 4: Analysis of question conversion rates and generation costs across 16 languages based on all pairs in our [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Performance of LLMs on our generated English-Chinese pairs. Even smaller models like Gemma-2-9B and Llama-3.1-8B achieve perfect accuracy in English, while more than half of the models score below 50% in Chinese. Despite their strong multilingual capabilities, GPT-4o…

**Figure 6.** Figure 6: Accuracy of GPT-4o-mini on expanded bilingual pairs (Asian and European language families). The [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Visualization of RAS Dx,y across 16 languages, highlighting linguistic and cultural proximities. The vertical axis denotes the source language and the horizontal axis denotes the target language. Darker shades of a block indicate stronger retention of shared cross-lin…

**Figure 8.** Figure 8: Heatmap of the pairwise cosine distances be [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: T-SNE visualization of the embeddings gen [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Performance of LLMs on English-Chinese pairs after incorporating GPT-4o into the LLM-Based simulation. The Chinese accuracy of GPT-4o dropped significantly [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

**Figure 14.** Figure 14: Performance of LLMs on English-French pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

**Figure 15.** Figure 15: Performance of LLMs on English-German pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]

**Figure 16.** Figure 16: Performance of LLMs on English-Hebrew pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

**Figure 21.** Figure 21: Performance of LLMs on English-Spanish pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]

**Figure 22.** Figure 22: Performance of LLMs on English-Swahili pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]

**Figure 23.** Figure 23: Performance of LLMs on English-Ukrainian [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]

**Figure 24.** Figure 24: Performance of LLMs on English-Yoruba pairs in our candidate list [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]

**Figure 25.** Figure 25: Performance of LLMs on English-Zulu pairs [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]

**Figure 26.** Figure 26: Case study: Gemma-2-9B’s responses to English-Korean question pairs. [PITH_FULL_IMAGE:figures/full_fig_p022_26.png]

**Figure 27.** Figure 27: Case study: GPT-4o-mini’s responses to English-French question pairs. [PITH_FULL_IMAGE:figures/full_fig_p023_27.png]

**Figure 28.** Figure 28: Case study: Llama-3.1-70B’s responses to English-German question pairs. [PITH_FULL_IMAGE:figures/full_fig_p024_28.png]

**Figure 29.** Figure 29: Case study: GPT-4o’s responses to English-Chinese question pairs. [PITH_FULL_IMAGE:figures/full_fig_p024_29.png]

**Figure 30.** Figure 30: Case study: Gemma-2-27B’s responses to English-Italian question pairs. [PITH_FULL_IMAGE:figures/full_fig_p025_30.png]

**Figure 31.** Figure 31: Case study: Llama-3.1-8B’s responses to English-Spanish question pairs. [PITH_FULL_IMAGE:figures/full_fig_p025_31.png]

**Figure 32.** Figure 32: Case study: Qwen2.5-72B’s responses to English-Japanese question pairs. [PITH_FULL_IMAGE:figures/full_fig_p026_32.png]

**Figure 33.** Figure 33: Case study: o1-mini’s responses to English-Ukrainian question pairs. [PITH_FULL_IMAGE:figures/full_fig_p026_33.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[1]

text": "<Texte traduit en français>

The distraction should naturally integrate with the context of the question but must not explicitly introduce incorrect information or contradict the correct answer. 2. The distraction must be subtle and should not make it obvious that it is related to the incorrect option. [The Start of the Question] {question} [The End of the Question] [The Start of the...

work page
[2]

Include 1-2 practical application examples after the question

work page
[3]

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica

Openai o1 system card.arXiv preprint arXiv:2412.16720. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- codebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprint arXiv:2403.07974. Yuan Li, Yue Huang, Hongyi Wang, ...

arXiv 2024
[4]

Preserve original question wording

work page
[5]

NEVER mention or include any answer choices

work page
[6]

S Enh." represents the model enhanced by SFT, and

Qwen2 technical report.arXiv preprint arXiv:2407.10671. Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Benchmarking llm-based machine translation on cultural awareness.arXiv preprint arXiv:2305.14328. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or p...

arXiv 2022
[8]

Add 2-3 background sentences before the original question

work page
[10]

Keep all technical terms but expand their explanations

work page

Show all 13 references

[13]

[Output Format] New question: <Your modified question WITHOUT ANY CHOICES>

Omit any reference to multiple-choice options. [Output Format] New question: <Your modified question WITHOUT ANY CHOICES>
[2021]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al

Generalization in nli: Ways (not) to go beyond simple heuristics.arXiv preprint arXiv:2110.01518. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluat...

arXiv
[2023]

Job Schepens, Ton Dijkstra, and Franc Grootjen

Direct preference optimization: Your lan- guage model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728– 53741. Job Schepens, Ton Dijkstra, and Franc Grootjen. 2012. Distributions of cognates in europe as based on lev- enshtein distance.Bil...

2012 arXiv
[2024]

Lang Gao, Xiangliang Zhang, Preslav Nakov, and Xi- uying Chen

Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabili- ties.arXiv preprint arXiv:2404.17790. Lang Gao, Xiangliang Zhang, Preslav Nakov, and Xi- uying Chen. 2024. Shaping the safety boundaries: Understanding and defending against jailbrea...

2024 arXiv
[2025]

Preprint, arXiv:2408.02900

Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. Preprint, arXiv:2408.02900. Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for llms: A survey.arXiv preprint arXiv:240...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

text": "<Texte traduit en français>

The distraction should naturally integrate with the context of the question but must not explicitly introduce incorrect information or contradict the correct answer. 2. The distraction must be subtle and should not make it obvious that it is related to the incorrect option. [The Start of the Question] {question} [The End of the Question] [The Start of the...

work page

[2] [2]

Include 1-2 practical application examples after the question

work page

[3] [3]

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica

Openai o1 system card.arXiv preprint arXiv:2412.16720. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- codebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprint arXiv:2403.07974. Yuan Li, Yue Huang, Hongyi Wang, ...

arXiv 2024

[4] [4]

Preserve original question wording

work page

[5] [5]

NEVER mention or include any answer choices

work page

[6] [6]

S Enh." represents the model enhanced by SFT, and

Qwen2 technical report.arXiv preprint arXiv:2407.10671. Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023. Benchmarking llm-based machine translation on cultural awareness.arXiv preprint arXiv:2305.14328. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or p...

arXiv 2022

[7] [8]

Add 2-3 background sentences before the original question

work page

[8] [10]

Keep all technical terms but expand their explanations

work page