Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MPO: Multilingual Safety Alignment via Reward Gap Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that matching each language's internal reward gap to English's transfers safety alignment and cuts attack success rates across languages while preserving utility.

desk verdict Reward-gap regression for multilingual safety works, but the paper's own constant-target ablation shows the per-sample signal is not what's doing the work. read the letter →

arxiv 2505.16869 v1 pith:TSOVKV6G submitted 2025-05-22 cs.CL

Weixiang Zhao , Yulin Hu , Yang Deng , Tongtong Wu , Wenxuan Zhang , Jiahe Guo , An Zhang , Yanyan Zhao
show 3 more authors
Bing Qin Tat-Seng Chua Ting Liu
This is my paper
classification cs.CL
keywords multilingualsafetyalignmentrewardgapoptimizationdirectpreferenceattacksuccessratelow-resourcelanguagescross-lingualtransferjailbreakrobustnessrepresentationretention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the safety weakness of multilingual LLMs outside English can be repaired by using the model's own English safety signal as a training target. The signal is the reward gap: the length-normalized difference in log-likelihood the model assigns to safe versus unsafe responses. Across six languages on three LLMs, the authors observe that a larger reward gap tracks a lower attack success rate, with English far ahead of lower-resource languages. Their method, MPO, minimizes the squared difference between each target language's reward gap (scaled) and the fixed English reward gap, while a representation-retention term prevents English abilities from degrading. If the claim holds, noisy translated preference data is no longer the bottleneck: the dominant language's internal alignment can supervise safety in many languages at once.

What carries the argument

The load-bearing object is the length-normalized reward gap, defined as the average per-token log-likelihood of the safe response minus the average per-token log-likelihood of the unsafe response, both computed by the policy under training. MPO's loss squares the difference between β times the target-language reward gap and the reference model's English reward gap; because the English gap is constant with respect to the parameters, it acts as a fixed scale that tells the optimizer how strongly to push the target language's safe and unsafe likelihoods apart. A second term penalizes changes to the last-token hidden representation of English inputs, which is what keeps the dominant language from degrading during transfer.

What would settle it

Train MPO on a set of languages, then evaluate it on a held-out target language and on benign versions of the same prompts. The central claim is false if ASR does not drop on the held-out language or if the model begins refusing benign requests, because either outcome shows the reward-gap increase came from generic refusal behavior rather than transferred safety.

Watch

Extended reading notes

Core claim

The central claim is that the implicit reward gap—computed from the policy's length-normalized log-likelihood difference between a safe and an unsafe response—is a cross-lingual measure of safety alignment, and that minimizing the gap difference between a target language and English transfers safety behavior directly. The authors show that on LLaMA-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen2.5-7B-Instruct, English exhibits a large reward gap and low ASR while Bengali and Swahili show small gaps and high ASR. MPO's objective uses the reference model's English reward gap as a fixed anchor for each target language's scaled reward gap, with a second term constraining the last-token hidden representation of English inputs to stay close to its original value. Empirically, MPO achieves lower ASR than DPO, SimPO, ORPO, KTO, and other preference methods on MultiJail, AdvBench-X, and CSRT across all backbones, with the largest gains in low-resource languages, and it keeps MT-Bench, M-MMLU, and MGSM scores near their original levels.

Load-bearing premise

The load-bearing premise is that a larger length-normalized reward gap causes better safety, rather than merely correlating with it, so that pushing target-language reward gaps toward the English value will lower attack success rates. The paper supports this with a correlation measured on three base models, not with evidence that the regression objective preserves the causal link during training.

Editorial extensions

If this is right

  • Bengali MultiJail ASR falls from 49.52 to 12.38 and Swahili from 37.78 to 10.79 on LLaMA-3.1, showing the method's largest gains in the lowest-resource languages.
  • The same method transfers to Gemma-2, Qwen2.5, and Aya-101, so the supervision signal is not tied to one model family.
  • Target-language safety improves even when the translated preference data is noisy; MPO's ASR remains stable as translation quality varies, while preference baselines degrade.
  • Retention of dominant-language hidden representations keeps MT-Bench, M-MMLU, and MGSM scores close to the pre-training model, so the safety gains do not come at the cost of general ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A held-out-language test would settle whether reward-gap matching transfers zero-shot or merely fits training languages; the paper does not run this test.
  • Because the anchor is English's refusal-style gap, MPO likely imposes English-style refusal norms on other languages; culturally specific safety expectations would need separate evaluation.
  • A direct testable extension is applying gap matching to other language-agnostic skills where English leads, such as reasoning and instruction following; the same fixed-anchor mechanism should transfer those capabilities.
  • The robustness to translation noise suggests that low-resource safety alignment can proceed from cheap machine translations, which would lower the practical barrier to multilingual safety.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes MPO, a multilingual safety alignment method. The authors first observe that a length-normalized reward gap (the SimPO-style log-likelihood difference between safe and unsafe responses) correlates with attack success rate across six languages and three backbone models. MPO then trains the model to minimize the squared difference between the target-language reward gap beta * RG_t and a fixed reference-model English reward gap RG_d, with an additional representation-retention term. The experiments compare against SFT and several preference-learning baselines on MultiJail, AdvBench-X, and CSRT for LLaMA-3.1, Gemma-2, and Qwen2.5, reporting large ASR reductions in low-resource languages with minimal utility loss.

Significance. The direction is interesting: using a fixed internal reward gap of a well-aligned language as a supervision signal is a plausible way to avoid training directly on noisy translated preference data. The paper provides a correct gradient derivation, releases code, and evaluates across three backbones, three benchmarks, and eight baselines. The RG-ASR diagnostic in Table 1 is a useful observation. However, the central mechanistic claim that per-sample English reward gaps carry safety-specific semantic information is not supported by the paper's own constant-target ablation in Table 13, as detailed below. If the method were reframed as a target-margin safety alignment objective, a meaningful contribution would remain, but the current presentation needs substantial revision.

major comments (4)
  1. [§5.1 / Table 13] The fixed-constant ablation in Table 13 undermines the central claim that per-sample English reward gaps are the mechanism. Replacing RG_d with the scalar 1.58 (the training-set average) gives MultiJail average ASR 5.13 on LLaMA-3.1, compared with 5.98 for MPO, at essentially unchanged MT-Bench (5.17 vs 5.25); the constant 2.0 gives ASR 2.12. The text in §5.1 that this 'yields limited gains' is therefore inaccurate on the safety axis. These results are precisely what one would expect if the loss acts as a scalar target-margin knob on the target-language reward gap, rather than transferring per-sample semantic safety information from English. Please add a matched-utility comparison against a constant-target version of Eq. (5), measure post-training reward gaps on held-out target-language prompts, and show that the ASR reduction corresponds to recognizing harmful content rather than to a uniform refusal-strength shift.
  2. [§3.2, Eqs. (5)-(7)] The objective in Eq. (5) is written as an expectation over a mixed distribution D, but the paper does not state how each target-language triplet (x_t, y_t^w, y_t^l) is paired with the dominant-language triplet (x_d, y_d^w, y_d^l). If the pairing is not per-prompt, the squared difference is only matching marginal reward-gap statistics, and §3.3's description of 'per-sample' supervision is not justified. Please specify the pairing rule and, if the English prompt is not visible to the model, explain why the fine-grained values of RG_d are needed at all, given the Table 13 result that a constant suffices.
  3. [§4.1 / Appendix E] All experimental numbers are single runs without seeds or error bars, training uses only 100 translated prompts per language, and the ASR metric depends on GPT-4o translating each response to English before classification. For claims of consistent superiority over all baselines, report at least three seeds with standard deviations and provide a human-validated subset of the GPT-4o safety judgments, particularly for Bengali and Swahili. Several reported differences between MPO and the best baseline are small (e.g., Table 2, Gemma-2 on AdvBench-X: 0.96 vs 0.58 or 1.35), so without variance estimates the headline comparison is not fully supported.
  4. [§4.1 / Table 9] For Qwen2.5, Chinese is listed as a dominant language and also appears as a target language, yet MPO improves the Chinese ASR from 10.16 to 6.67 while a separate 'MPO - En Align' row is reported. Please state explicitly which reward gap is used as the pivot in each row of Table 9, how a target language that is also a dominant language is treated in Eq. (5), and why the default MPO for Qwen2.5 uses Chinese as the pivot while still improving Chinese safety.
minor comments (5)
  1. [Eq. (1)] Equation (1) contains an apparent stray 'x' at the end ('... + β log Z(x), x (1)'); please remove it.
  2. [Abstract / Figure 1] The inconsistent and ungrammatical capitalization 'reward gaP Optimization' and 'Log-Likelyhood' should be corrected.
  3. [Tables 2 and 9] The column header 'A VG.' is hard to parse and the subheaders are misaligned; replace with 'Avg.' and format each benchmark's language columns clearly.
  4. [§3.1 / Table 1] The 'strong correlation' between RG and ASR is supported only by visual inspection of six languages per model; please report the correlation coefficient (e.g., Spearman) and note the sample size.
  5. [Limitations] The Limitations section does not acknowledge the fixed-constant ablation or the proxy-stability question; a sentence noting that the mechanism has not been isolated would improve transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training signal is a fixed reference-model reward gap and safety is measured by an external GPT-4o ASR pipeline.

full rationale

The derivation chain is self-contained. The paper's premise is an empirical correlation (Table 1) between the length-normalized reward gap RG_t and externally judged ASR across languages; this correlation is an observation, not a consequence of the definitions of RG or ASR. The MPO objective L1 = E[(beta RG_t - RG_d)^2] treats RG_d as a constant computed from the reference model pi_ref on dominant-language data, so the supervision signal is exogenous to the policy being optimized, and the final safety evaluation uses held-out benchmarks (MultiJail, AdvBench-X, CSRT) with a GPT-4o-based ASR pipeline. No fitted parameter is relabeled as a prediction, and no equation reduces to its own input. The only self-citations (e.g., LENS) appear as baseline comparisons and are not load-bearing premises. The constant-target ablation in Table 13 (c=1.58 yields MultiJail ASR 5.13 vs MPO 5.98) is a substantive threat to the paper's claim that per-sample reward gaps are the causal mechanism, but it is a correctness/robustness concern, not a circularity: it does not show that MPO's output is equivalent, by construction, to its training signal. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are standard hyperparameters, beta and a fixed L2 weight, plus a training-data size choice. The load-bearing assumptions are the empirical reliability of the reward gap as a safety proxy and the transferability of safety via reward-gap regression; both are domain assumptions supported by experiments, not by derivation.

free parameters (3)
  • beta (reward gap scale) = 1.0 (LLaMA-3.1), 1.5 (Gemma-2, Qwen2.5)
    Searched in [1.0, 1.5, 2.0] in Appendix G; best value picked per backbone.
  • L2 retention weight = 1.0 (fixed)
    The representation retention loss in Eq. 8 is added with weight 1.0 without a reported search; it is a design choice.
  • training data size per language = 100 prompts
    Sampled from PKU-SafeRLHF and translated; data quantity is a choice that affects results, as shown in Fig. 3b.
assumptions (4)
  • domain assumption Reward gap (length-normalized log-likelihood difference) is a reliable indicator of safety across languages
    Motivated by Table 1 correlation with ASR; used to justify the MPO objective.
  • domain assumption Matching target-language reward gap to the English reference gap transfers safety behavior
    Core premise of Eq. 5; not proven, only empirically supported.
  • domain assumption The frozen reference model's English reward gap is a stable, well-calibrated target
    RG_d is computed once from pi_ref; assumes the original model's English safety gap is the right level for all languages.
  • domain assumption Constraint on hidden representations of dominant language preserves utility without blocking safety transfer
    Eq. 8 is justified by ablation; no theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPO: Multilingual Safety Alignment via Reward Gap Optimization." pith.science (2026). https://pith.science/paper/TSOVKV6G

@misc{pith2026250516869,
  author       = {Pith},
  title        = {Pith review of: MPO: Multilingual Safety Alignment via Reward Gap Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TSOVKV6G}},
  note         = {Machine review of arXiv:2505.16869}
}
read the original abstract

Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility.

Figures

Figures reproduced from arXiv: 2505.16869 by the authors.

Figure 1
Figure 1. Top: Current preference learning methods [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The results of replacing the dominant lan [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Impact of the preference data. (a) Multilingual safety performance on MultiJail with varied data quality. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Reward gap across languages for the original [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: The visualization of multilingual representations for safe and unsafe inputs across different languages. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

    cs.CL 2025-05 accept novelty 6.0 of 10

    LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.

Reference graph

Works this paper leans on

95 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker, et al. 2024. The multilingual alignment prism: Aligning global and local preferences to reduce harm. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12027--12049

    2024

  4. [4]

    Meta AI. 2024. https://about.meta.com/actions/safety/topics/safety-basics/policies Meta safety policies . Meta

    2024

  5. [5]

    Carlo Alfano, Silvia Sapora, Jakob Nicolaus Foerster, Patrick Rebeschini, and Yee Whye Teh. 2024. Learning loss landscapes in preference optimization. arXiv preprint arXiv:2411.06568

    arXiv 2024

  6. [6]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  7. [7]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  8. [8]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

Show all 95 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  2. [10]

    Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, and Pascale Fung. 2024. High-dimension human value representation in large language models. arXiv preprint arXiv:2404.07900

    2024 arXiv

  3. [11]

    Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xianpei Han, et al. 2024. Towards scalable automated alignment of llms: A survey. arXiv preprint arXiv:2406.01252

    2024 arXiv

  4. [12]

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning

    2024

  5. [13]

    Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. 2024. Provably robust dpo: Aligning language models with noisy feedback. In Forty-first International Conference on Machine Learning

    2024

  6. [14]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  7. [15]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    2022 arXiv

  8. [16]

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations

    2024

  9. [17]

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767

    2023 arXiv

  10. [18]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  11. [19]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  12. [20]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, and Mikel Artetxe. 2024. Do multilingual language models think better in english? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    2024

  13. [21]

    Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, and Kristian Kersting. 2024. Llms lost in translation: M-alert uncovers cross-linguistic safety gaps. arXiv preprint arXiv:2412.15035

    2024 arXiv

  14. [22]

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858

    2022 arXiv

  15. [23]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  16. [24]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792

    2024 arXiv

  17. [25]

    Prannaya Gupta, Le Yau, Hao Low, I-Shiang Lee, Hugo Lim, Yu Teoh, Koh Hng, Dar Liew, Rishabh Bhardwaj, Rajat Bhardwaj, et al. 2024. Walledeval: A comprehensive safety evaluation toolkit for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Na...

    2024

  18. [26]

    Vera F Guti \'e rrez-Clellen. 1999. Language choice in intervention with bilingual children. American Journal of Speech-Language Pathology, 8(4):291--302

    1999

  19. [27]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861--1870. PMLR

    2018

  20. [28]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

    2021

  21. [29]

    Jiwoo Hong, Noah Lee, Rodrigo Mart \' nez-Casta \ n o, C \'e sar Rodr \' guez, and James Thorne. 2024 a . Cross-lingual transfer of reward models in multilingual alignment. arXiv preprint arXiv:2410.18027

    2024 arXiv

  22. [30]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024 b . Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

    2024

  23. [31]

    Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. 2024. Large language models are cross-lingual knowledge-free reasoners. arXiv preprint arXiv:2406.16655

    2024 arXiv

  24. [32]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  25. [33]

    Yue Huang, Chenrui Fan, Yuan Li, Siyuan Wu, Tianyi Zhou, Xiangliang Zhang, and Lichao Sun. 2024. 1+ 1> 2: Can large language models serve as cross-lingual knowledge aggregators? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13...

    2024

  26. [34]

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513

    2024 arXiv

  27. [35]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  28. [36]

    Arturs Kanepajs, Vladimir Ivanov, and Richard Moulange. 2024. Towards safe multilingual frontier ai. In Workshop on Socially Responsible Language Modelling Research

    2024

  29. [37]

    Sungdong Kim and Minjoon Seo. 2024. Rethinking the role of proxy rewards in language model alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20656--20674

    2024

  30. [38]

    Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024 a . Improving in-context learning of multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Li...

    2024

  31. [39]

    Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024 b . A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765

    2024 arXiv

  32. [40]

    Yahan Li, Yi Wang, Yi Chang, and Yuan Wu. 2024 c . Xtrust: On the multilingual trustworthiness of large language models. arXiv preprint arXiv:2409.15762

    2024 arXiv

  33. [41]

    Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024 a . Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258

    2024 arXiv

  34. [42]

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. 2024 b . Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878

    2024 arXiv

  35. [43]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

    2024 arXiv

  36. [44]

    OpenAI. 2024 a . https://cdn.openai.com/gpt-4o-system-card.pdf Gpt-4o system card . OpenAI

    2024

  37. [45]

    OpenAI. 2024 b . https://openai.com/policies/usage-policies/ Openai use policies . OpenAI

    2024

  38. [46]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  39. [47]

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159

    2024 arXiv

  40. [48]

    Samuele Poppi, Zheng-Xin Yong, Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, and Jianfeng Chi. 2024. Towards understanding the fragility of multilingual llms against fine-tuning attacks. arXiv preprint arXiv:2410.18210

    2024 arXiv

  41. [49]

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695--2709

    2023

  42. [50]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925

    2024 arXiv

  43. [51]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

    2023

  44. [52]

    Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto. 2023. Empowering multi-step reasoning across languages via tree-of-thoughts. arXiv preprint arXiv:2311.08097

    2023 arXiv

  45. [53]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,...

    2020

  46. [54]

    Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. 2024. Unintentional unalignment: Likelihood displacement in direct preference optimization. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning

    2024

  47. [55]

    Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715

    2024 arXiv

  48. [56]

    Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. 2024. MAPO : Advancing multilingual reasoning through multilingual-alignment-as-preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  49. [57]

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. The language barrier: Dissecting safety challenges of LLM s in multilingual contexts. In Findings of the Association for Computational Linguisti...

    2024

  50. [58]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations

    2023

  51. [59]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716

    2023 arXiv

  52. [60]

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024 a . Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990--18998

    2024

  53. [61]

    Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2024 b . Multilingual blending: Llm safety alignment evaluation with language mixture. arXiv preprint arXiv:2407.07342

    2024 arXiv

  54. [62]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070

    2024 arXiv

  55. [63]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  56. [64]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  57. [65]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  58. [66]

    Bibek Upadhayay and Vahid Behzadan. 2024. Sandwich attack: Multi-language mixture adaptive attack on llms. arXiv preprint arXiv:2404.07242

    2024 arXiv

  59. [67]

    Nikhil Verma and Manasa Bharadwaj. 2025. The hidden space of safety: Understanding preference-tuned llms in multilingual context. arXiv preprint arXiv:2504.02708

    2025 arXiv

  60. [68]

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. 2024 a . Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080

    2024 arXiv

  61. [69]

    Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. 2024 b . Sharing matters: Analysing neurons across languages and tasks in llms. arXiv preprint arXiv:2406.09265

    2024

  62. [70]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024 c . All languages matter: On the multilingual safety of LLM s. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865--5877

    2024

  63. [71]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  64. [72]

    Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 2024 a . -dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410.10148

    2024 arXiv

  65. [73]

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024 b . Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675

    2024 arXiv

  66. [74]

    Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami. 2024 c . Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1332--1353

    2024

  67. [75]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024 a . Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In Forty-first International Conference on Machi...

    2024

  68. [76]

    Nan Xu, Fei Wang, Ben Zhou, Bangzheng Li, Chaowei Xiao, and Muhao Chen. 2024 b . Cognitive overload: Jailbreaking large language models with overloaded logical thinking. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3526--3548

    2024

  69. [77]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  70. [78]

    Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, and Jiajun Zhang. 2024 b . Language imbalance driven rewarding for multilingual self-improving. arXiv preprint arXiv:2410.08964

    2024 arXiv

  71. [79]

    Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, and Xing Xie. 2024. Value fulcra: Mapping large language models to the multidimensional spectrum of basic human value. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu...

    2024

  72. [80]

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387

    2025 arXiv

  73. [81]

    Zheng Xin Yong, Cristina Menghini, and Stephen Bach. 2023. Low-resource languages jailbreak gpt-4. In Socially Responsible Language Modelling Research

    2023

  74. [82]

    Haneul Yoo, Yongjin Yang, and Hwaran Lee. 2024. Code-switching red-teaming: Llm evaluation for safety and multilingual understanding. arXiv preprint arXiv:2406.15481

    2024 arXiv

  75. [83]

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36

    2023

  76. [84]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning

    2024

  77. [85]

    Shimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. 2024 a . Getting more from less: Large language models are good spontaneous multilingual learners. In Proceedings of the 2024 Conference on Empirical Methods i...

    2024

  78. [86]

    Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024 b . Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. In Proceedings of the 62nd Annual Meeting of the Ass...

    2024

  79. [87]

    Weixiang Zhao, Yulin Hu, Jiahe Guo, Xingyu Sui, Tongtong Wu, Yang Deng, Yanyan Zhao, Bing Qin, Wanxiang Che, and Ting Liu. 2024 a . Lens: Rethinking multilingual enhancement for large language models. arXiv preprint arXiv:2410.04407

    2024 arXiv

  80. [88]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024 b . Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations

    2024

  81. [89]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2024 a . Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations

    2024

  82. [90]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  83. [91]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024 b . L lama F actory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), ...

    2024

  84. [92]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36

    2023

  85. [93]

    Zhenglin Zhou, Xiaobo Xia, Fan Ma, Hehe Fan, Yi Yang, and Tat-Seng Chua. 2025. Dreamdpo: Aligning text-to-3d generation with human preferences via direct preference optimization. arXiv preprint arxiv:2502.04370

    2025 arXiv

  86. [94]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

  87. [95]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.