Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Meta-learning an initialization from other languages' preference data lets a model align to a low-resource target with as few as 100 samples, improving win rates by up to 28%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:30 UTC pith:O2V7JVHF

load-bearing objection A competent first pass at MAML for multilingual preference learning, with the big gains coming from cross-lingual transfer rather than the meta loop; still worth a referee. the 4 major comments →

arxiv 2607.13315 v2 pith:O2V7JVHF submitted 2026-07-14 cs.CL

Meta-Learning Preferences for Multilingual LLM Alignment

Jiaying Lin , Seongho Son , Nam Phuong Tran , Long Tran-thanh , Ilija Bogunovic , Debmalya Mandal This is my paper
classification cs.CL
keywords languagesdatapreferenceacrosslanguagemultilingualtargetalignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that multilingual LLM alignment can be made sample-efficient by treating each language as a task and meta-learning a shared initialization from preference data of high-resource languages. It instantiates this idea as MAML-RLHF and MAML-DPO and claims that, with only 100 target-language preference samples, MAML-DPO improves win rates by up to 28% over baselines across several languages and model sizes. On the theory side, the paper provides the first convergence guarantees for MAML applied to Bradley-Terry preference losses, for linear reward and log-linear policy classes. A sympathetic reader would care because preference data is scarce for most languages, so a provably transferable start point could lower the cost of aligning LLMs in low-resource settings.

Core claim

The central claim is that the initialization matters more than the amount of target data: a meta-initialization trained on preferences from other languages places the policy near the manifold of task optima, so a handful of gradient steps on the target language suffices to reach a well-aligned policy. The paper proves this for a log-linear policy class, where the per-task DPO loss is strongly convex with constant µ_τ = β²ν/(2(1+e^{Z_max})) under uniform feature coverage and a bounded reference-policy log-ratio, and shows that first-order MAML converges to the meta-optimal initialization with error E[‖θ*_M − θ̂M‖²] ≤ 2ασ/m_M, where m_M = β²cν³/(1+e^{Z_max})³. It also analyses the adaptation p

What carries the argument

The carrying mechanism is the MAML objective applied to preference losses: each language is a task with loss L(θ;τ) = −E[log σ(β θ^⊤Δψ − J)], and the meta-objective is L_M(θ) = E_τ[L(θ − α∇_θ L(θ;τ); τ)], whose minimizer is a first-order 'learned initialization' for fast adaptation. The argument rides on the per-task strong convexity of the Bradley-Terry sigmoid loss (Lemma D.3, constant µ_τ = β²ν/(2(1+e^{Z_max}))), which turns the meta-objective into an m_M-strongly convex function for small inner-loop step size, giving the convergence bound of Theorem 4.6. In the adaptation-phase comparison, two additional quantities do the work: the SFT preference-loss gap R_SFT, which measures how far an

Load-bearing premise

The convergence results assume each language's DPO loss is strongly convex, which requires the feature covariance E[ΔψΔψ^T] to be uniformly bounded below by νI (Assumption 4.1) and the reference-policy log-ratio to be bounded by J_0 (Assumption 4.4); the actual deep-network losses used in the experiments are not shown to satisfy these, so the theory does not directly certify the 28% win-rate improvement.

What would settle it

Measure the smallest eigenvalue of the empirical Hessian of the DPO loss (or the feature covariance) on the Okapi and orca_dpo_pairs datasets for the BLOOM 7.1B and Gemma 270M models after meta-training with LoRA; if ν is not positive or the log-ratio bound J_0 fails on held-out target data, the premise of strong convexity collapses. Alternatively, permute language labels during meta-training and repeat the 100-sample adaptation: if the ~28% win-rate gain persists under shuffled tasks, the effect cannot be attributed to cross-lingual preference structure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • With as few as 100 target-language preference samples, MAML-DPO improves win rates by up to 28% over target-only and multitask baselines, across model scales and language families.
  • The initialization advantage persists when adaptation data grows to 4k–40k samples, with MAML-DPO retaining the highest win rates in nearly all configurations.
  • The adaptation analysis implies cross-lingual pretraining of any kind (MAML or multitask) dominates target-only adaptation by a factor of log(R_SFT µ_τ/σ²) in required gradient steps, and that the gain grows when the SFT-initialized policy is far from preference-optimal.
  • The paper provides the first MAML convergence guarantees for Bradley-Terry preference losses, covering both reward-modeling (RLHF) and policy-optimization (DPO) settings.
  • Larger gains are predicted for harder targets: the bound scales with R_SFT, matching the larger observed gaps on Romanian and Indonesian.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the theory is proven for linear/log-linear models, a natural extension is to test the coverage and bounded-log-ratio assumptions on real preference data; if they fail, the 28% gain would need a different explanation (e.g., implicit regularization of LoRA).
  • The RSFT-scaling prediction is testable out-of-the-box: sort target languages by base-model win rate and check whether MAML-DPO's relative gain increases monotonically.
  • The finite-sample floor noted in the paper suggests the headline 28% figure includes variation not captured by the population bound; a finite-sample analysis with per-task gradient concentration would clarify how much of the empirical gain is certified.
  • Combining meta-learned initialization with gradient-filtering methods (which mitigate negative interference in joint training) could be complementary, since the meta-objective replaces joint training rather than filtering its gradients.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MAML-RLHF and MAML-DPO, two gradient-based meta-learning algorithms that treat each language as a task and learn a shared initialization from multilingual preference data. The authors state that with only 100 target-language preference samples, MAML-DPO improves win rates by up to 28% over baseline DPO, and claim to provide the first convergence guarantees for MAML on Bradley-Terry preference losses (Theorems 4.3 and 4.6). The empirical section uses Okapi and orca_dpo_pairs, BLOOM 7.1B and Gemma 270M models, and compares against target-only DPO/RLHF, a multitask baseline with the inner-loop learning rate set to zero, and a translate-test baseline. The paper also includes additional comparisons at larger adaptation budgets (4k/8k/12k/40k samples), out-of-distribution benchmark evaluations, and resource-consumption measurements.

Significance. If all claims are sustained, the paper would make a useful contribution: it identifies multilingual preference alignment as a meta-learning problem, proposes practical algorithms, supplies an analysis of MAML for sigmoidal preference losses, and provides initial evidence for sample-efficient adaptation in low-resource languages. The strengths include the multitask baseline designed to isolate meta-learning, standard errors reported for the 100-sample experiments, a translation-test baseline, an OOD robustness check, and public code. The theoretical results are nontrivial for the linear/log-linear model class analyzed. However, the distinctive "meta-learning beats multitask" claim is not yet established, and the theory is not verified on the actual deep-network experiments.

major comments (4)
  1. [§4.2, Theorem 4.6, Lemma D.3] The convergence guarantee is stated for a log-linear policy class (12) under Assumptions 4.1 and 4.4, with per-task strong convexity constant μτ = β²ν/(2(1+e^{Zmax})) (Lemma D.3). The experiments use BLOOM 7.1B and Gemma 270M with LoRA adapters (§5.1, Appendix A), whose DPO loss is neither log-linear nor verified to satisfy uniform feature coverage or the bounded reference-log-ratio assumption. The paper's own Remark §F.4(v) further limits Theorem F.3 to the population regime with a finite-sample floor. Consequently Theorems 4.3 and 4.6 do not mathematically certify the headline 28% improvement; they certify convergence to a meta-objective in a simplified model. The authors should either verify the required constants on the actual data or explicitly present the theory as illustrative and separate from the empirical claims.
  2. [§E.3, Eq. (37), Tables 2 and 6] The paper's central novelty is that meta-learning, not just multilingual pretraining, accelerates target-language adaptation. Theorem E.3 hinges on Φ > 0, defined in Eq. (37) as Φ = -⟨θ*_base − θ̄*, H_base^{-1}∇G(θ*_base)⟩. This quantity is never measured, bounded, or estimated; §E.3 only says one should "expect" it to be positive. §F.4(i) concedes that the large log(R_SFT μτ/σ²) improvement over baseline-DPO is shared by any cross-lingual initialization, and that the additional MAML-specific gain is O(α). Empirically, Tables 2 and 6 show no significant MAML-vs-multitask advantage for French (66.88±1.47 vs. 66.28±0.84) or Catalan (70.55±0.72 vs. 69.75±0.29; RLHF: 66.23±1.32 vs. 64.55±1.40 and 63.28±1.35 vs. 62.97±1.07) with only three seeds. Thus the distinctive meta-learning-specific claim is not established. I ask for a direct estimate or upper bound for Φ (or for the underlying task-g
  3. [§F.3, Eq. (56)] The derivation of the simplified lower bound (56) appears to have a direction error. In Theorem F.3, D²_BL = min(4B_θ², 2R_SFT/μτ). If D²_BL = 4B_θ² (which happens exactly when R_SFT ≥ 2B_θ² μτ), then replacing D²_BL by 2R_SFT/μτ inside the logarithm of (55) makes the argument larger, not smaller, so the claimed lower bound (1/2ημτ) ln(R_SFT μτ/(2σ²)) does not follow. In that regime the right-hand side of (55) is instead lower-bounded by (1/2ημτ) ln(B_θ² μτ²/σ²). The statement that the speed-up scales with log(R_SFT μτ/σ²), which is used in §F.4(iii) to explain larger gains on Romanian and Indonesian, therefore needs a case analysis or a corrected bound.
  4. [Tables 3, 8, 9, 10; §5.2] The claim that meta-trained models "consistently outperform" baselines at larger adaptation budgets is based on single point estimates: Tables 3, 8, 9, and 10 report no standard errors, number of seeds, or significance tests. In some cells the multitask baseline is close to or better than MAML, e.g., Table 8 Romanian 4k (86.95 vs. 84.50) and Catalan 4k (71.95 vs. 70.20); Table 9 French 12k (72.05 vs. 73.45 for multitask-RLHF) and French 40k (72.30 vs. 74.90). Without error bars the "highest win rates in every case" statement is not supported. Reporting three-seed standard errors for at least the 4k/12k rows would be necessary to sustain the empirical claims.
minor comments (4)
  1. [Figure 2 and §5.1] The legend/caption of Figure 2 is garbled: the text lists "DPO Multitask-DPO TT Multitask-DPO TT DPO MAML-DPO" in a way that does not identify which bars correspond to which methods. Please redraw with a clean legend.
  2. [Eq. (14) and Appendix D] The notation for the bound on |z_θ| is inconsistent between Zmax and Z_max. Please unify.
  3. [Lemma E.1] Lemma E.1 says "Let Theorem 4.5 hold," but the main text has Assumption 4.5 rather than Theorem 4.5. This cross-reference should be corrected.
  4. [Appendix A, Table 12] Table 12 lists training steps and samples in a way that is easy to misread (e.g., "Adaptation-DPO 100 100"). Please label the columns or add units so that steps, samples, wall-clock time, and memory are unambiguous.

Circularity Check

0 steps flagged

No circularity found: the derivation chain is a conditional convergence analysis supported by held-out self-contained experiments; the unmeasured Phi parameter and shared cross-lingual bound are validation caveats, not circular steps.

full rationale

The paper's derivation chain is not circular. Algorithm 1 directly minimizes the meta-objective (10); Theorem 4.6 shows first-order MAML converges to the meta-minimizer theta*_M under Assumptions 4.1/4.4/4.5. This is a convergence guarantee for an optimizer, not a result that presupposes the predicted outcome. The comparison against multitask-DPO is explicitly conditional: Theorem E.3's acceleration bound (43) contains the task-geometry parameter Phi defined in Eq. (37), and the proof only yields faster adaptation when Phi > 0. The paper does not measure or lower-bound Phi; its Remark in Section E.3 only says one should 'expect' Phi > 0. That is an unverified assumption and a correctness/validation gap, but not circular: the theorem does not redefine Phi as its conclusion. The comparison against baseline-DPO is also non-circular: Theorem F.3 quantifies the advantage in terms of R_SFT, mu_tau, and sigma^2, and the paper itself flags in Section F.4(i) that the large bound (56) is 'shared by MAML and multitask' and applies to 'any cross-lingual preprocessing'. This concession weakens the abstract's attribution of the 28% gain to meta-learning specifically, but it is an honest limitation rather than a fitted input renamed as a prediction. Section F.4(v) similarly concedes the bound is population/infinite-sample with a finite-sample floor, so the headline number is not directly certified by the theory; again a limitation, not a circular step. No load-bearing self-citation appears: the MAML convergence theorem invoked (Fallah et al., [17]) is external, and the self-citations to Son et al. [40] and Tang et al. [42] appear only in related-work discussions of non-stationary DPO and self-play, not as premises of the main derivation. Empirically, the evaluation uses separately trained judge reward models on held-out 1,000-prompt test splits and standard errors over three seeds; no parameter is fit to the reported win rates and then re-reported as a prediction. The central theory is therefore self-contained: it gives conditional convergence and adaptation-speed bounds from stated assumptions, and the remaining concerns (Phi unmeasured, population-level Assumptions 4.1/4.4/4.5 unverified on Okapi/orca_dpo_pairs, multitask vs MAML gaps often within error) are empirical validity risks, not circular reasoning.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

The central empirical result stands mainly on dataset/model choices and standard hyperparameters. The theory imports strong-convexity/coverage/bounded-log-ratio axioms that are not verified on the actual data, and the comparative advantage over multitask depends on an unmeasured Φ. The paper is honest about some of this in §F.4 (the MAML-specific gain is only O(α), the finite-sample floor is shared), so the ledger is moderate rather than severe.

free parameters (4)
  • β (DPO temperature) = Not disclosed in the text; standard DPO hyperparameter
    The DPO loss (4) and theory (Assumption 4.4, Theorem 4.6) depend on β, and its value affects Z_max; the empirical sections never state β, so the theory cannot be quantitatively checked against experiments.
  • Inner-loop learning rate coefficient = 3 (BLOOM), 1 or 3 (Gemma, chosen from pairs)
    The meta-learning inner-loop step is a free hyperparameter; Appendix A describes choosing from pairs like (5e-5,5) and (1e-5,3). This is a standard MAML hyperparameter rather than a physically meaningful constant, but it controls the meta-objective in Theorem 4.6.
  • Outer learning rate = 3e-5 (BLOOM), 5e-5/1e-5 (Gemma)
    Standard training hyperparameter, not a scientific constant; included for completeness since the convergence bounds depend on the optimization trajectory.
  • Adaptation learning rate = 1e-5 or 1e-4 to 5e-6 (selected between)
    The adaptation gradient steps use η chosen by validation loss; the theoretical Theorems E.3/F.3 require η ≤ 1/L_τ, but L_τ = β² is never connected to the chosen η in the experiments.
axioms (6)
  • domain assumption Assumption 4.1: uniform feature coverage E[ΔψΔψ^T] ⪰ νI for every task
    Required for strong convexity of the meta-objective (Lemma C.1, Lemma D.3). Not verified on Okapi/orca_dpo_pairs; the actual feature map ψ is a deep network embedding, and no ν is measured.
  • domain assumption Assumption 4.4: bounded reference log-ratio |log(π_SFT(y_w|x)/π_SFT(y_l|x))| ≤ J_0
    Used to bound z_θ by Z_max and to get the per-task strong convexity μ_τ (Lemma D.3). Plausible but not checked; if some DPO pairs have very low SFT probability, Z_max can be large and the bound degrades.
  • domain assumption Task distribution variance bound: E_τ[‖∇L(θ)-∇L(θ,τ)‖²] ≤ σ² (Assumptions 4.2/4.5)
    Standard in MAML literature [17]; needed for the convergence rate in Theorems 4.3/4.6 and for the comparative bounds D²_base = σ²/μ_τ². σ is never estimated from the language-task data.
  • standard math Linear reward and log-linear policy classes for theory (Eq. 11, 12)
    Theorems 4.3/4.6 are proved for r_ϕ(x,y)=ϕ^Tψ and softmax policy π_θ ∝ exp(θ^Tψ); this is a standard theoretical abstraction but does not cover the BLOOM/Gemma LoRA models used in Section 5.
  • ad hoc to paper Positive task-geometry parameter Φ > 0 for MAML to beat multitask (Theorem E.3)
    Theorem E.3 states the relative gain is positive only if Φ > 0, with Φ = -⟨θ⋆_base - θ̄⋆, H^{-1}_base ∇G(θ⋆_base)⟩. The paper asserts in a Remark that 'we should expect' this to be positive, but does not measure it. If Φ ≤ 0, the MAML-vs-multitask advantage is not guaranteed by the theory.
  • domain assumption Reference policy independence for MAML-DPO
    The DPO loss uses π_SFT as a fixed reference. In the MAML-DPO pipeline (Figure 4), the meta-trained policy is used directly as the adaptation start, and the paper's theory keeps π_SFT fixed; the interaction between a meta-changed θ and a fixed π_SFT is assumed not to invalidate the loss geometry.
invented entities (2)
  • MAML-RLHF / MAML-DPO meta-initializations independent evidence
    purpose: Transferable starting points for reward or policy adaptation across languages
    These are not physical entities but algorithmic constructs. Independent evidence is provided by empirical win rates against baselines (Tables 1-2, 6-7) and OOD benchmark tables, so there is a falsifiable handle outside the paper's own definitions.
  • Task-geometry parameter Φ no independent evidence
    purpose: Controls whether MAML adaptation is faster than multitask adaptation (Theorem E.3)
    Φ is defined through internal quantities θ⋆_base, θ̄⋆, H_base, and ∇G; the paper does not estimate it from data, so the theoretical condition Φ > 0 is not empirically established. This is a conceptual entity with no independent evidence.

pith-pipeline@v1.3.0-alltime-deepseek · 33745 in / 11534 out tokens · 83727 ms · 2026-08-02T05:30:34.368172+00:00 · methodology

0 comments
read the original abstract

Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to $28\%$ win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.

Figures

Figures reproduced from arXiv: 2607.13315 by Debmalya Mandal, Ilija Bogunovic, Jiaying Lin, Long Tran-Thanh, Nam Phuong Tran, Seongho Son.

Figure 1
Figure 1. Figure 1: When the target language has low resource availability (e.g. Romanian) and has a very [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: [Left] Win rate of Gemma 270M models adapted to 100 samples. MAML-DPO models [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: [Left] Win rates of BLOOM 7.1B models with different language groups for the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of DPO training pipelines. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of baseline RLHF pipeline for both model architectures. For Gemma3 270M [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of MAML-RLHF pipeline for BLOOM 7.1B model and Gemma3 270M [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average reward accuracies with standard deviation bands of Gemma 270M reward [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 29 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Apertus: Democratizing open and compliant llms for global language environments, 2025

    Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Ange- lika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, et al. Apertus: Democratizing open and compliant llms for global language environments, 2025. URL https://arxiv.org/abs/2509.14233

    arXiv 2025

  3. [3]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Investigating the translation performance of a large multilingual language model: the case of bloom.arXiv preprint arXiv:2303.01911, 2023

    Rachel Bawden and François Yvon. Investigating the translation performance of a large multilingual language model: the case of bloom.arXiv preprint arXiv:2303.01911, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    BLOOM (revision 4ab0472), 2022

    BigScience Workshop. BLOOM (revision 4ab0472), 2022. URL https://huggingface.co/ bigscience/bloom

    2022

  6. [6]

    Llms are few-shot in-context low- resource language learners.arXiv preprint arXiv:2403.16512, 2024

    Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. Llms are few-shot in-context low- resource language learners.arXiv preprint arXiv:2403.16512, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Meta-learning via lan- guage model in-context tuning

    Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via lan- guage model in-context tuning. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 719–730, Dublin, Ireland, May 2022. Association ...

    doi:10.18653/v1/2022.acl-long.53 2022

  8. [8]

    Tigerbot: An open multilingual multitask llm, 2023

    Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhanxuan Xin, and Cong Fu. Tigerbot: An open multilingual multitask llm, 2023. URLhttps://arxiv.org/abs/2312.08688

    Pith/arXiv arXiv 2023

  9. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  10. [10]

    Wang, and Eric Schulz

    Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matthew Botvinick, Jane X. Wang, and Eric Schulz. Meta-in-context learning in large language models, 2023. URL https://arxiv.org/ abs/2305.12907

    Pith/arXiv arXiv 2023

  11. [11]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.arXiv e-prints, pages arXiv–2307, 2023

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.arXiv e-prints, pages arXiv–2307, 2023

    2023

  13. [13]

    Evaluating and mitigating linguistic discrimination in large language models: Perspectives on safety equity and knowledge equity

    Guoliang Dong, Haoyu Wang, Jun Sun, and Xinyu Wang. Evaluating and mitigating linguistic discrimination in large language models: Perspectives on safety equity and knowledge equity. 2025

    2025

  14. [14]

    Rlhf workflow: From reward modeling to online rlhf,

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf,

  15. [15]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024. 11

    Pith/arXiv arXiv 2024

  16. [16]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    On the convergence theory of gradient- based model-agnostic meta-learning algorithms

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. InInternational Conference on Artificial Intelligence and Statistics, pages 1082–1092. PMLR, 2020

    2020

  18. [18]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. InInternational conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  19. [19]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, vo...

    2024

  20. [20]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  21. [21]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  22. [22]

    Llms as in-context meta-learners for model and hyperparameter selection, 2025

    Youssef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, and Balázs Kégl. Llms as in-context meta-learners for model and hyperparameter selection, 2025. URLhttps://arxiv.org/abs/2510.26510

    arXiv 2025

  23. [23]

    Meta-learning in neural networks: A survey.IEEE transactions on pattern analysis and machine intelligence, 44 (9):5149–5169, 2021

    Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey.IEEE transactions on pattern analysis and machine intelligence, 44 (9):5149–5169, 2021

    2021

  24. [24]

    Meta-learning online adaptation of language models.arXiv preprint arXiv:2305.15076, 2023

    Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. Meta-learning online adaptation of language models.arXiv preprint arXiv:2305.15076, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Krutrim llm: Multilingual foundational model for over a billion people, 2025

    Aditya Kallappa, Palash Kamble, Abhinav Ravi, Akshat Patidar, Vinayak Dhruv, Deepak Kumar, Raghav Awasthi, Arveti Manjunath, Himanshu Gupta, Shubham Agarwal, Kumar Ashish, Gautam Bhargava, and Chandra Khatri. Krutrim llm: Multilingual foundational model for over a billion people, 2025. URLhttps://arxiv.org/abs/2502.09642

    Pith/arXiv arXiv 2025

  26. [26]

    Mistral–a journey towards reproducible language model training, 2021

    Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, et al. Mistral–a journey towards reproducible language model training, 2021

    2021

  27. [27]

    Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback.arXiv preprint arXiv:2406.01771, 2024

    Wen Lai, Mohsen Mesgar, and Alexander Fraser. Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback.arXiv preprint arXiv:2406.01771, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Improving in-context learning of multilingual generative language models with cross-lingual alignment

    Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Improving in-context learning of multilingual generative language models with cross-lingual alignment. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8058–8076, 2024

    2024

  29. [29]

    Meta in-context learning makes large language models better zero and few-shot relation extractors,

    Guozheng Li, Peng Wang, Jiajun Liu, Yikai Guo, Ke Ji, Ziyu Shang, and Zijie Xu. Meta in-context learning makes large language models better zero and few-shot relation extractors,

  30. [30]

    Congrad: Conflicting gradient filtering for multilingual preference align- ment

    Jiangnan Li, Thuy Vu, Christian Herold, Amirhossein Tebbifakhr, Shahram Khadivi, and Gholamreza Haffari. Congrad: Conflicting gradient filtering for multilingual preference align- ment. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6371–6387, 2026. 12

    2026

  31. [31]

    URLhttps://arxiv.org/abs/2404.17807

    Pith/arXiv arXiv

  32. [32]

    Mend: Meta demonstration distillation for efficient and effective in-context learning, 2024

    Yichuan Li, Xiyao Ma, Sixing Lu, Kyumin Lee, Xiaohu Liu, and Chenlei Guo. Mend: Meta demonstration distillation for efficient and effective in-context learning, 2024. URL https: //arxiv.org/abs/2403.06914

    Pith/arXiv arXiv 2024

  33. [33]

    Language-emphasized cross-lingual in-context learning for multilingual llm

    Junlei Li, Xiao Wei, Xiaobao Wang, Ning Zhuang, Longbiao Wang, and Jianwu Dang. Language-emphasized cross-lingual in-context learning for multilingual llm. InCCF Interna- tional Conference on Natural Language Processing and Chinese Computing, pages 327–339. Springer, 2024

    2024

  34. [34]

    Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  35. [35]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  37. [37]

    Reward model learning vs

    Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovi´c, and Adish Singla. Reward model learning vs. direct policy optimization: A compar- ative analysis of learning from human preferences. InProceedings of the 41st International Conference on Machine Learning, 2024

    2024

  38. [38]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  39. [39]

    From r to Q*: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q*: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Right now, wrong then: Non-stationary direct preference optimization under preference drift

    Seongho Son, William Bankes, Sayak Ray Chowdhury, Brooks Paige, and Ilija Bogunovic. Right now, wrong then: Non-stationary direct preference optimization under preference drift. arXiv preprint arXiv:2407.18676, 2024

    arXiv 2024

  41. [41]

    Maml-en-llm: Model agnostic meta-training of llms for improved in-context learning

    Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, and Aidong Zhang. Maml-en-llm: Model agnostic meta-training of llms for improved in-context learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2711–2720, 2024

    2024

  42. [42]

    Game-theoretic regularized self-play alignment of large language models.arXiv preprint arXiv:2503.00030, 2025

    Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, and Ilija Bogunovic. Game-theoretic regularized self-play alignment of large language models.arXiv preprint arXiv:2503.00030, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Spo: Self preference optimization with self regularization

    Yuhao Sun, Yifan Zhang, Quandong Wang, QinZhuo Wu, Wei Liu, and Jian Luan. Spo: Self preference optimization with self regularization. InFindings of the Association for Computa- tional Linguistics: EMNLP 2025, pages 5601–5614, 2025

    2025

  44. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URLhttps://arxiv.org/abs/2403.05530

    Pith/arXiv arXiv 2024

  45. [45]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  46. [46]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Dai, Anja Hauth, Katie Millican, David Silver, et al

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, et al. Gemini: A family of highly capable multimodal models, 2025. URL https://arxiv.org/abs/2312. 11805. 13

    2025

  48. [48]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Towards multilingual llm evaluation for european languages.arXiv preprint arXiv:2410.08928, 2024

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, et al. Towards multilingual llm evaluation for european languages.arXiv preprint arXiv:2410.08928, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Performance unfairness of large language models in cross-language fact-checking.Information Processing & Management, 63 (4):104616, 2026

    Dandan Wang, Stephanie Jean Tsang, and Yadong Zhou. Performance unfairness of large language models in cross-language fact-checking.Information Processing & Management, 63 (4):104616, 2026. ISSN 0306-4573. doi: https://doi.org/10.1016/j.ipm.2026.104616. URL https://www.sciencedirect.com/science/article/pii/S0306457326000087

    arXiv 2026

  53. [53]

    Matching networks for one shot learning.Advances in neural information processing systems, 29, 2016

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning.Advances in neural information processing systems, 29, 2016

    2016

  54. [54]

    Pangea: A fully open multilingual multimodal llm for 39 languages

    Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neu- big. Pangea: A fully open multilingual multimodal llm for 39 languages. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  55. [55]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more, 2024

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu, Zhu, Xiang-Bo Mao, Sitaram Asur, Na, and Cheng. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more, 2024. URL https://arxiv.org/ abs/2407.16216

    Pith/arXiv arXiv 2024

  56. [56]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Pith/arXiv arXiv 1905

  57. [57]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, edi- tors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguis...

    doi:10.18653/v1/p19-1472 2019

  58. [58]

    anycross-lingual preprocessing dominates target-only adaptation by a log(RSFTµτ /σ2) factor in adaptation step count

    Yingtian Zou, Fusheng Liu, and Qianxiao Li. Unraveling model-agnostic meta-learning via the adaptation learning rate. InInternational Conference on Learning Representations, 2022. 14 Appendix Table of Contents A Further Details of Experimental Setting 16 B Additional Experimental Results 18 B.1 Additional Results of MAML-DPO with LoRA adapters . . . . . ....

    arXiv 2022

  59. [59]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences,

  60. [62]

    H Impact Statement and Limitations This work proposes a meta-learning approach for LLM preference learning, designed to mitigate the unequal performance across different languages

    proposes regularizing the self-play optimization method [41], further stabilizing the training and improving performances on an independent benchmark such as AlpacaEval [15]. H Impact Statement and Limitations This work proposes a meta-learning approach for LLM preference learning, designed to mitigate the unequal performance across different languages. B...

  61. [2020]

    URLhttps://arxiv.org/abs/1909.08593

    Pith/arXiv arXiv 1909

  62. [2024]

    URLhttps://arxiv.org/abs/2405.07863

    Pith/arXiv arXiv