Pith. sign in

REVIEW 4 major objections 6 minor 32 references

A bilevel reweighting scheme can recover the clean-preference optimum of Direct Preference Optimization from noisy preference labels, and a metadata-free implementation outperforms robust DPO baselines under 20–40% random label flips.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:54 UTC pith:BBUWX23D

load-bearing objection Useful method paper with a correct but partly tautological theory; the empirical claims need multi-seed support and the pseudo-label signal is validated only on a clean probe. the 4 major comments →

arxiv 2607.09796 v2 pith:BBUWX23D submitted 2026-07-09 cs.LG

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

Hua Qu , Yifan Li , Xiaodong Yuan This is my paper
classification cs.LG
keywords direct preference optimizationnoisy preference labelsmeta-learning reweightingprompt augmentation consistencybilevel optimizationsample weightingLLM alignmentlabel flipping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the damage noisy preference labels do to Direct Preference Optimization can be corrected by sample-level reweighting: under idealized conditions, a bilevel optimization whose outer objective uses clean-preference risk can recover exactly the model that clean data would have produced. It proves this recovery result, then builds a practical, metadata-free variant called PACMR-DPO, replacing clean validation labels with a consistency signal between a prompt and its back-translated version. Experiments on TL;DR summarization and Anthropic HH dialogue show that this learned weighting improves alignment over several robust DPO baselines at 20%, 30%, and 40% random label flips. The reason to care is that real preference data is noisy, and this is a route to robustness without requiring costly clean meta-preference labels.

Core claim

The central claim is that weighted DPO under noisy preferences has a pointwise optimum u*(g+,g−,z) = log(q/(1−q)) + log(g+(z)/g−(z)), so choosing the weight ratio equal to log(η/(1−η)) − log(q/(1−q)) cancels the noise-induced shift and recovers the clean DPO optimum u*_DPO-clean(z) = logit(η). Under idealized conditions this ratio is the optimum of a bilevel problem whose outer objective is the clean-preference risk. The paper then uses the noisy-trained model's implicit reward difference u(z) as a practical stand-in for the clean preference strength and trains a small network, VNet, to map [Δ(z), u(z)] to a weight g(z) = σ(a(z)u(z) + b(z)), with the outer signal supplied by prompt-augmentat

What carries the argument

The load-bearing identity is the weight-ratio correction: u*_W-DPO-noisy(z) = log(q/(1−q)) + log(g+(z)/g−(z)), so setting log(g+/g−) to logit(η) − logit(q) makes weighted noisy DPO match clean DPO. The implemented version replaces the two-sided weights with a single sigmoid gate VNet, g(z;Θ) = σ(a(z;Θ)u(z) + b(z;Θ)), where u(z) is the policy's implicit reward difference and Δ(z) is the implicit reward sum; the outer meta-objective is a pseudo-labelled consistency loss between the original prompt and its back-translation, with confidence threshold τ and a central-difference/LoRA approximation of the higher-order meta-gradient.

Load-bearing premise

The load-bearing premise is that the current policy's confident preference predictions are reliable enough to serve as pseudo-labels; the paper validates this with a fixed clean-trained probe, not with the noisy-trained policy whose errors the outer loop is supposed to correct.

What would settle it

Measure the clean-label accuracy of the pseudo-labels produced by the actual noisy-trained policy at τ=0.60 and 40% flips during PACMR-DPO training. If that accuracy is close to the flip rate (i.e., pseudo-labels are no better than random preference guesses), or if VNet assigns similar average weights to flipped and unflipped pairs, the reported win-rate gains would not be attributable to the augmentation-consistency reweighting.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Robust alignment no longer requires a clean validation preference set; consistency under prompt transformation can stand in as the outer meta-signal.
  • The learned weight function is interpretable: unflipped pairs receive higher mean weights than flipped pairs, and the gap shrinks as noise increases.
  • The theoretical recovery result implies that any noise model with label-independent sampling can in principle be corrected by the right weight ratio, not only symmetric random flips.
  • The central-difference treatment makes bilevel reweighting memory-feasible for LLM LoRA training, removing the need to retain the higher-order inner-update graph.
  • At 40% flips on HH, the win-rate advantage narrows while the win-score remains best, suggesting the method preserves overall comparison quality even when per-pair discrimination weakens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the learned weight shapes generalize across tasks at matched flip rates—as the figures suggest—VNet may be capturing a task-agnostic reliability signal; a direct test would be to pretrain VNet on one task and apply it to another without retraining.
  • The augmentation-consistency outer objective could be ported to other preference objectives (e.g., IPO or KTO) or other semantics-preserving transforms beyond English–Chinese back-translation, though the paper only tests the DPO-style loss and this one augmenter.
  • A testable extension is an annealed confidence threshold τ(t) that rises as the policy aligns; the paper uses a fixed τ=0.60, but pseudo-label accuracy should improve during training, so a schedule could help most at 40% noise.
  • The method's safety margin depends on the initial reference policy: if the reference is already biased toward flipped labels, the consistency objective could reinforce those biases rather than correct them, so the method's benefit may shrink when starting from a heavily misaligned policy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PACMR-DPO, a bilevel meta-reweighting method for DPO under randomly flipped preference labels. The inner level trains a weighted DPO objective, with per-sample weights produced by a small MLP (VNet) taking the implicit reward difference and sum as inputs. The outer level uses a prompt-augmentation consistency objective instead of clean metadata: pseudo-labels are derived from the current policy's own sigmoid implicit-reward score, confidence-filtered, and then used to enforce consistency between the original and back-translated prompt. The paper proves an idealized clean-optimum recovery theorem, derives a weight-prior construction under a general label-flipping model, and introduces a central-difference approximation of the meta-gradient in LoRA space. Experiments on TL;DR and Anthropic HH report improvements over cDPO, IPO, rDPO, and Dr.DPO at 20–40% label-flip noise.

Significance. If the empirical claims hold, the paper makes a useful practical contribution by showing that a meta-learned reweighting can be driven by prompt-augmentation consistency rather than clean metadata, with a scalable LoRA-based bilevel update. The algebraic derivations in §4 and Appendices C–D are coherent and appear correct, and the central-difference complexity analysis is a useful practical addition. The paper is also honest about the idealized nature of Theorem 1. However, the central methodological bridge — that the policy's own pseudo-labels track the clean preference direction while training on noisy data — is not validated for the actual in-loop policy, and the experimental evaluation is single-seed with no variance estimates. The significance therefore depends on closing that validation gap.

major comments (4)
  1. [§5.3 and Appendix E] The load-bearing assumption of the method is that the pseudo-labels ŷ_i(ω) in Eq. (35) track the clean preference direction for the policy actually being trained on noisy data. This is not established. Appendix E validates the prompt-augmentation-consistency signal only with a fixed clean-DPO probe on uncorrupted data (Table 7). The reported clean accuracies of 82.74% (TL;DR) and 75.58% (HH) at τ=0.60 therefore say nothing about a policy that may have confidently adopted flipped labels during noisy training. The E.3 relabeling-invariance statement further confirms that these metrics are convention-dependent and do not measure in-loop pseudo-label quality. I would like to see, during PACMR-DPO training, pseudo-label accuracy and weight statistics on known flipped vs. unflipped samples (the noise mask is available in the synthetic setup), or an equivalent analysis; without this, the outer
  2. [§6.1–6.3, Tables 1–2] All main results are single-seed point estimates (seed 42) with no confidence intervals, standard errors, or significance tests. Several reported differences are small, e.g., TL;DR 30% win-score 1.5263 vs. 1.5300 for fixed σ(u) in Table 4, and HH 40% win rate 40.63% vs. rDPO's 41.38% in Table 2. Pairwise evaluation is by GPT-5.1 with a single deterministic call per order and ties counted, which does not provide reliable error bars. I recommend multi-seed runs and/or bootstrap intervals, and ideally human or judge-agreement validation, before the claimed superiority over rDPO/Dr.DPO at high noise rates is established.
  3. [§6.5, Table 5] The main experiments use τ=0.60, which is selected after inspecting the sensitivity results in Table 5 and described as a compromise. Since τ is a free hyperparameter tuned on the same evaluation setup, the reported main-table numbers are selected rather than predictive. Please prespecify τ, or perform a validation-based selection and report the corresponding test performance, to avoid the appearance of post-hoc selection.
  4. [§4, Eq. (19) vs. §5.2, Eq. (36)] Theorem 1 is an idealized recoverability statement for a bilevel problem whose outer objective uses the clean preference posterior η_meta. The actual method replaces that outer objective with a consistency loss based on the policy's own pseudo-labels, restricts weights to an MLP on [Δ,u], and uses a one-step virtual update. The theorem therefore does not imply that PACMR-DPO has a fixed point at the clean DPO optimum. The paper should state this limitation explicitly and either prove a surrogate result under a model of consistent pseudo-labels or reframe Theorem 1 as motivation only. Currently §5.1 relies on Eq. (25) as if the in-training u(z) already approximates logit η(z), which is precisely the unvalidated step.
minor comments (6)
  1. [Abstract / Introduction] Missing space in 'KeywordsDirect Preference Optimization' and similar spacing issues in the title/header. Minor copyediting.
  2. [§3.2] The phrase 'becauseη train(z) =η(z)' lacks spaces around 'because'. Also, the notation Δ(z) for implicit reward sum is introduced in §5.1 without a formal definition; please define it explicitly.
  3. [Table 12] The cDPO row states 'ϵ=0.1 under 10% random flipping', but the experiments use 20%, 30%, and 40%. Clarify how the noise parameter is set for the reported rates.
  4. [Appendix E] In the first paragraph of E, 'we conduct a analysis' is a typo. Also, Eq. (118) defines exact clean accuracy but the connection to the relabeling invariance in E.3 would be clearer if the formula were written in terms of the latent clean label.
  5. [References] The win-score definition is cited to [11], but the reference list and the text in §6.2 would benefit from checking the source; if it is a standard definition, cite the original method that introduced it.
  6. [Appendix A] The complexity analysis is sound, but the notation 'P L' is used ambiguously: it should be written as P_L or 'P·L' to avoid confusion with the product of P and L.

Circularity Check

1 steps flagged

Theorem 1 is a self-definitional restatement of the clean-risk outer objective; the practical PAC signal is self-referential, but the win-rate claims rest on external baselines, so no deep circularity.

specific steps
  1. self definitional [Theorem 1, Eqs. (19)-(21); Appendix C]
    "min_{g+>0,g−>0} E_{z′∼Pmeta} [−η_meta(z′) log σ(u*(g+,g−,z′)) − (1−η_meta(z′)) log σ(−u*(g+,g−,z′))] ... Theorem 1 (Clean-optimum recovery by ideal bilevel reweighting). ... Then u*(g*+,g*−,z)=u*_DPO-clean(z)=log η(z)/(1−η(z))."

    The outer objective in (19) is by definition the clean DPO conditional risk (eq. 14) evaluated at the inner solution, with η_meta=η by Prop. 1. Appendix B proves this risk is uniquely minimized at u=logit η. Because g+ and g− are unconstrained pointwise, the inner formula (18) can realize any u by choosing log(g+/g−), so Theorem 1's conclusion (20) is simply the minimizer of the objective just written, not a prediction derived from noisy data. Appendix C confirms: 'this outer conditional objective reaches its unique minimum at u=log η/(1−η)' and then chooses weights to attain it. The clean-optimum recovery is thus built into the problem definition.

full rationale

The only formal reduction I can exhibit is Theorem 1. In eq. (19), the outer objective is the clean DPO conditional risk (eq. 14) under η_meta=η, and Appendix B shows its pointwise minimizer is logit(η). Since the inner weighted-DPO optimum (18) spans all real values as g+/g− varies, eq. (20) states little more than 'the minimizer of the clean risk is the clean optimum.' The proof in Appendix C says exactly that: the outer conditional objective reaches its unique minimum at u=log η/(1−η), then chooses weights to hit it. This is not an independent prediction from noisy-data mechanics; it is definitional. I therefore count it as one self-definitional step. That said, the paper's empirical claim is not circular: Tables 1-2 compare model outputs against external DPO baseline outputs using GPT-5.1, and the noise mask is fixed across methods. The PAC outer objective (28)-(36) is a self-training/consistency signal built from the model's own u; this is self-referential and Appendix E's validation uses a fixed clean DPO probe rather than the noisy-trained policy (E.3 explicitly limits it to 'the augmentation transformation rather than the degradation of a policy trained at a particular noise rate'). This is a support gap for the method's inner-loop reliability, but it is not a circular derivation of the win-rate results. No load-bearing self-citation or imported uniqueness theorem is present; [19] is cited as inspiration only. Score 3 rather than 6 because the central empirical contribution has independent external content and the tautological theorem is acknowledged as an idealized capacity statement.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No new physical or formal entities are introduced; VNet is a learned MLP parameterization. The main ledger items are idealized theoretical conditions, a self-referential pseudo-label reliability assumption, and hand-tuned hyperparameters such as τ and ε.

free parameters (4)
  • Confidence threshold τ = 0.6
    Selected from the sensitivity analysis in Table 5 using the same evaluation protocol as the main results; controls pseudo-label filtering in the outer objective.
  • Central-difference scale ε = 3e-3
    Hand-chosen finite-difference scale for eq. (44); no sensitivity analysis is reported for it.
  • Central-difference coefficient clipping constant = 10.0
    Clips the approximate directional-derivative coefficients ĉ_i in eq. (45); set by hand.
  • VNet architecture and training hyperparameters = MLP 64-16; lr 1e-3 (TL;DR), 5e-4 (HH); init a=1, b=0; update every 10 steps
    Architecture and training choices for the learned weighting network; no ablation is given except fixed-vs-learned weighting.
axioms (6)
  • domain assumption Assumption 1: label-independent sampling into train and meta sets
    Used in Proposition 1 and Theorem 1 (§3.1, eqs. (2)-(3)); the practical method avoids metadata but the clean-recovery theorem depends on it.
  • ad hoc to paper Ideal pointwise separability and unrestricted weight function space for g+, g-
    Appendix C invokes "distribution risk, pointwise separability, and sufficiently expressive function spaces" to make Theorem 1 an existence statement rather than a practical construction.
  • domain assumption Label noise follows a two-rate random-swap model
    The weight-prior construction in Appendix D assumes q = (1-ε0)η + ε1(1-η); experiments inject iid random candidate swaps, which is a special case. Real annotation noise may not satisfy this.
  • domain assumption Confident policy predictions approximate the clean preference posterior
    Pseudo-labels in §5.3 are derived from the current policy's implicit reward difference, with no guarantee under noisy training; Appendix E tests this only with a clean pre-trained probe.
  • standard math Losses are differentiable in LoRA parameters and central differences are accurate at ε=3e-3
    Eq. (44) approximates a directional derivative by finite differences; the paper sets ε and clipping without a bias analysis.
  • domain assumption Back-translation is semantics-preserving for preference direction
    Table 7 shows 94–97% direction agreement at τ=0.60 using a clean probe; the assumption may be weaker for noisy-trained policies and out-of-domain prompts.

pith-pipeline@v1.3.0-alltime-deepseek · 30985 in / 23333 out tokens · 213968 ms · 2026-08-02T07:54:27.246641+00:00 · methodology

0 comments
read the original abstract

Direct Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning. However, its performance depends heavily on the quality of preference data, and noisy preference data in real-world settings can weaken alignment performance. To address this issue, we propose a bilevel optimization framework and prove, under some idealized conditions, that this framework can recover the DPO optimum under clean data. We further derive a prior form for the learnable weighting function under label-flipping noise. Considering that high-quality metadata may be difficult to obtain, we propose a prompt augmentation consistency method that enables meta-learning even when metadata is completely unavailable. To reduce the high cost of higher-order gradients in LLM meta-learning, we combine central-difference approximation with LoRA fine-tuning and develop a scalable training scheme. Experiments on TL;DR summarization and Anthropic Helpful and Harmless dialogue show that the proposed method improves alignment performance over multiple DPO baselines under different noise rates.

Figures

Figures reproduced from arXiv: 2607.09796 by Hua Qu, Xiaodong Yuan, Yifan Li.

Figure 1
Figure 1. Figure 1: Final learned weight functions on TL;DR under 20%, 30%, and 40% random preference flips. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Final learned weight distributions on TL;DR. Unflipped and flipped pairs are identified using the [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Learned weight distributions on TL;DR. As the noise rate increases to 40%, the proportion of noisy samples grows, and clean and noisy samples increasingly overlap in the model’s implicit-reward feature space, making the task more difficult. The weight distributions still exhibit some separability, but the overlap increases, indicating that weight learning becomes harder at higher noise rates and sample dis… view at source ↗
Figure 3
Figure 3. Figure 3: Final learned weight functions on Anthropic HH under 20%, 30%, and 40% random preference [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Final learned weight distributions on Anthropic HH. Unflipped and flipped pairs are identified [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Learned weight distributions on Anthropic HH. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: VNet parameter and weight-function surfaces on TL;DR at step 500. Columns correspond to 20%, [PITH_FULL_IMAGE:figures/full_fig_p030_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: VNet training dynamics on TL;DR at step 500. Columns correspond to 20%, 30%, and 40% random flips; [PITH_FULL_IMAGE:figures/full_fig_p029_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: VNet parameter and weight-function surfaces on TL;DR at step 1000. Columns correspond to [PITH_FULL_IMAGE:figures/full_fig_p031_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: VNet training dynamics on TL;DR at step 1000. Columns correspond to 20%, 30%, and 40% random flips; [PITH_FULL_IMAGE:figures/full_fig_p030_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: VNet parameter and weight-function surfaces on TL;DR at step 2000. Columns correspond to [PITH_FULL_IMAGE:figures/full_fig_p032_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: VNet training dynamics on TL;DR at step 2000. Columns correspond to 20%, 30%, and 40% random flips; [PITH_FULL_IMAGE:figures/full_fig_p031_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: VNet parameter and weight-function surfaces on TL;DR at step 2901. Columns correspond to [PITH_FULL_IMAGE:figures/full_fig_p033_8.png] view at source ↗
Figure 8
Figure 8. Figure 8: VNet training dynamics on TL;DR at step 2901. Columns correspond to 20%, 30%, and 40% random flips; [PITH_FULL_IMAGE:figures/full_fig_p032_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: VNet parameter and weight-function surfaces on Anthropic HH at step 1000. Columns correspond [PITH_FULL_IMAGE:figures/full_fig_p034_9.png] view at source ↗
Figure 9
Figure 9. Figure 9: VNet training dynamics on Anthropic HH at step 1000. Columns correspond to 20%, 30%, and 40% random [PITH_FULL_IMAGE:figures/full_fig_p033_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: VNet parameter and weight-function surfaces on Anthropic HH at step 2000. Columns correspond [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: VNet training dynamics on Anthropic HH at step 2000. Columns correspond to 20%, 30%, and 40% [PITH_FULL_IMAGE:figures/full_fig_p034_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: VNet parameter and weight-function surfaces on Anthropic HH at step 3000. Columns correspond [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 11
Figure 11. Figure 11: VNet training dynamics on Anthropic HH at step 3000. Columns correspond to 20%, 30%, and 40% [PITH_FULL_IMAGE:figures/full_fig_p035_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: VNet parameter and weight-function surfaces on Anthropic HH at step 4699. Columns correspond [PITH_FULL_IMAGE:figures/full_fig_p037_12.png] view at source ↗
Figure 12
Figure 12. Figure 12: VNet training dynamics on Anthropic HH at step 4699. Columns correspond to 20%, 30%, and 40% [PITH_FULL_IMAGE:figures/full_fig_p036_12.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 9 linked inside Pith

  1. [1]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  4. [4]

    A survey of reinforcement learning from human feedback,

    T. Kaufmann, P. Weng, V . Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” arXiv preprint arXiv:2312.14925, 2023

    arXiv 2023

  5. [5]

    Secrets of rlhf in large language models part i: Ppo,

    R. Zheng, S. Dou, S. Gao, Y . Hua, W. Shen, B. Wang, Y . Liu, S. Jin, Q. Liu, Y . Zhouet al., “Secrets of rlhf in large language models part i: Ppo,”arXiv preprint arXiv:2307.04964, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  7. [7]

    Impact of preference noise on the alignment performance of generative language models,

    Y . Gao, D. Alon, and D. Metzler, “Impact of preference noise on the alignment performance of generative language models,”arXiv preprint arXiv:2404.09824, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Secrets of rlhf in large language models part ii: Reward modeling,

    B. Wang, R. Zheng, L. Chen, Y . Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shiet al., “Secrets of rlhf in large language models part ii: Reward modeling,”arXiv preprint arXiv:2401.06080, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Provably robust dpo: Aligning language models with noisy feedback,

    S. R. Chowdhury, A. Kini, and N. Natarajan, “Provably robust dpo: Aligning language models with noisy feedback,”arXiv preprint arXiv:2403.00409, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Towards robust alignment of language models: Distributionally robustifying direct preference optimization,

    J. Wu, Y . Xie, Z. Yang, J. Wu, J. Chen, J. Gao, B. Ding, X. Wang, and X. He, “Towards robust alignment of language models: Distributionally robustifying direct preference optimization,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 16 123–16 150

    2025

  11. [11]

    Robust reinforcement learning from corrupted human feedback,

    A. Bukharin, I. Hong, H. Jiang, Z. Li, Q. Zhang, Z. Zhang, and T. Zhao, “Robust reinforcement learning from corrupted human feedback,”Advances in Neural Information Processing Systems, vol. 37, pp. 124 093–124 113, 2024

    2024

  12. [12]

    A note on dpo with noisy preferences & relationship to ipo,

    E. Mitchell, “A note on dpo with noisy preferences & relationship to ipo,” 2023

    2023

  13. [13]

    ROPO: Robust preference optimization for large language models,

    X. Liang, C. Chen, S. Qiu, J. Wang, Y . Wu, Z. Fu, Z. Shi, F. Wu, and J. Ye, “ROPO: Robust preference optimization for large language models,”arXiv preprint arXiv:2404.04102, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Perplexity-aware correction for robust alignment with noisy preferences,

    K. Kong, X. Xu, D. Wang, J. Zhang, and M. Kankanhalli, “Perplexity-aware correction for robust alignment with noisy preferences,”Advances in Neural Information Processing Systems, vol. 37, pp. 28 296–28 321, 2024

    2024

  15. [15]

    Aligner, diagnose thyself: A meta-learning paradigm for fusing intrinsic feedback in preference alignment,

    M. Li, P. Zhao, and Z. Zhang, “Aligner, diagnose thyself: A meta-learning paradigm for fusing intrinsic feedback in preference alignment,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  16. [16]

    Learning temporally-aware sample weights for preference optimization,

    M. Li, X. Zhou, and P. Zhao, “Learning temporally-aware sample weights for preference optimization,” inFindings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 12 361–12 377

    2026

  17. [17]

    Meta-weight-net: Learning an explicit mapping for sample weighting,

    J. Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, and D. Meng, “Meta-weight-net: Learning an explicit mapping for sample weighting,”Advances in neural information processing systems, vol. 32, 2019

    2019

  18. [18]

    Cmw-net: Learning a class-aware sample weighting mapping for robust deep learning,

    J. Shu, X. Yuan, D. Meng, and Z. Xu, “Cmw-net: Learning a class-aware sample weighting mapping for robust deep learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 11 521–11 539, 2023

    2023

  19. [19]

    Dac-mr: Data augmentation consistency based meta-regularization for meta-learning,

    ——, “Dac-mr: Data augmentation consistency based meta-regularization for meta-learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    2026

  20. [20]

    Bilevel programming for hyperparameter optimization and meta-learning,

    L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil, “Bilevel programming for hyperparameter optimization and meta-learning,” inInternational conference on machine learning. PMLR, 2018, pp. 1568–1577

    2018

  21. [21]

    Meta-learning in neural networks: A survey,

    T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 9, pp. 5149–5169, 2021

    2021

  22. [22]

    Improving neural machine translation models with monolingual data,

    R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), 2016, pp. 86–96. 14

    2016

  23. [23]

    Understanding back-translation at scale,

    S. Edunov, M. Ott, M. Auli, and D. Grangier, “Understanding back-translation at scale,” inProceedings of the 2018 conference on empirical methods in natural language processing, 2018, pp. 489–500

    2018

  24. [24]

    Unsupervised data augmentation for consistency training,

    Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” Advances in neural information processing systems, vol. 33, pp. 6256–6268, 2020

    2020

  25. [25]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  26. [26]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models,”Iclr, vol. 1, no. 2, p. 3, 2022

    2022

  27. [27]

    Tl; dr: Mining reddit to learn automatic summarization,

    M. Völske, M. Potthast, S. Syed, and B. Stein, “Tl; dr: Mining reddit to learn automatic summarization,” in Proceedings of the workshop on new frontiers in summarization, 2017, pp. 59–63

    2017

  28. [28]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighanet al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,”arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    Learning to summarize with human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,”Advances in neural information processing systems, vol. 33, pp. 3008–3021, 2020

    2020

  30. [30]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework,

    J. Hu, X. Wu, Z. Zhu, W. Wang, D. Zhang, Y . Caoet al., “Openrlhf: An easy-to-use, scalable and high-performance rlhf framework,”arXiv preprint arXiv:2405.11143, vol. 6, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    A general theoretical paradigm to understand learning from human preferences,

    M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello, “A general theoretical paradigm to understand learning from human preferences,” inInternational Conference on Artificial Intelligence and Statistics. PMLR, 2024, pp. 4447–4455

    2024

  32. [32]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023. 15 Appendix A Exact Meta-Gradient and Complexity Analysis of the Central-Difference Approximation A.1 Notation an...

    2023