Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Symmetric losses preserve reward ranking under asymmetric preference noise, yielding provable policy improvement in RLHF and offline optimization.

desk verdict The paper's central claim—that symmetric losses make reward learning robust to asymmetric preference noise—is false as stated; a two-action counterexample within the paper's own assumptions flips the risk minimizer. read the letter →

arxiv 2505.24709 v1 pith:6CLJL7MI submitted 2025-05-30 cs.LG cs.AI

Soichiro Nishimori , Yu-Jie Zhang , Thanawat Lodkaew , Masashi Sugiyama This is my paper
classification cs.LGcs.AI MSC 68T0562F35
keywords reinforcementlearningfromhumanfeedbackpreferenceoptimizationnoisylabelssymmetriclossrank-preservingrewardpolicyimprovementdirectnoise-robustalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reward models trained on human preference data can be made robust to annotation noise by viewing reward modeling as binary classification and using symmetric losses, those satisfying $\ell(z)+\ell(-z)=K$ for a constant $K$. Its central proof shows that under an asymmetric per-instance noise model, the risk minimizer of a symmetric loss preserves the ranking of the true reward, so no noise-rate estimate is needed. Because rank preservation is sufficient for policy improvement in both RLHF and offline preference optimization, the paper derives a policy objective, SymPO, with a provable improvement guarantee. Synthetic MNIST reward-modeling tests and language-model alignment experiments with noisy preference labels are presented as empirical support.

What carries the argument

The argument is carried by three connected pieces. The flip-symmetrization lemma (Lemma 1) exploits the antisymmetry $g(a_1,a_2)=-g(a_2,a_1)$ of reward-based scoring functions to reduce asymmetric label noise to symmetric noise with a known effective rate. The symmetric loss condition $\ell(z)+\ell(-z)=K$ makes risk rankings invariant under that symmetric noise, so noisy-risk minimizers match clean-risk minimizers. Finally, rank preservation (Definition 2) is shown to be the property that transfers correctness of the reward to improvement of the policy, via the analytical form of the KL-regularized optimal policy and the implicit-reward identity used in DPO-style methods.

What would settle it

Construct a two-action space with $p(a,b)=0.9$, $p(b,a)=0.1$, asymmetric noise $\varepsilon_p=0.4$, $\varepsilon_n=0$, and any symmetric loss. If the noisy-risk minimizer's ordering of $a$ and $b$ differs from the clean-risk minimizer's ordering for some sample size, Proposition 1 fails in a configuration the paper's assumptions allow.

Watch

Extended reading notes

Core claim

The central discovery is Proposition 1: for any symmetric loss, the risk minimizer under the asymmetric noise model of Eq. (7) coincides with the clean risk minimizer in the sense that the learned reward is rank-preserving with respect to the true reward. The proof rests on Lemma 1, which shows that because the scoring function changes sign when the two actions are swapped, randomly flipping action order with probability $1/2$ converts asymmetric noise into symmetric noise with effective rate $\pi_p\varepsilon_p+(1-\pi_p)\varepsilon_n$. Combined with Theorem 2, which states that classification-calibrated losses induce rank-preserving rewards under full support, the paper obtains Corollary 1: symmetric losses give robust policy improvement for both RLHF and offline preference optimization. This turns the known noise-robustness of symmetric losses in classification into a policy-level guarantee.

Load-bearing premise

The key premise is that randomly swapping the two actions converts asymmetric label noise into symmetric noise without changing the risk; that holds only when preference pairs are swap-symmetric in distribution or when the two noise rates are equal, and the paper's assumptions guarantee neither.

Editorial extensions

If this is right

  • SymPO needs no prior knowledge of the noise rate, unlike rDPO, and its robustness holds for asymmetric as well as symmetric noise.
  • A reward that only ranks actions correctly is enough for policy improvement; exact reward values are not required, so non-CPE symmetric losses are usable.
  • Sigmoid, ramp, and unhinged losses all inherit the same guarantee when they are classification-calibrated.
  • The guarantee covers both explicit reward-then-RL (RLHF) pipelines and offline implicit-reward objectives such as DPO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Lemma 1's equivalence is a distribution-level identity; it presumes the ordered-pair marginal is swap-symmetric or that $\varepsilon_p=\varepsilon_n$. Measuring how rank error grows as the ordered-pair distribution becomes skewed would test the practical scope of Proposition 1.
  • Because symmetric losses are not class-probability-estimation losses, SymPO trades away reward calibration; a two-stage design that first ranks with a symmetric loss and then calibrates magnitudes on a clean subset could recover both.
  • The theory is asymptotic and instance-independent; for finite samples and instance-dependent noise (harder pairs flipped more often), the flip-symmetrization step may need margin weighting, which is a testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Symmetric Preference Optimization (SymPO), an offline preference optimization method that treats reward modeling as binary classification and uses symmetric losses (sigmoid, ramp, unhinged) to be robust to noisy preference labels. The central theoretical claim is that under an asymmetric label-noise model, symmetric losses preserve the ranking of the true reward, so that the risk minimizer under noisy preferences is rank-preserving and yields policy improvement. The paper proves a connection between classification-calibrated losses and rank-preserving rewards, and reports experiments on MNIST-based synthetic preferences and on Anthropic HH and UltraFeedback Binarized language-model alignment tasks.

Significance. If the central claim were correct, the paper would provide a simple, hyperparameter-free method for preference optimization under asymmetric noise, which is a realistic and important setting. The paper also contributes a useful framing that connects classification calibration to policy improvement, and it releases code. However, the load-bearing theoretical result, Proposition 1, is false: the reduction from asymmetric to symmetric noise is only aggregate and does not produce the per-instance symmetric noise required by the invoked robustness theorems. The experiments do not test asymmetric noise at all, since they only inject symmetric flips. The paper's main contribution is therefore not established, and the theoretical guarantee for the headline setting is invalid.

major comments (3)
  1. [§3.1, Lemma 3 and Proposition 1] The proof of Proposition 1 is invalid because Lemma 3 does not reduce asymmetric noise to the per-instance symmetric noise required by the robustness theorem of van Rooyen et al. [35]. Random flipping of the input pair matches only the aggregate class prior and the aggregate error rate πpεp+(1−πp)εn; after flipping, the conditional flip rates for a specific pair (a,b) are P(Ỹ'=−1|Y'=+1,X'=(a,b)) = (p(a,b)εp + p(b,a)εn)/(p(a,b)+p(b,a)) and P(Ỹ'=+1|Y'=−1,X'=(a,b)) = (p(a,b)εn + p(b,a)εp)/(p(a,b)+p(b,a)). These two rates are equal and input-independent only if p(a,b)=p(b,a) for all unordered pairs or εp=εn; Assumption 2 does not provide this. The failure is concrete: take A={a,b}, p(a,b)=0.9, p(b,a)=0.1 (adding arbitrarily small mass on (a,a) and (b,b) to satisfy Assumption 2), η=P(a≻b)=0.6, εp=0.4, εn=0, and sigmoid loss ℓ(z)=(1+e^z)^{−1}. The clean risk (Eq. (6)) is 0.6ℓ(θ)+0.4ℓ(−θ), minimized as θ=r(a)−r(b)→+∞, while the noisy risk (Eq. (8)) is 0.4ℓ(θ)+0.6ℓ(−θ), minimized as θ→−∞. This directly contradicts the rank-preservation claim of Proposition 1 and invalidates Corollary 1 for the asymmetric noise setting.
  2. [§5.1 and §5.2] The experiments do not test the asymmetric-noise model that motivates the paper. In both the MNIST experiment (§5.1) and the language-model experiment (§5.2), noise is injected by randomly flipping the label with a single rate ε ∈ {0.0,0.1,0.2,0.3,0.4}, which is symmetric noise with εp=εn. Thus Tables 2 and 3 only demonstrate the known symmetric-noise robustness of symmetric losses, not the paper's claim of robustness under the asymmetric model in Eq. (7). Moreover, the claimed advantage over ROPO on UFB is small relative to the reported standard deviations (e.g., Table 6: at ε=0.4, sigmoid 60.7±1.3 vs ROPO 58.4±0.9), so the empirical results would not, by themselves, support the asymmetric-noise conclusion even if the theory were correct.
  3. [§4.2 and Corollary 1] The policy-improvement guarantee for SymPO under noisy preferences is stated for the asymmetric noise model via Corollary 1, but the proof of Corollary 1 relies entirely on Proposition 1. Since Proposition 1 fails, the corollary's guarantee under Eq. (7) is unsupported. The paper's Theorem 2 and the rank-preservation result for classification-calibrated losses under clean labels may be correct, but they do not salvage the asymmetric-noise claim; at best they establish a clean-label or symmetric-noise guarantee, which is a substantially weaker statement than the abstract and introduction promise.
minor comments (4)
  1. [§3.1, Lemma 1] The informal statement of Lemma 1 says the risk is equivalent to that with symmetric noise 'for error rate πpεp+(1−πp)εn with p(y=+1)=1/2'; this conflates the class prior of the flipped distribution with an assumption on the original data. Please clarify that p(y=+1)=1/2 is a property of the distribution after the random flip, not a condition imposed on the original preference distribution.
  2. [App. C.2, Lemma 4] The notation in Lemma 4 is inconsistent: rmax_i is defined as min_{a∈Ai} r(a), while the surrounding text and inequalities suggest it should be a maximum; similarly, the chain of inequalities leading to Δ_i < Δ_j omits justification for the final inequality. Please correct the notation and fill in the missing steps.
  3. [§5.2, Table 3] The text states that SymPO 'outperforms ROPO at higher noise rates' on UFB, but the reported standard deviations in App. D.2 show overlapping intervals at several settings (e.g., ε=0.2: 63.5±1.4 vs 62.8±1.0; ε=0.3: 62.8±1.1 vs 61.0±1.3). Please report the actual t-test results or temper the claim to reflect the statistical uncertainty.
  4. [§5.2, Eq. (10)] The ramp loss as defined in App. B.2 is non-differentiable, but the paper does not describe how gradients are computed for the ramp loss in the language-model experiments; please state the subgradient or surrogate used in the implementation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the derivation uses external theorems and original algebra, though Proposition 1 has a non-circular proof gap.

full rationale

SymPO's load-bearing claims do not reduce to their own inputs. Proposition 1 (Eq. 9) is derived from Lemma 1 (random-flip symmetrization) plus Theorem 3 of Van Rooyen et al. (2015), an external result on symmetric losses under symmetric noise; neither Lemma 1 nor the proposition assumes the rank-preserving conclusion. Theorem 2 uses Bartlett et al. (2006) classification calibration and Assumptions 1-2, which do not encode the desired conclusion. No noise rates are fitted and no fitted constants are renamed as predictions: SymPO's loss (Eq. 10) is a fixed sigmoid or ramp loss with beta = 0.1, not estimated from noisy labels. The self-citations that exist (Charoenphakdee et al. 2019 and Sugiyama et al. 2022, both with a present coauthor) supply background facts on symmetric-loss calibration and weak supervision; these facts are published and independent of the current asymmetric-noise claim, so they are not load-bearing. The genuine weakness is mathematical, not circular: Lemma 3 computes only marginal flip-symmetric rates, while the conditional per-pair rates after flipping can remain asymmetric unless p(a,b) = p(b,a) or epsilon_p = epsilon_n, so Proposition 1 is likely unproven. That is a proof error, not a definitional equivalence or a fitted-input-as-prediction, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two external theorems plus the paper's three stated assumptions. The unstated pair-symmetry assumption is the fragile one; it is not in Assumptions 1-3, is not satisfied by typical real preference datasets, and is required for the asymmetric-noise reduction. No new entities and no fitted free parameters are introduced.

assumptions (5)
  • domain assumption Preference labels are consistent with a true reward: 2p(a1≻a2)>1 iff rtrue(a1)>rtrue(a2) (Assumption 1).
    Connects observed preferences to the ground-truth reward; satisfied by the BT model but not empirically verified.
  • domain assumption The input distribution has full support: p(a1,a2)>0 for all pairs (Assumption 2).
    Turns in-expectation consistency into pointwise rank preservation.
  • domain assumption Axiom 3: at least two actions have different true rewards and πref covers all actions.
    Ensures there is room for policy improvement over the reference.
  • ad hoc to paper The ordered-pair marginal is symmetric: p(a1,a2)=p(a2,a1), so that random flipping produces per-input symmetric noise. This is unstated.
    Needed for Lemma 1 and Proposition 1; without it the per-input flip rates after flipping are unequal across classes, producing a counterexample.
  • standard math External theorems: symmetric losses are robust to input-independent symmetric noise (van Rooyen et al. 2015) and classification-calibrated losses are consistent (Bartlett et al. 2006).
    Foundational results the proofs invoke; accepted in the literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Symmetric Losses for Robust Policy Optimization with Noisy Preferences." pith.science (2026). https://pith.science/paper/6CLJL7MI

@misc{pith2026250524709,
  author       = {Pith},
  title        = {Pith review of: On Symmetric Losses for Robust Policy Optimization with Noisy Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CLJL7MI}},
  note         = {Machine review of arXiv:2505.24709}
}
read the original abstract

Optimizing policies based on human preferences is key to aligning language models with human intent. This work focuses on reward modeling, a core component in reinforcement learning from human feedback (RLHF), and offline preference optimization, such as direct preference optimization. Conventional approaches typically assume accurate annotations. However, real-world preference data often contains noise due to human errors or biases. We propose a principled framework for robust policy optimization under noisy preferences, viewing reward modeling as a classification problem. This allows us to leverage symmetric losses, known for their robustness to label noise in classification, leading to our Symmetric Preference Optimization (SymPO) method. We prove that symmetric losses enable successful policy optimization even under noisy labels, as the resulting reward remains rank-preserving -- a property sufficient for policy improvement. Experiments on synthetic and real-world tasks demonstrate the effectiveness of SymPO.

Figures

Figures reproduced from arXiv: 2505.24709 by the authors.

Figure 1
Figure 1. The symmetric losses are plot￾ted with a solid line, while the 0 − 1 loss and convex losses with a dashed line. Methodologically, this classification viewpoint provides a principled way to handle asymmetric noise and enables the development of a robust objective function for reward modeling. Specifically, it makes an inherent structure in preference data explicit: swapping the positions of in￾put pairs flips the lab… view at source ↗
Figure 2
Figure 2. We sample action pairs (a1, a2) from a 2D Gaussian and assign preference labels using y = sign(a1−a2). The class prior πp of the original data is 0.8. Asymmetric label noise is introduced with εp = 0.0 and εn = 0.5, producing noisy labels y˜. Figure (a) shows the action pairs before and after random flipping, along with class priors and noise rates. After noise is applied, the class prior becomes 0.5 and the overall… view at source ↗
Figure 3
Figure 3. The label is generated following the BT model, where the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting

    cs.LG 2025-08 reject novelty 5.0 of 10

    S-GRPO reweights GRPO advantages by an assumed noise level p to down-weight unbalanced groups, claiming improved and more noise-robust math reasoning training.

Reference graph

Works this paper leans on

44 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [35]

    Learning with symmetric label noise: The importance of being unhinged

    Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in neural information processing systems, 28, 2015. 11

    work page 2015

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447–4455. PMLR, 2024

    work page 2024

  4. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [4]

    Convexity, classification, and risk bounds

    Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138–156, 2006

    2006

  6. [5]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  7. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  8. [7]

    On symmetric losses for learning from corrupted labels

    Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning, pp. 961–970. PMLR, 2019

    work page 2019

Show all 44 references
  1. [8]

    Learning with instance-dependent label noise: A sample sieve approach

    Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. arXiv preprint arXiv:2010.02347, 2020

    2010 arXiv

  2. [9]

    Provably robust dpo: Aligning language models with noisy feedback

    Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024

    2024 arXiv

  3. [10]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023

    2023 arXiv

  4. [11]

    The mnist database of handwritten digit images for machine learning research

    Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012

    2012

  5. [12]

    Variance-based regularization with convex objectives

    John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. Journal of Machine Learning Research, 20(68):1–55, 2019

    2019

  6. [13]

    Learning models with uniform performance via distributionally robust optimization

    John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406, 2021

    2021

  7. [14]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  8. [15]

    Robust preference optimization through reward model distillation

    Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024

    2024 arXiv

  9. [16]

    Impact of preference noise on the alignment performance of generative language models

    Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2404.09824, 2024

    2024 arXiv

  10. [17]

    Robust loss functions under label noise for deep neural networks

    Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017. 10

    2017

  11. [18]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  12. [19]

    A composite likelihood approach to binary spatial data

    Patrick J Heagerty and Subhash R Lele. A composite likelihood approach to binary spatial data. Journal of the American Statistical Association, 93(443):1099–1111, 1998

    1998

  13. [20]

    Robust preference optimization with provable noise tolerance for llms

    Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for llms. arXiv preprint arXiv:2404.04102, 2024

    2024 arXiv

  14. [21]

    On the minimal supervision for training any binary classifier from only unlabeled data

    Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint arXiv:1808.10585, 2018

    2018 arXiv

  15. [22]

    Mitigating overfitting in super- vised classification from two unlabeled datasets: A consistent risk correction approach

    Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in super- vised classification from two unlabeled datasets: A consistent risk correction approach. In International Conference on Artificial Intelligence and Statistics, pp. 1115–1125. PMLR, 2020

    2020

  16. [23]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  17. [24]

    Linking losses for density ratio and class-probability estimation

    Aditya Menon and Cheng Soon Ong. Linking losses for density ratio and class-probability estimation. In International Conference on Machine Learning, pp. 304–313. PMLR, 2016

    2016

  18. [25]

    Learning from corrupted binary labels via class-probability estimation

    Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In International conference on machine learning, pp. 125–134. PMLR, 2015

    2015

  19. [26]

    A note on dpo with noisy preferences & relationship to ipo, 2023

    Eric Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023

    2023

  20. [27]

    Learning with noisy labels

    Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013

    2013

  21. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  22. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [30]

    Composite binary losses

    Mark D Reid and Robert C Williamson. Composite binary losses. The Journal of Machine Learning Research, 11:2387–2422, 2010

    2010

  24. [31]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  25. [32]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  26. [33]

    Machine learning from weak supervision: An empirical risk minimization approach

    Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, and Tomoya Sakai. Machine learning from weak supervision: An empirical risk minimization approach. MIT Press, 2022

    2022

  27. [34]

    Generalized preference optimization: A unified approach to offline alignment

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024

    2024 arXiv

  28. [36]

    Symmetric cross entropy for robust learning with noisy labels

    Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 322–330, 2019

    2019

  29. [37]

    Towards robust alignment of language models: Distributionally robustifying direct preference optimization

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.07880, 2024

    2024 arXiv

  30. [38]

    Repo: Relu-based preference optimization

    Junkang Wu, Kexin Huang, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Repo: Relu-based preference optimization. arXiv preprint arXiv:2503.07426, 2025

    2025

  31. [39]

    Fair classification with instance-dependent label noise

    Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. In Conference on Causal Learning and Reasoning, pp. 927–943. PMLR, 2022

    2022

  32. [40]

    Generalized cross entropy loss for training deep neural networks with noisy labels

    Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018

    2018

  33. [41]

    Slic- hf: Sequence likelihood calibration with human feedback

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023. 12 A Related Work A.1 Loss functions in Reward Modeling Direct Preference Optimization (DP...

    2023 arXiv

  34. [42]

    A" or "B

    uses a squared loss to learn policies without relying on the BT model, while [ 41] employs a hinge loss to achieve more efficient preference optimization. Among these works, Generalized Preference Optimization (GPO) [34] is most closely related to our research. GPO frames pref...

  35. [43]

    For at least two actions ai, aj ∈ Awith rtrue(ai) ̸= rtrue(aj)

  36. [44]

    Then, I+ and I− are non-empty

    πref (a) > 0 for all a ∈ A. Then, I+ and I− are non-empty. Proof. From the first condition, we have two actions ai, aj ∈ Awith rtrue(ai) ̸= rtrue(aj). Let assume rtrue(ai) > rtrue(aj), then from the rank-preservingness of r with respect to rtrue, we have r(ai) > r(aj). Also, f...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.