Pith. sign in

REVIEW 3 major objections 5 minor 75 references

Learning from Noisy Labels via Conditional Distributionally Robust Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that noisy-label learning can be formulated as conditional distributionally robust optimization, solved in closed form per data point, and that the resulting pseudo-labeling algorithm outperforms competing methods on…

desk verdict Genuinely new CDRO formulation for noisy-label posteriors with strong empirical gains, but the robustness guarantee rests on an unverified reference posterior and the theory-practice link has gaps. read the letter →

arxiv 2411.17113 v1 pith:EJFAW3TU submitted 2024-11-26 cs.LG

Hui Guo , Grace Y. Yi , Boyu Wang This is my paper · ORCID
classification cs.LG
keywords noisylabelscrowdsourcingconditionaldistributionallyrobustoptimizationWassersteinambiguitysetpseudo-labelingtruelabelposteriorclassificationnoise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that learning from crowdsourced noisy labels can be made robust without needing an accurate estimate of the true label posterior. It reformulates the problem as conditional distributionally robust optimization: minimize the worst-case expected loss over a Wasserstein ball of radius $\epsilon$ around the estimated posterior $P_{y|x,\tilde y}$ for each instance and annotation vector. The paper derives the dual of this problem, solves the per-data-point robust risk in closed form, and uses that solution as a likelihood-ratio pseudo-labeling rule with an adaptively chosen Lagrange multiplier. If the central claim is correct, the method hedges against misspecification of the posterior rather than requiring the estimate to be right, and the reported results show consistent accuracy gains over competing approaches on synthetic and real crowdsourced benchmarks.

What carries the argument

The object that carries the argument is the conditional ambiguity set $\Gamma_\epsilon(P_{y|x,\tilde y})$, a Wasserstein ball of radius $\epsilon$ around the estimated true-label posterior for each $(x,\tilde y)$. Its dual (Proposition 2.1) turns the worst-case expectation into an infimum over a scalar Lagrange multiplier $\gamma$, which is what makes the robust risk solvable. Theorem 3.3 then identifies the optimal $\gamma$ with the sorted gap $\alpha_{(s^\star)}/\kappa^p$, so the algorithm can update its robustness level in a single step instead of solving a nested optimization at every epoch.

What would settle it

On a dataset whose clean labels are known, deliberately corrupt the noise-transition estimates (for instance by warming up on heavily memorized noisy labels), then measure the Wasserstein distance between the true posterior and the estimated reference; if accuracy falls below a non-robust baseline whenever that distance exceeds $\epsilon$, the claim that the ambiguity ball covers realistic misspecification is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the conditional distributionally robust risk has a tractable dual whose per-point solution is a likelihood-ratio test. In the binary case with a concave loss transform $T$, the optimal output for an instance is class $0$ or $1$ when the corresponding posterior probability exceeds $\varrho(\epsilon)+\varpi_1$, and the abstaining uniform value $1/2$ otherwise; a convex $T$ introduces an additional soft region solved by a one-dimensional equation. For multi-class problems, the optimal action puts mass $1/k_0$ on the $k_0$ largest posterior probabilities and zero elsewhere. Because the optimal Lagrange multiplier is available in closed form as $\gamma^\star_\psi=\alpha_{(s^\star)}/\kappa^p$, the empirical robust risk reduces to the nominal loss plus a penalty that prevents overconfident predictions. The paper builds the AdaptCDRP algorithm on these facts, using confidently pseudo-labeled points to form a pseudo-empirical distribution as the reference for CDRO.

Load-bearing premise

The argument assumes that for every instance the true conditional distribution of the label lies inside the epsilon-Wasserstein ball around the estimated posterior; if the warm-up phase has memorized the noisy labels, the transition estimates are biased and may put the true posterior outside any reasonable ball.

Editorial extensions

If this is right

  • If the central claim holds, practitioners can train on noisy crowdsourced labels without first obtaining high-quality posterior estimates, because the CDRO objective hedges against errors in those estimates.
  • The likelihood-ratio pseudo-label rule only trains on instances where the top posterior dominates the second by a threshold, so the method should remain stable in high-noise regimes where argmax pseudo-labeling degrades.
  • The closed-form optimal Lagrange multiplier provides a one-step, principled update for the robustness level, avoiding expensive bi-level optimization during training.
  • The generalization bounds imply the empirical robust risk tracks its population counterpart at rate $O(n^{-1/2})$, so the reported gains are not explained by overfitting the training set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the threshold $C_1$ in the pseudo-label rule is an explicit precision-recall dial; tuning it per class or per annotator could trade coverage against pseudo-label accuracy without changing the CDRO derivation.
  • Editorial inference: because the closed-form robust risk adds a penalty on overconfident points, the method is likely compatible with logit-normalization or confidence-penalty tricks, which may further stabilize very high noise rates.
  • Editorial inference: the algorithm's reference distribution is modular, so substituting a better transition estimator should improve accuracy; the paper reports partial evidence for this, and a systematic comparison across estimators would be a direct test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a conditional distributionally robust optimization (CDRO) framework for learning from noisy crowdsourced labels. The true-label posterior P_{y|x,ey} is treated as an uncertain distribution inside a Wasserstein ambiguity ball centered at an estimated reference distribution. The authors derive a dual formulation, an analytical per-instance solution, a closed-form expression for the empirical robust risk, and an adaptive update for the Lagrange multiplier. These ingredients lead to Algorithm 1 (AdaptCDRP), which uses likelihood-ratio-based robust pseudo-labels and a pseudo-empirical reference distribution. Experiments on CIFAR-10, CIFAR-100, CIFAR-10N, CIFAR-100N, LabelMe, and Animal-10N report consistent improvements over a broad set of baselines.

Significance. If the proposed framework is fully validated, it would be a useful contribution to noisy-label learning: it gives a clean DRO treatment of posterior misspecification, provides closed-form dual quantities that are computationally attractive, and demonstrates strong empirical gains, especially at high noise. The paper ships detailed appendix proofs, code, and extensive experiments including real human annotations and ablation studies, which are strengths. However, the central theoretical guarantee is conditional on the estimated reference distribution being close to the true posterior, and the paper does not verify this condition or quantify the effect of reference misspecification. The link between the concave-case pseudo-label theory and the convex cross-entropy implementation is also not fully established. These issues do not invalidate the empirical findings, but they do limit the force of the paper's robustness claims.

major comments (3)
  1. [§2.1, Eq. (1); §3.3] The DRO guarantee in Eq. (1) controls only deviations of Q from the reference P̂_{y|x,ey}. Theorem 2.2 and Corollary 2.3 quantify the difference between empirical and population robust risk for a fixed center, but they say nothing about what happens when the true posterior P* lies outside the ε-Wasserstein ball. In Section 3.3, P̂ is built from warm-up model predictions and frequency-counted transition estimates on the smallest-loss subset; Figures 6–7 show that the warm-up model can overfit strongly at high noise, so D*_0 may contain mislabeled examples, and τ̂_j(ey) may be biased. For the small ε values used in the experiments (e.g., 0.001–0.06 for CIFAR-10), the paper provides no evidence that P* ∈ Γ_ε(P̂). Without such a coverage check, the worst-case risk in Eq. (1) is not an upper bound for the actual risk, and the theoretical claim of robustness under misspecified true-label posteriors is not supported. I recommend adding controlled experiments that measure or estimate the distance between P̂ and the true posterior, or a sensitivity analysis showing how accuracy degrades as the center is perturbed.
  2. [§3.1, Theorem 3.1 and Remark 3.1; §3.3, Algorithm 1; §4] The hard-threshold likelihood-ratio form in Remark 3.1 is derived specifically for a concave function T. For a convex T, Theorem 3.1 instead gives intermediate optimal values t*_0 and t*_1 in the middle probability regime, not a hard 0/1 pseudo-label. The experiments use the cross-entropy loss, i.e., T(t) = -log t, which is convex (Remark 3.5), yet Algorithm 1 line 6 and the surrounding text apply the hard threshold C from Remark 3.1 without stating which case of Theorem 3.1 is being used or deriving a convex analogue. This leaves a gap between the analytic per-instance solution and the deployed pseudo-labeling rule. If the hard threshold is intended as a heuristic rather than a direct consequence of Theorem 3.1, the paper should say so explicitly and justify it empirically.
  3. [§3.3, Algorithm 1 line 13] The adaptive update γ_t = γ0,t - (1/λ){ε^p - E_{P*_{t,\iota}} c^p(y',Y)} is the unconstrained minimizer of a quadratic surrogate, not the optimal Lagrange multiplier of the empirical robust risk in Eq. (5). The paper states that this is a 'one-step update method' and 'a principled balance', but no convergence or optimality guarantee is provided for this update, and it is not shown that repeated updates approach the γ*_ψ characterized in Theorem 3.3. Since the robust loss and the resulting classifier update depend on γ, this heuristic is load-bearing. Please either analyze the surrogate update (e.g., under what conditions it preserves the minimization direction of Theorem 3.3) or clearly label it as an approximation and report sensitivity to λ.
minor comments (5)
  1. [Corollary 2.3] The first displayed inequality in Corollary 2.3 reads 'Rϵ(ψ̂_ϵ,n; P) ≤ Rϵ(ψ̂_ϵ,n; P)', with the same quantity on both sides; this is likely a typo and should be corrected (for example, to relate bRϵ and Rϵ).
  2. [§3.3] The paper says the small-loss subset size m is '1 minus the estimated noise rate', but it does not specify how the noise rate is estimated before the transition-matrix estimation. Please provide the estimation procedure or the chosen m/n values used in the experiments.
  3. [§3.3, Algorithm 1; §4] The hyperparameters C (pseudo-label threshold), λ (Lagrange multiplier step), and κ (label metric scale) are not reported in the main text or in Appendix B.1. Since these directly control pseudo-label selection and the γ update, please report their values and, if possible, their sensitivity.
  4. [Appendix A.8] In the opening of Appendix A.8, the notation 'For i ∈ [K] and j ∈ [K]' should presumably be 'i ∈ [n] and j ∈ [K]', since the data index ranges over the n samples.
  5. [Appendix B.1] There are small typos in the appendix, including 'Algorihtm 2' and 'Lines 11-12 f Algorithm 1'; these should be cleaned up before publication.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the CDRO derivation is self-contained, with only minor non-load-bearing self-citation in baseline sourcing.

full rationale

The central derivation chain is mathematically self-contained: Proposition 2.1 follows from finite-dimensional LP duality; Theorems 3.1 and 3.3 are closed-form solutions to the stated dual/empirical risk problems; and Algorithm 1 applies those results. The reference posterior P_{y|x,ey} is treated as an input, and the theorems condition on it rather than assuming the conclusion. The likelihood-ratio pseudo-labeling rule in Remark 3.1 is the derived optimal action, not a separately fitted quantity renamed as a prediction. Experimental support includes external benchmarks (CIFAR-10N, CIFAR-100N, LabelMe, Animal-10N) and many non-author baselines. There are minor self-citations: Table 1 baseline numbers for several methods are sourced from the authors' prior paper [10], and the warm-up/transition-estimation recipe follows [10, 46], but these are reproducibility and implementation choices, not load-bearing for the theoretical claims. The paper's own limitation statement, 'Our development here does not focus on precisely estimating the noise transition matrix or the true label posterior,' accurately describes the scope: the DRO guarantee controls deviations from the given reference distribution and does not by itself certify that the true posterior lies inside the epsilon-ball, so center misspecification remains an unverified assumption rather than a circular step. Accordingly, no circular reduction is exhibited in the derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central derivation is self-contained given standard LP duality, but the algorithm depends on several unverified modeling choices: the epsilon-ball coverage of the true posterior, the clean-subset assumption for transition estimation, and heuristic updates for C and gamma. No new physical entities are introduced.

free parameters (5)
  • epsilon (robustness radius) = chosen by validation in (0, 0.1) for CIFAR-10 and (0, 0.01) for CIFAR-100; exact values not reported
    Defines the Wasserstein ambiguity set; no analytical selection rule, tuned per dataset.
  • kappa (label metric scale) = not reported; only ratio rho = epsilon^p / kappa^p matters
    Cost c(y,y') = kappa * 1(y != y'); absorbed into rho; never specified.
  • pseudo-label threshold C = not reported
    User input in Algorithm 1; implementation does not set it to the analytic threshold C1 in Remark 3.1, so it is effectively tuned.
  • lambda (Lagrange multiplier learning rate) = not reported
    Controls the one-step gamma update in Algorithm 1; no value or sensitivity analysis.
  • small-loss subset ratio m/n = set to 1 minus estimated noise rate; noise rate estimation method not specified
    Used to estimate transition probabilities; the choice affects posterior quality.
assumptions (5)
  • domain assumption The estimated posterior lies within an epsilon-Wasserstein ball of the true posterior for each (x, ey).
    Central to Eq (1)-(2); if violated, the worst-case ball may exclude the true distribution.
  • domain assumption The loss is bounded in [0, M] and Lipschitz in both arguments with respect to the label metric c.
    Required for Theorem 2.2; cross-entropy is clipped to satisfy boundedness.
  • domain assumption Annotator noise transition probabilities can be consistently estimated from the small-loss subset selected by the warm-up model.
    Used in Section 3.3 bP_j proportional to psi_j(x) * btau_j(ey); no guarantee that the selected subset is clean.
  • standard math Linear programming strong duality and standard measurability conditions hold.
    Used in the proofs of Proposition 2.1 and Remark 2.3; standard but not machine-checked.
  • ad hoc to paper The one-step gamma update in Algorithm 1 minimizes a quadratic surrogate and preserves convergence.
    Line 13 uses gamma_t = gamma_0 - (1/lambda) * (epsilon^p - E c^p); this is not the exact minimizer from Theorem 3.3 and no convergence proof is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Noisy Labels via Conditional Distributionally Robust Optimization." pith.science (2026). https://pith.science/paper/EJFAW3TU

@misc{pith2026241117113,
  author       = {Pith},
  title        = {Pith review of: Learning from Noisy Labels via Conditional Distributionally Robust Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJFAW3TU}},
  note         = {Machine review of arXiv:2411.17113}
}
read the original abstract

While crowdsourcing has emerged as a practical solution for labeling large datasets, it presents a significant challenge in learning accurate models due to noisy labels from annotators with varying levels of expertise. Existing methods typically estimate the true label posterior, conditioned on the instance and noisy annotations, to infer true labels or adjust loss functions. These estimates, however, often overlook potential misspecification in the true label posterior, which can degrade model performances, especially in high-noise scenarios. To address this issue, we investigate learning from noisy annotations with an estimated true label posterior through the framework of conditional distributionally robust optimization (CDRO). We propose formulating the problem as minimizing the worst-case risk within a distance-based ambiguity set centered around a reference distribution. By examining the strong duality of the formulation, we derive upper bounds for the worst-case risk and develop an analytical solution for the dual robust risk for each data point. This leads to a novel robust pseudo-labeling algorithm that leverages the likelihood ratio test to construct a pseudo-empirical distribution, providing a robust reference probability distribution in CDRO. Moreover, to devise an efficient algorithm for CDRO, we derive a closed-form expression for the empirical robust risk and the optimal Lagrange multiplier of the dual problem, facilitating a principled balance between robustness and model fitting. Our experimental results on both synthetic and real-world datasets demonstrate the superiority of our method.

Figures

Figures reproduced from arXiv: 2411.17113 by the authors.

Figure 2
Figure 2. Average accuracy on the CIFAR-10 and CIFAR-100 datasets (R = 5) for different ϵ values. 5 10 30 50 100 Number of Annotators 74 76 78 80 82 84 86 88 Test Accuracy CE(Clean) CE(MV) CE(EM) Co-teaching Co-teaching+ CoDis LogitClip DoctorNet MBEM CrowdLayer TraceReg Max-MIG CoNAL CCC AdaptCDRP (Ours) (a) IDN-LOW 5 10 30 50 100 Number of Annotators 65 70 75 80 85 Test Accuracy CE(Clean) CE(MV) CE(EM) Co-teaching Co-teachi… view at source ↗
Figure 1
Figure 1. Average test accuracy on the CIFAR-10 dataset with varying numbers of annotators. The [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 3
Figure 3. Average test accuracy on the CIFAR-100 dataset with varying numbers of annotators. The [PITH_FULL_IMAGE:figures/full_fig_p036_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average accuracy of robust pseudo-labels on the CIFAR-10 and CIFAR-100 datasets [PITH_FULL_IMAGE:figures/full_fig_p037_4.png]
Figure 5
Figure 5. Figure 5: Average accuracy of robust pseudo-labels on the CIFAR-10 dataset with varying number of [PITH_FULL_IMAGE:figures/full_fig_p038_5.png]
Figure 6
Figure 6. Figure 6: Average test accuracy on learning the CIFAR-10 dataset ( [PITH_FULL_IMAGE:figures/full_fig_p038_6.png]
Figure 7
Figure 7. Figure 7: Average test accuracy on learning the CIFAR-100 dataset ( [PITH_FULL_IMAGE:figures/full_fig_p039_7.png]
Figure 8
Figure 8. Figure 8: Average test accuracies for learning the CIFAR-10 and CIFAR-100 datasets with varying [PITH_FULL_IMAGE:figures/full_fig_p039_8.png]
Figure 9
Figure 9. Figure 9: Average accuracy of robust pseudo-labels on the CIFAR-10 dataset ( [PITH_FULL_IMAGE:figures/full_fig_p040_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 64 canonical work pages

  1. [1]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016

    2016

  2. [2]

    Deep learning in neural networks: An overview

    Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015

    work page 2015

  3. [3]

    A closer look at memorization in deep networks

    Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 233–242, 2017

    work page 2017

  4. [4]

    Measurement Error in Nonlinear Models: A Modern Perspective

    Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. Measurement Error in Nonlinear Models: A Modern Perspective. Chapman and Hall/CRC, 2006

    work page 2006

  5. [5]

    Statistical Analysis with Measurement Error or Misclassification

    Grace Y Yi. Statistical Analysis with Measurement Error or Misclassification. Springer, 2017

    work page 2017

  6. [6]

    Handbook of Measurement Error Models

    Grace Y Yi, Aurore Delaigle, and Paul Gustafson. Handbook of Measurement Error Models. CRC Press, 2021

    work page 2021

  7. [7]

    Maximum likelihood estimation of observer error-rates using the em algorithm

    Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C, 28(1):20–28, 1979

    work page 1979

  8. [8]

    Whose vote should count more: Optimal integration of labels from labelers of unknown expertise

    Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, volume 22, pages 2035–2043, 2009

    work page 2009

Show all 75 references
  1. [9]

    Learning from noisy singly-labeled data

    Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577, 2017

    2017 arXiv

  2. [10]

    Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model

    Hui Guo, Boyu Wang, and Grace Y Yi. Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model. In Advances in Neural Information Processing Systems , volume 36, pages 347–386, 2023

    2023

  3. [11]

    Max-mig: an information theoretic approach for joint learning from crowds

    Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-mig: an information theoretic approach for joint learning from crowds. arXiv preprint arXiv:1905.13436, 2019

    1905 arXiv

  4. [12]

    Learning from noisy labels by regularized estimation of annotator confusion

    Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11244–11253, 2019

    2019

  5. [13]

    Pre-train your loss: Easy bayesian transfer learning with informative priors

    Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. In Advances in Neural Information Processing Systems, volume 35, pages 27706–27715, 2022

    2022

  6. [14]

    Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems, volume 32, pages 6838–6849, 2019

    Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems, volume 32, pages 6838–6849, 2019

    2019

  7. [15]

    Adversarial interpretation of bayesian inference

    Hisham Husain and Jeremias Knoblauch. Adversarial interpretation of bayesian inference. In Proceedings of The 33rd International Conference on Algorithmic Learning Theory , volume 167, pages 553–572. Proceedings of Machine Learning Research, 2022

    2022

  8. [16]

    Conditional distributionally robust functionals

    Alexander Shapiro and Alois Pichler. Conditional distributionally robust functionals. Operations Research, 2023

    2023

  9. [17]

    Quantifying distributional model risk via optimal transport

    Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathe- matics of Operations Research, 44(2):565–600, 2019

    2019

  10. [18]

    Distributionally robust stochastic optimization with Wasserstein distance

    Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research, 48(2):603–655, 2023. 11

    2023

  11. [19]

    Joint optimization framework for learning with noisy labels

    Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5552–5560, 2018

    2018

  12. [20]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience, 2006

    2006

  13. [21]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  14. [22]

    Learning with noisy labels revisited: A study using real-world human annotations

    Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022

    2022

  15. [23]

    Learning supervised topic models for classification and regression from crowds

    Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C Pereira. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2409–2422, 2017

    2017

  16. [24]

    Russell, and Jenny Yuen

    Antonio Torralba, Bryan C. Russell, and Jenny Yuen. Labelme: Online image annotation and applications. Proceedings of the IEEE, 98(8):1467–1484, 2010

    2010

  17. [25]

    SELFIE: Refurbishing unclean samples for robust deep learning

    Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 5907–5915, 2019

    2019

  18. [26]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  19. [27]

    Deep learning from crowds

    Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 1611–1618, 2018

    2018

  20. [28]

    Very deep convolutional networks for large-scale image recogni- tion

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015

    2015

  21. [29]

    Part-dependent label noise: Towards instance-dependent label noise

    Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In Advances in Neural Information Processing Systems, volume 33, pages 7597–7610, 2020

    2020

  22. [30]

    Co-teaching: Robust training of deep neural networks with extremely noisy labels

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, volume 31, pages 8527–8537, 2018

    2018

  23. [31]

    How does disagreement help generalization against label corruption? In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 7164–7173, 2019

    Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 7164–7173, 2019

    2019

  24. [32]

    Combating noisy labels with sample selection by mining high-discrepancy examples

    Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833–1843, 2023

    2023

  25. [33]

    Mitigating memorization of noisy labels by clipping the model prediction

    Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 36868–36886, 2023

    2023

  26. [34]

    Who said what: Modeling individual labelers improves classification

    Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. InProceedings of the AAAI conference on artificial intelligence, volume 32, pages 3109–3118, 2018

    2018

  27. [35]

    Learning from crowds by modeling common confusions

    Zhendong Chu, Jing Ma, and Hongning Wang. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5832–5840, 2021

    2021

  28. [36]

    Coupled confusion correction: Learning from crowds with sparse annotations

    Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. Coupled confusion correction: Learning from crowds with sparse annotations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16732–16740, 2024

    2024

  29. [37]

    Linear and Nonlinear Programming, volume 2

    David G Luenberger and Yinyu Ye. Linear and Nonlinear Programming, volume 2. Springer, 1984

    1984

  30. [38]

    Applied Mathematical Programming

    Stephen P Bradley, Arnoldo C Hax, and Thomas L Magnanti. Applied Mathematical Programming. Addison-Wesley Publishing Company, 1977. 12

    1977

  31. [39]

    High-Dimensional Statistics: A Non-Asymptotic Viewpoint

    Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019

    2019

  32. [40]

    Minimax statistical learning with wasserstein distances

    Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. In Advances in Neural Information Processing Systems, volume 31, pages 2687–2696, 2018

    2018

  33. [41]

    The supremum and infimum

    John Hunter. The supremum and infimum. https://www.math.ucdavis.edu/~hunter/m125b/ch2. pdf

  34. [42]

    Foundations of Machine Learning

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2018

    2018

  35. [43]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  36. [44]

    Error rate bounds and iterative weighted majority voting for crowdsourcing

    Hongwei Li and Bin Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086, 2014

    2014 arXiv

  37. [45]

    Leverag- ing inter-rater agreement for classification in the presence of noisy labels

    Maria Sofia Bucarelli, Lucas Cassano, Federico Siciliano, Amin Mantrach, and Fabrizio Silvestri. Leverag- ing inter-rater agreement for classification in the presence of noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3...

    2023

  38. [46]

    Error-bounded correction of noisy labels

    Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 11447–11457, 2020

    2020

  39. [47]

    parameter

    Shahana Ibrahim, Tri Nguyen, and Xiao Fu. Deep learning from crowdsourced labels: Coupled cross- entropy minimization, identifiability, and regularization. In The Eleventh International Conference on Learning Representations, 2023. 13 SUPPLEMENTARY MATERIAL A Technical Details...

    2023

  40. [48]

    We complete the proof by considering the following two cases

    are the arguments of minψ1∈[0, 1 2 ] inf γ≥0 g(γ; ψ1) and minψ2∈[ 1 2 ,1] inf γ≥0 g(γ; ψ2), respectively. We complete the proof by considering the following two cases. Case 1: ψ1 ∈ [0, 1 2 ]. In this case, T (ψ1) ≥ T( 1 2 ) ≥ T(1 − ψ1) ≥ T(1 − ψ1) − γκp. Let γ0 = T (ψ1) − T(1 ...

  41. [49]

    If g(γ∗ 1/2; 1 2 ) ≤ g(γ∗ 0 ; 0), i.e., P0 ≤ ϱ(ϵ)+ T (0)−T (1/2) T (0)−T (1) , the optimal value for ψ1 in Case 1 is ψ∗ 1 = 1 2 with γ∗ ψ∗ 1 = 0; otherwise, the optimal value is ψ∗ 1 = 0 with γ∗ ψ∗ 1 = T (0)−T (1) κp by (A22). Summarizing the discussion in Step 2.1, we obtain ...

  42. [50]

    = (P1 + ϱ(ϵ))T (0) + (P0 − ϱ(ϵ))T (1); (ii) otherwise, ψ∗ 1 = 1 2 and γ∗ ψ∗ 1 = 0, yielding g(γ∗ ψ∗ 1 ; ψ∗

  43. [51]

    Step 2.2: Assume T is convex

    = T ( 1 2 ). Step 2.2: Assume T is convex. Then by twice differentiability of T , T ′′(ψ1) ≥ 0 for ψ1 ∈ [0, 1], leading to T ′(0) ≤ T′(1) < 0, and hence P (2) 0 ≤ P (1) 0 . Additionally, g′′ ψ1 (γ∗ ψ1 ; ψ1) = ( P1 + ϱ(ϵ))T ′′(ψ1) + (P0 − ϱ(ϵ))T ′′(1 − ψ1) ≥ 0, and thus, g′ ψ1 ...

  44. [52]

    Thus, ψ∗ 1 = ψ⋄ 1 with γ∗ ψ∗ 1 = T (ψ⋄ 1 )−T (1−ψ⋄ 1 ) κp by (A22)

    + (P0 − ϱ(ϵ))T (1 − ψ⋄ 1). Thus, ψ∗ 1 = ψ⋄ 1 with γ∗ ψ∗ 1 = T (ψ⋄ 1 )−T (1−ψ⋄ 1 ) κp by (A22). Case 2: ψ2 ∈ [ 1 2 , 1]. In this case, we set ψ2 ≜ 1 − ψ2, yielding ψ2 ∈ [0, 1 2 ], and the objective function g(γ; ψ2) defined in (A17) can be written as g(γ; ψ2) =γϵp + P0 max{T (1...

  45. [53]

    25 • Step 2: If P1 ≥ ϱ(ϵ), then, by (A22), γ∗ ψ∗ 2 is set as γ∗ ψ ∗ 2 = T (ψ ∗ 2 )−T (1−ψ ∗ 2 ) κp = T (1−ψ∗ 2 )−T (ψ∗ 2 ) κp

    = g(0; 1 2 ) = T ( 1 2 ). 25 • Step 2: If P1 ≥ ϱ(ϵ), then, by (A22), γ∗ ψ∗ 2 is set as γ∗ ψ ∗ 2 = T (ψ ∗ 2 )−T (1−ψ ∗ 2 ) κp = T (1−ψ∗ 2 )−T (ψ∗ 2 ) κp . • Step 2.1: Assume T is concave. We can directly derive the following result from the summary in Step 2.1 of Case 1. – If P...

  46. [54]

    – If P1 > ϱ(ϵ) + T (0)−T (1/2) T (0)−T (1) , then ψ ∗ 2 = 0 and γ∗ ψ∗ 2 = T (0)−T (1) κp

    = g(0; 1 2 ) = T ( 1 2 ). – If P1 > ϱ(ϵ) + T (0)−T (1/2) T (0)−T (1) , then ψ ∗ 2 = 0 and γ∗ ψ∗ 2 = T (0)−T (1) κp . Hence, ψ∗ 2 = 1−ψ ∗ 2 = 1, and g(γ∗ ψ∗ 2 ; ψ∗

  47. [55]

    • Step 2.2: Assume T is convex

    = g( T (0)−T (1) κp ; 1) = (P0 +ϱ(ϵ))T (0)+( P1 −ϱ(ϵ))T (1). • Step 2.2: Assume T is convex. From the results on ψ∗ 1 and γ∗ ψ∗ 1 in Step 2.2 of Case 1, we obtain the following conclusion. – If P1 ≥ ϱ(ϵ)+ T ′(0) T ′(1)+T ′(0), then ψ ∗ 2 = 0 and γ∗ ψ∗ 2 = T (0)−T (1) κp . Henc...

  48. [56]

    – If ϱ(ϵ) ≤ P1 ≤ ϱ(ϵ) + 1 2, then ψ ∗ 2 = 1 2 and γ∗ ψ∗ 2 = 0

    = g( T (0)−T (1) κp ; 1) = (P0 + ϱ(ϵ))T (0) + (P1 − ϱ(ϵ))T (1). – If ϱ(ϵ) ≤ P1 ≤ ϱ(ϵ) + 1 2, then ψ ∗ 2 = 1 2 and γ∗ ψ∗ 2 = 0. Hence, ψ∗ 2 = 1 − ψ ∗ 2 = 1 2, and g(γ∗ ψ∗ 2 ; ψ∗

  49. [57]

    = g(0; 1 2 ) = T ( 1 2 ). – If ϱ(ϵ) + 1 2 < P1 < ϱ(ϵ) + T ′(0) T ′(1)+T ′(0), then ψ ∗ 2 = ψ ⋄ 2 and γ∗ ψ∗ 2 = T (ψ ⋄ 2 )−T (1−ψ ⋄ 2 ) κp , where ψ ⋄ 2 is the unique solution to (P0 + ϱ(ϵ))T ′(ψ2) − (P1 − ϱ(ϵ))T ′(1 − ψ2) = 0 on [0, 1 2 ]. Hence, ψ∗ 2 = ψ⋄ 2 and γ∗ ψ∗ 2 = T (1...

  50. [58]

    = (P0 + ϱ(ϵ))T (1 − ψ⋄

  51. [59]

    In summary, we present the derived results in Tables 3 and 4 for the scenarios whereT is concave and convex, respectively

    + (P1 − ϱ(ϵ))T (ψ⋄ 2). In summary, we present the derived results in Tables 3 and 4 for the scenarios whereT is concave and convex, respectively. ψ∗ j γ∗ ψ∗ j robust risk g(γ∗ ψ∗ j ; ψ∗ j ) Case 1 P0 ≥ ϱ(ϵ) + T (0)−T (1/2) T (0)−T (1) ψ∗ 1 = 0 γ∗ ψ∗ 1 = T (0)−T (1) κp (P1 + ϱ(...

  52. [60]

    Update by Disagreement

    with synthetic annotations, and on four real-world datasets with human annotations: CIFAR- 10N, CIFAR-100N [22], LabelMe [ 23, 24], and Animal-10N [ 25]. CIFAR-10 has 10 classes of 32 × 32 × 3 color images, with 50,000 training images and 10,000 test images; CIFAR-10N provides...

  53. [61]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect our contributions and scope. Guidelines: • Th...

  54. [62]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work performed by the authors in the conclusion. Guidelines: • The answer NA means that the paper has no limitation...

  55. [63]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 41 Answer: [Yes] Justification: We provide the assumptions in the main text, and the proofs can be found in the appendix....

  56. [64]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  57. [65]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 42 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We upload our code and u...

  58. [66]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide all the details an...

  59. [67]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We provide the error bars. Guidelines: • The answe...

  60. [68]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the training...

  61. [69]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper conform, in every respect, with the NeurIPS...

  62. [70]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: It’s not appropriate for the scope and focus of our paper, and we don’t see any direct negative social imp...

  63. [71]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our model doesn...

  64. [72]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the orig...

  65. [73]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide the data generation details. Guidelines: • The answer NA means that the paper does not release new assets...

  66. [74]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  67. [75]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.