Pith. sign in

REVIEW 4 major objections 3 minor 67 references

Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that replacing the empirical distribution with a reweighted distribution inside an f-divergence ball lets a model forget mislabeled samples, and proves the true labels are the unique optimum under uniform noise below 50%.

desk verdict Fresh min-min relaxation with strong noisy-label gains, but the main theorem is population-level; the empirical objective can't recover true labels for observed corrupt samples, so the paper overclaims. read the letter →

arxiv 2508.06622 v1 pith:RX6V4PDI submitted 2025-08-08 cs.LG stat.ML

Jeremiah Birrell , Reza Ebrahimi This is my paper · ORCID
classification cs.LGstat.ML
keywords noisylabelsf-divergencerelaxation-optimizationsamplereweightingadversarialtrainingconvexdualitylabelnoisecross-entropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes training on a reweighted version of the empirical data distribution, where the reweighting is allowed to move inside a small f-divergence ball. Because the ball is minimized over rather than maximized over, the optimizer can downweight or discard samples with large loss — the ones most likely to carry wrong labels — without ever being told which labels are bad. Under a uniform label-noise model with noise rate below 1/2 and a model family that can represent the true labeling function, the paper proves the true labels are the unique minimizer of this relaxed objective. Convex duality turns the relaxation into a modified loss plus a one- or two-dimensional concave maximization, so the extra cost over plain cross-entropy is small. On CIFAR-10/100, WebVision, ImageNet, and human-annotated CIFAR-100N, the method reports accuracy above leading robust losses.

What carries the argument

The central object is the f-divergence ball $\{Q : D_f(Q\|P_n)\le \delta\}$, where $D_f(Q\|P)=E_P[f(dQ/dP)]$ and $f$ is convex with $f(1)=0$ (e.g., KL). Replacing the empirical distribution $P_n$ by any $Q$ in this ball reweights the training samples without creating new points, so 'forgetting' a sample means assigning it weight near zero. Convex duality (Theorem 3.2) converts the inner minimization over $Q$ into a dual objective involving the Legendre transform $f^*$ and two adversarial scalar parameters $\lambda,\rho$; in the KL case only $\lambda$ remains and the gradient of the dual objective is the weighted average $\frac{1}{B}\sum_i w_i \nabla_\theta L_\theta(x_i,y_i)$ with weights $w_

What would settle it

Build a two-point input space with $P_X$ uniform, true labels $A,B$, and instance-dependent transition $T^{x_1}_{A,A}=0.6$, $T^{x_2}_{B,B}=1.0$ (overall noise 20%). Set $\delta$ as in Theorem 3.1 with $r=0.2$ for KL divergence and solve $\inf_Q E_Q[L_\theta]$; if the minimizer over $\theta$ predicts $B$ on $x_1$ with positive probability, or if training with ANTIDOTE on this distribution fails to recover $h^*$, the uniform-noise assumption is doing the load-bearing work in the uniqueness theorem.

Watch

Extended reading notes

Core claim

Define the noisy training distribution $P_T$ as the true distribution $P_*$ whose labels are flipped by a sample-dependent transition matrix $T^x$. Instead of minimizing the expected loss over $P_T$ directly, the paper minimizes $\inf_{\theta} \inf_{Q: D_f(Q\|P_T)\le \delta} E_Q[L_\theta]$: the inner minimization searches over distributions $Q$ that reweight the training samples within a divergence ball of size $\delta$. The paper's main theoretical result, Theorem 3.1, states that if every sample has the same probability $1-r$ of keeping its true label with $r<1/2$, the model family contains the true labeling map, and the loss vanishes exactly at the true label, then every minimizer $\tilde

Load-bearing premise

The uniqueness proof assumes the noise rate is uniform across all samples with $r<1/2$, and the training procedure is assumed to solve the exact relaxed problem on the full distribution even though it alternates a per-minibatch scalar maximization with one SGD step on the network.

Editorial extensions

If this is right

  • If the theorem holds, practitioners do not need clean labels, transition-matrix estimates, or label correction to remove the effect of mislabeled samples; reweighting inside a divergence ball suffices.
  • The dual reformulation keeps the method's runtime close to standard cross-entropy — the paper reports less than 25 seconds difference on CIFAR-10/100 — so the forgetting behavior comes at negligible overhead.
  • The $\alpha$-divergence variant exhibits an explicit forgetting threshold: samples with $L_\theta(x_i,y_i) > -\rho$ receive no gradient, which the histograms show separates noisy from clean samples as training proceeds.
  • Because the method does not need per-sample noise rates, it applies to symmetric, asymmetric, human-annotation, and real-world noise; the reported accuracy exceeds the leading comparable losses on all five datasets tested.
  • When combined with add-ons such as semi-supervised learning, dual networks, or MixUp on CIFAR-100N, the gain over the respective baselines is preserved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: the per-sample weight $w_i$ (or the hard threshold in the $\alpha$ case) is a ready-made noisy-sample detector, so ANTIDOTE could be paired with an explicit sample-cleaning pass or used to estimate the noise rate per epoch.
  • Since the relaxation only reweights samples, the same objective should carry over to regression with corrupted targets — but the uniqueness proof's loss condition ($L(p,y)=0$ iff $p=y$) and the finite-label setting would need to be rethought for continuous $Y$.
  • The empirical success on instance-dependent noise (CIFAR-100N, WebVision) suggests the uniform-noise condition in Theorem 3.1 is sufficient rather than necessary; testing whether the uniqueness result survives a more general transition model with the same $\delta$ would sharpen the theory.
  • The penalized variant that optimizes $r_{\max}$ against $\delta(r_{\max})$ effectively adapts the forgetting budget to the data; this could be used as an online noise-rate estimator, though the paper only presents it as a hyperparameter-reduction device.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes ANTIDOTE, a family of loss objectives for learning with noisy labels. The core idea is to replace the empirical distribution used in supervised learning with an inf-optimization over an f-divergence neighborhood, allowing the method to downweight or 'forget' high-loss, likely mislabeled samples. The main theoretical results are: (i) Theorem 3.1, which states that under a uniform noise model with rate r<1/2, realizability, and a strictly convex f-divergence, the minimizer of the relaxed population objective over the divergence ball around the noisy distribution PT recovers the true labeling function h* P_X-almost surely; and (ii) Theorem 3.2, a convex-duality reformulation that turns the inf over Q into a low-dimensional concave maximization over dual variables. The paper then gives an implementation (Algorithm 1) as alternating per-minibatch updates of the network and the dual parameters, and reports extensive experiments on CIFAR-10, CIFAR-100, CIFAR-100N, WebVision, and ImageNet under symmetric, asymmetric, human-annotation, and real-world noise, finding consistent improvements over several baseline losses.

Significance. If the theoretical and empirical claims hold, ANTIDOTE is a practically interesting and computationally cheap robust-loss method. The paper's strengths are the nontrivial population-level uniqueness theorem, the clean duality derivation that leads to a low-dimensional convex inner problem, and a broad benchmark study with code provided. The empirical gains over strong baselines such as epsilon-softmax are substantial, and the runtime is close to standard cross-entropy. However, the theoretical contribution is currently disconnected from the implemented algorithm: Theorem 3.1 is proved for the population distribution PT, while the actual method optimizes an empirical objective with a minibatch SGD procedure. That gap, together with the uniform-noise assumption and the per-noise-level tuning of delta, means the headline theoretical guarantee does not directly justify the empirical results. The work is still valuable if these limitations are clearly stated and the theory is treated as a conditional population-level idealization.

major comments (4)
  1. [§3.2 vs §1.1, Theorem 3.1 / Eq. (2)] Theorem 3.1 is proved for the population distribution PT (Eq. 5), not for the empirical distribution Pn used in the actual objective (2). Because D_f(Q||Pn)=∞ unless Q≪Pn, any feasible Q in (2) is supported only on the observed pairs (x_i,y_i). If y_i is a corrupted label, the true pair (x_i,h*(x_i)) is not in the support, so the relaxation can zero out that sample but cannot place mass on its true label. Consequently, even an exact solution of the empirical inf-sup problem is not guaranteed to recover h*, and the sentence in §1.1 invoking Theorem 3.1 as a rigorous analysis of (2) overstates the result. I recommend either proving a finite-sample analogue with explicit dependence on n and the noise model, or clearly restricting the theoretical claim to the population distribution and presenting the finite-sample behavior as empirical only.
  2. [Algorithm 1; §3.3, Eqs. (10)–(11)] No convergence or equivalence result links the alternating per-minibatch procedure (exact inner maximization over λ, followed by one SGD step on θ) to the exact solution of the inf-sup problem in (10)/(11). Theorems 3.1 and 3.2 concern the exact joint problem over the full distribution; they do not imply that the stochastic minibatch algorithm converges to the claimed minimizer. Without a convergence or consistency statement, the theory does not justify the implemented optimization. The authors should either add such a result or explicitly identify Algorithm 1 as a heuristic whose behavior is documented empirically in Section 4.
  3. [§3.2, condition (6)] Theorem 3.1 relies on a uniform noise model: T^x_{h*(x),h*(x)}=1-r for all x, with r<1/2 (or the weaker condition T^x_{h*(x),y}<1-r). Many of the experiments, particularly CIFAR-100N and WebVision/ImageNet, involve instance-dependent or non-uniform real-world label noise, for which this condition is not satisfied. The paper reports strong results there, but Theorem 3.1 does not apply. I ask that the authors either restrict their theoretical discussion to the uniform case and explicitly separate the nonuniform empirical results, or provide an extension, e.g., a bound under nonuniform transition matrices, so that the theoretical and experimental scopes are aligned.
  4. [§3.5 / Appendix D.1, Eq. (15)] The theorem requires δ to equal the population value δ = r f(0)+(1-r)f(1/(1-r)), but in the experiments δ is tuned separately for each noise level (Appendix D.1 lists different δ for each rate), and Table 9 shows that accuracy is sensitive to δ. The adaptive variant in Appendix C.3 introduces an additional hyperparameter C. This does not invalidate the empirical study, but it means that the reported configurations generally operate outside the exact δ for which the uniqueness theorem is proved. The paper should state explicitly that the theoretical guarantee holds only for this particular δ, and clarify that tuned δ or C configurations are supported only empirically.
minor comments (3)
  1. [§4.1/Appendix G and §6.3/6.4] The manuscript contains stray formatting artifacts: duplicated Figure 1 and Figure 2 captions, a leftover 'Under review as a conference paper at ICLR 2024' header, and placeholders such as an unresolved citation for L2RW. These should be cleaned up before resubmission.
  2. [Appendix F] The text says results are 'relatively sensitive' to κ, yet later claims tuning is easy; Table 8 shows accuracy roughly between 88.8 and 89.2 across κ. Please make the intended claim about sensitivity and ease of tuning more precise and consistent.
  3. [Algorithm 1, lines 9 and 12] The notation 'argmax_{λ≥0}' suggests uniqueness; GKL is concave but not necessarily strictly concave, so ties are possible. The bisection method described in Appendix C.2 returns a maximizer, which is sufficient, but a remark noting this would avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular steps; Theorem 3.1 is a genuine conditional result and the empirical benchmark tuning is not a disguised prediction.

full rationale

The central theoretical claim, Theorem 3.1 / A.4, is not circular. Its assumptions (uniform noise rate r, realizability h_{θ*}=h*, zero-set loss, strict convexity of f) do not assert that h* minimizes the relaxed objective. The proof constructs P* and verifies D_f(P*||P_T)=δ exactly, then shows that any optimal Q must have Q_X=P_X and be supported on h*, forcing h_θ=h* P_X-a.s. Here δ is computed from r, not fitted to the target accuracy; the conclusion h*=h_θ is derived, not assumed. Theorem 3.2 / A.6 is a convex-duality reformulation proven via Slater's condition and the Gibbs variational formula; the paper explicitly acknowledges its similarity to the DRO result in [42], so it is not a renaming of a known result presented as new. The δ–r_max relation in Eq. (15) is a derived interpretation for the special case of forgetting a fraction with equal weights, and the paper explicitly treats δ as a tunable hyperparameter and only uses Eq. (15) as a starting point (Appendix D, Appendix F); choosing δ from the known synthetic noise ratio is standard hyperparameter selection, not a fitted parameter renamed as a prediction. The self-citations [38] and [46] are technical lemmas and related-work context, not the source of the noisy-label uniqueness claim; [38] supplies general compactness/LSC properties of f-divergences that are independent of the paper's target result. The population-versus-empirical support mismatch raised by the skeptic (Theorem 3.1 uses P_T, while Algorithm 1 optimizes over P_n) is a genuine scope/limitation issue, but it is not circularity: the theorem's assumptions do not encode its conclusion, and the empirical accuracies are held out. No load-bearing step reduces by construction to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard convex-duality results for f-divergences, a realizability and uniform-noise assumption in Theorem 3.1, and an unverified algorithmic convergence step. Three hyperparameters (δ, κ, C) are tuned to the data; the headline numbers in Table 1 use per-noise-level δ tuned with knowledge of the true noise ratio, which is the main practical Achilles heel.

free parameters (3)
  • δ (divergence neighborhood size) = e.g., 0.27 (CIFAR-10 20% sym), 0.57 (40% sym), 1.0 (60% sym), 1.62 (80% sym); 0.27 (WebVision/ImageNet)
    Controls the forgetting budget; tuned per noise level and dataset via 'small manual search' (Appendix F). Headline results depend on these values.
  • κ (KL penalty strength) = 0.05-0.07 for CIFAR-10, 0 for CIFAR-100, 0.05 for WebVision/ImageNet
    Regularizes the inner Q optimization and prevents numerical issues; selected by hand.
  • C (penalty strength in adaptive δ variant) = 1.2 for CIFAR-100N; 0.05-5 for CIFAR-10 (Table 6)
    Used in variant (58) when δ is adaptively chosen; tuned per noise level.
assumptions (5)
  • standard math f-divergence convexity and Gibbs variational formula (Theorem 4.2 of Ben-Tal and Teboulle 2007)
    Used in Section 3.3 and Appendix A.2 to derive the dual (10)-(11).
  • domain assumption Realizability: there exists θ* with h_{θ*} = h* (the true labeling map is in the model family)
    Theorem 3.1 condition 2; without it the unique-solution claim fails.
  • domain assumption Uniform label-noise model: T^x_{h*(x),h*(x)} = 1-r for all x, with r < 1/2
    Theorem 3.1 condition 1; real-world noise is instance-dependent, so the theorem does not cover the WebVision/CIFAR-100N settings.
  • domain assumption Loss zero-set: L(p,y) = 0 iff p = y (e.g., cross-entropy with one-hot targets)
    Theorem 3.1 condition 3; ensures zero expected loss implies perfect prediction on the support of Q.
  • ad hoc to paper The alternating minibatch scheme converges to the solution of the relaxed problem
    Algorithm 1 steps between λ-maximization and θ-SGD on minibatches; no convergence theorem links this to Theorem 3.2's exact solution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels." pith.science (2026). https://pith.science/paper/RX6V4PDI

@misc{pith2026250806622,
  author       = {Pith},
  title        = {Pith review of: Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RX6V4PDI}},
  note         = {Machine review of arXiv:2508.06622}
}
read the original abstract

We introduce ANTIDOTE, a new class of objectives for learning under noisy labels which are defined in terms of a relaxation over an information-divergence neighborhood. Using convex duality, we provide a reformulation as an adversarial training method that has similar computational cost to training with standard cross-entropy loss. We show that our approach adaptively reduces the influence of the samples with noisy labels during learning, exhibiting a behavior that is analogous to forgetting those samples. ANTIDOTE is effective in practical environments where label noise is inherent in the training data or where an adversary can alter the training labels. Extensive empirical evaluations on different levels of symmetric, asymmetric, human annotation, and real-world label noise show that ANTIDOTE outperforms leading comparable losses in the field and enjoys a time complexity that is very close to that of the standard cross entropy loss.

Figures

Figures reproduced from arXiv: 2508.06622 by the authors.

Figure 1
Figure 1. Evaluating ANTIDOTE’s Loss for Different Noise Ratios During Training [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 1
Figure 1. Evaluating ANTIDOTE’s Loss for Different Noise Ratios During Training Figure 2: ANTIDOTEs Loss for Different Noise Ratios During Training [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 63 canonical work pages

  1. [2]

    Asymmetric loss functions for learning with noisy labels,

    X. Zhou, X. Liu, J. Jiang, X. Gao, and X. Ji, “Asymmetric loss functions for learning with noisy labels,” in International conference on machine learning, pp. 12846–12856, PMLR, 2021

    work page 2021

  2. [3]

    Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity,

    D. Zhu, Y . Ying, and T. Yang, “Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity,” in International Conference on Machine Learning , pp. 43289–43325, PMLR, 2023

    work page 2023

  3. [4]

    Asymmetric loss functions for noise-tolerant learning: Theory and applications,

    X. Zhou, X. Liu, D. Zhai, J. Jiang, and X. Ji, “Asymmetric loss functions for noise-tolerant learning: Theory and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8094–8109, 2023

    work page 2023

  4. [5]

    ϵ-softmax: Approximating one- hot vectors for mitigating label noise,

    J. Wang, X. Zhou, D. Zhai, J. Jiang, X. Ji, and X. Liu, “ ϵ-softmax: Approximating one- hot vectors for mitigating label noise,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  5. [6]

    Learning with noisy labels,

    N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, “Learning with noisy labels,” Advances in neural information processing systems, vol. 26, 2013

    work page 2013

  6. [7]

    Classification with noisy labels by importance reweighting,

    T. Liu and D. Tao, “Classification with noisy labels by importance reweighting,”IEEE Transac- tions on pattern analysis and machine intelligence, vol. 38, no. 3, pp. 447–461, 2015

    work page 2015

  7. [9]

    Are anchor points really indispensable in label-noise learning?,

    X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama, “Are anchor points really indispensable in label-noise learning?,” Advances in neural information processing systems, vol. 32, 2019

    work page 2019

  8. [10]

    Dirichlet-based per-sample weighting by transi- tion matrix for noisy label learning,

    H. Bae, S. Shin, B. Na, and I. chul Moon, “Dirichlet-based per-sample weighting by transi- tion matrix for noisy label learning,” in The Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 67 references
  1. [11]

    Contrast to divide: Self-supervised pre-training for learning with noisy labels,

    E. Zheltonozhskii, C. Baskin, A. Mendelson, A. M. Bronstein, and O. Litany, “Contrast to divide: Self-supervised pre-training for learning with noisy labels,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1657–1667, 2022

    2022

  2. [12]

    Early-learning regularization prevents memorization of noisy labels,

    S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda, “Early-learning regularization prevents memorization of noisy labels,” Advances in neural information processing systems, vol. 33, pp. 20331–20342, 2020

    2020

  3. [13]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in International Conference on Learning Representations, 2018

    2018

  4. [14]

    Dividemix: Learning with noisy labels as semi-supervised learning,

    J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” in International Conference on Learning Representations, 2020

    2020

  5. [15]

    Robust training of deep neural networks with extremely noisy labels,

    B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Robust training of deep neural networks with extremely noisy labels,” in Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), vol. 2, p. 4, 2020

    2020

  6. [16]

    Robust training under label noise by over-parameterization,

    S. Liu, Z. Zhu, Q. Qu, and C. You, “Robust training under label noise by over-parameterization,” in International Conference on Machine Learning, pp. 14153–14172, PMLR, 2022

    2022

  7. [17]

    Csot: Curriculum and structure-aware optimal transport for learning with noisy labels,

    W. Chang, Y . Shi, and J. Wang, “Csot: Curriculum and structure-aware optimal transport for learning with noisy labels,” Advances in Neural Information Processing Systems, vol. 36, pp. 8528–8541, 2023. 11

    2023

  8. [18]

    Fixmatch: Simplifying semi-supervised learning with consistency and confidence,

    K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Advances in neural information processing systems, vol. 33, pp. 596–608, 2020

    2020

  9. [19]

    L2B: Learning to bootstrap robust models for combating label noise,

    Y . Zhou, X. Li, F. Liu, Q. Wei, X. Chen, L. Yu, C. Xie, M. P. Lungren, and L. Xing, “L2B: Learning to bootstrap robust models for combating label noise,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23523–23533, 2024

    2024

  10. [20]

    Badlabel: A robust perspective on evaluating and enhancing label-noise learning,

    J. Zhang, B. Song, H. Wang, B. Han, T. Liu, L. Liu, and M. Sugiyama, “Badlabel: A robust perspective on evaluating and enhancing label-noise learning,”IEEE transactions on pattern analysis and machine intelligence, 2024

    2024

  11. [21]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in International Conference on Learning Representations, 2018

    2018

  12. [22]

    Learning to reweight examples for robust deep learning,

    M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight examples for robust deep learning,” in International Conference on Machine Learning, pp. 4334–4343, PMLR, 2018

    2018

  13. [23]

    Combating noisy labels with sample selection by mining high-discrepancy examples,

    X. Xia, B. Han, Y . Zhan, J. Yu, M. Gong, C. Gong, and T. Liu, “Combating noisy labels with sample selection by mining high-discrepancy examples,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833–1843, 2023

    2023

  14. [24]

    Focal loss for dense object detection,

    T.-Y . Ross and G. Dollár, “Focal loss for dense object detection,” inproceedings of the IEEE conference on computer vision and pattern recognition, pp. 2980–2988, 2017

    2017

  15. [25]

    Robust loss functions under label noise for deep neural networks,

    A. Ghosh, H. Kumar, and P. S. Sastry, “Robust loss functions under label noise for deep neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, 2017

    2017

  16. [26]

    Generalized cross entropy loss for training deep neural networks with noisy labels,

    Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018

    2018

  17. [28]

    Symmetric cross entropy for robust learning with noisy labels,

    Y . Wang, X. Ma, Z. Chen, Y . Luo, J. Yi, and J. Bailey, “Symmetric cross entropy for robust learning with noisy labels,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 322–330, 2019

    2019

  18. [29]

    Normalized loss functions for deep learning with noisy labels,

    X. Ma, H. Huang, Y . Wang, S. Romano, S. Erfani, and J. Bailey, “Normalized loss functions for deep learning with noisy labels,” in International conference on machine learning, pp. 6543– 6553, PMLR, 2020

    2020

  19. [30]

    Mitigating memorization of noisy labels by clipping the model prediction,

    H. Wei, H. Zhuang, R. Xie, L. Feng, G. Niu, B. An, and Y . Li, “Mitigating memorization of noisy labels by clipping the model prediction,” in International Conference on Machine Learning, pp. 36868–36886, PMLR, 2023

    2023

  20. [31]

    When optimizing f-divergence is robust with label noise,

    J. Wei and Y . Liu, “When optimizing f-divergence is robust with label noise,” inInternational Conference on Learning Representations, 2021

    2021

  21. [32]

    Mitigating the impact of labeling errors on training via Rockafellian relaxation,

    L. L. Chen, B. Chern, E. Eckstrand, A. Mahapatra, and J. O. Royset, “Mitigating the impact of labeling errors on training via Rockafellian relaxation,” arXiv preprint arXiv:2405.20531, 2024

    2024 arXiv

  22. [33]

    How does disagreement help generalization against label corruption?,

    X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?,” in International conference on machine learning , pp. 7164–7173, PMLR, 2019

    2019

  23. [34]

    A general class of coefficients of divergence of one distribution from another,

    S. M. Ali and S. D. Silvey, “A general class of coefficients of divergence of one distribution from another,” Journal of the Royal Statistical Society: Series B (Methodological) , vol. 28, no. 1, pp. 131–142, 1966

    1966

  24. [35]

    On information-type measure of difference of probability distributions and indirect observations,

    I. Csiszár, “On information-type measure of difference of probability distributions and indirect observations,” Studia Sci. Math. Hungar., vol. 2, pp. 299–318, 1967. 12

    1967

  25. [36]

    Minimization of divergences on sets of signed measures,

    M. Broniatowski and A. Keziou, “Minimization of divergences on sets of signed measures,” Studia Scientiarum Mathematicarum Hungarica, vol. 43, no. 4, p. 403–442, 2006

    2006

  26. [37]

    Estimating divergence functionals and the likelihood ratio by convex risk minimization,

    X. Nguyen, M. J. Wainwright, and M. I. Jordan, “Estimating divergence functionals and the likelihood ratio by convex risk minimization,” IEEE Transactions on Information Theory , vol. 56, no. 11, pp. 5847–5861, 2010

    2010

  27. [38]

    (f, Γ)-divergences: Interpolating between f-divergences and integral probability metrics,

    J. Birrell, P. Dupuis, M. A. Katsoulakis, Y . Pantazis, and L. Rey-Bellet, “(f, Γ)-divergences: Interpolating between f-divergences and integral probability metrics,” Journal of machine learning research, vol. 23, no. 39, pp. 1–70, 2022

    2022

  28. [39]

    On divergences and informations in statistics and information theory,

    F. Liese and I. Vajda, “On divergences and informations in statistics and information theory,” IEEE Transactions on Information Theory, vol. 52, no. 10, pp. 4394–4412, 2006

    2006

  29. [40]

    Ponstein, Approaches to the Theory of Optimization

    J. Ponstein, Approaches to the Theory of Optimization . Cambridge Tracts in Mathematics, Cambridge University Press, 2004

    2004

  30. [41]

    An old-new concept of convex risk measures: The optimized certainty equivalent,

    A. Ben-Tal and M. Teboulle, “An old-new concept of convex risk measures: The optimized certainty equivalent,” Mathematical Finance, vol. 17, no. 3, pp. 449–476, 2007

    2007

  31. [42]

    Entropic value-at-risk: A new coherent risk measure,

    A. Ahmadi-Javid, “Entropic value-at-risk: A new coherent risk measure,” Journal of Optimiza- tion Theory and Applications, vol. 155, pp. 1105–1123, 2012

    2012

  32. [43]

    Theoretically principled trade-off between robustness and accuracy,

    H. Zhang, Y . Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” inInternational conference on machine learning, pp. 7472–7482, PMLR, 2019

    2019

  33. [44]

    Improving adversarial robustness requires revisiting misclassified examples,

    Y . Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu, “Improving adversarial robustness requires revisiting misclassified examples,” inInternational Conference on Learning Representations, 2020

    2020

  34. [45]

    A unified Wasserstein distributional robustness framework for adversarial training,

    A. T. Bui, T. Le, Q. H. Tran, H. Zhao, and D. Phung, “A unified Wasserstein distributional robustness framework for adversarial training,” in International Conference on Learning Repre- sentations, 2022

    2022

  35. [46]

    Adversarially robust deep learning with optimal-transport- regularized divergences,

    J. Birrell and M. Ebrahimi, “Adversarially robust deep learning with optimal-transport- regularized divergences,”arXiv preprint arXiv:2309.03791, 2023

    2023 arXiv

  36. [47]

    Nlnl: Negative learning for noisy labels,

    Y . Kim, J. Yim, J. Yun, and J. Kim, “Nlnl: Negative learning for noisy labels,” inProceedings of the IEEE/CVF international conference on computer vision, pp. 101–110, 2019

    2019

  37. [48]

    Making deep neural networks robust to label noise: A loss correction approach,

    G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944–1952, 2017

    1944

  38. [49]

    Peer loss functions: Learning from noisy labels without knowing noise rates,

    Y . Liu and H. Guo, “Peer loss functions: Learning from noisy labels without knowing noise rates,” in International conference on machine learning, pp. 6226–6236, PMLR, 2020

    2020

  39. [50]

    Provably end-to-end label-noise learning without anchor points,

    X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama, “Provably end-to-end label-noise learning without anchor points,” inInternational conference on machine learning, pp. 6403–6413, PMLR, 2021

    2021

  40. [51]

    To smooth or not? when label smoothing meets noisy labels,

    J. Wei, H. Liu, T. Liu, G. Niu, M. Sugiyama, and Y . Liu, “To smooth or not? when label smoothing meets noisy labels,” in International Conference on Machine Learning, pp. 23589– 23614, PMLR, 2022

    2022

  41. [52]

    Rethinking noisy label learning in real-world annotation scenarios from the noise-type perspective,

    R. Zhu, H. Liu, R. Wu, M. Lin, T. Lv, C. Fan, and H. Wang, “Rethinking noisy label learning in real-world annotation scenarios from the noise-type perspective,” arXiv preprint arXiv:2307.16889, 2023

    2023 arXiv

  42. [53]

    Understanding and improving early stopping for learning with noisy labels,

    Y . Bai, E. Yang, B. Han, Y . Yang, J. Li, Y . Mao, G. Niu, and T. Liu, “Understanding and improving early stopping for learning with noisy labels,” Advances in Neural Information Processing Systems, vol. 34, pp. 24392–24403, 2021. 13

    2021

  43. [54]

    Learning with instance-dependent label noise: A sample sieve approach,

    H. Cheng, Z. Zhu, X. Li, Y . Gong, X. Sun, and Y . Liu, “Learning with instance-dependent label noise: A sample sieve approach,” in International Conference on Learning Representations, 2021

    2021

  44. [55]

    A second-order approach to learning with instance-dependent label noise,

    Z. Zhu, T. Liu, and Y . Liu, “A second-order approach to learning with instance-dependent label noise,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10113–10123, 2021

    2021

  45. [56]

    Luenberger, Optimization by Vector Space Methods

    D. Luenberger, Optimization by Vector Space Methods. Professional Series, Wiley, 1997

    1997

  46. [57]

    Optnet: Differentiable optimization as a layer in neural networks,

    B. Amos and J. Z. Kolter, “Optnet: Differentiable optimization as a layer in neural networks,” in International conference on machine learning, pp. 136–145, PMLR, 2017

    2017

  47. [58]

    Differentiable convex optimization layers,

    A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter, “Differentiable convex optimization layers,” Advances in neural information processing systems, vol. 32, 2019

    2019

  48. [59]

    Dual t: Reducing estimation error for transition matrix in label-noise learning,

    Y . Yao, T. Liu, B. Han, M. Gong, J. Deng, G. Niu, and M. Sugiyama, “Dual t: Reducing estimation error for transition matrix in label-noise learning,” Advances in neural information processing systems, vol. 33, pp. 7260–7271, 2020. A Proofs A.1 Relaxed Problem Solution Set Her...

    2020

  49. [60]

    inf x∈X T x h∗(x),h∗(x) > 0

  50. [61]

    We have a classifier family hθ : X → P(Y), θ ∈ Θ, where P(Y) denotes the space of probability vectors over the class labels. We also suppose that there exists θ∗ ∈ Θ such that hθ∗ = h∗ (more precisely, this means hθ∗ = 1h∗, but to simplify notation we will identify each label ...

  51. [62]

    The training objective loss is defined by Lθ(x, y) := L(hθ(x), y)

    We have a loss function L : P(Y) × Y →[0, ∞] such that L(p, y) = 0 if and only if p = y. The training objective loss is defined by Lθ(x, y) := L(hθ(x), y)

  52. [63]

    We have a convex function f : (0, ∞) → R such that f (1) = 0 and f (0) := limt→0+ f (t) is finite. Then there exists δ ∈ (0, ∞) such that inf θ∈Θ inf Q:Df (Q∥PT )≤δ EQ[Lθ] = 0 , (18) with θ∗ being a minimizer, i.e., the relaxed optimization problem is solved by the exact non-n...

  53. [64]

    (23) Note that the latter condition automatically follows from the former if r <1/2

    We have r ∈ (0, 1) such that T x h∗(x),h∗(x) = 1 − r for all x ∈ Xand T x h∗(x),y < 1 − r for all x ∈ X, y∈ Ythat satisfy y ̸= h∗(x) . (23) Note that the latter condition automatically follows from the former if r <1/2

  54. [65]

    We also suppose that there exists θ∗ ∈ Θ such that hθ∗ = h∗ (again, note that we identify each label with its corresponding one-hot vector)

    We have a classifier family hθ : X → P(Y), θ ∈ Θ, where P(Y) denotes the space of probability vectors over the class labels. We also suppose that there exists θ∗ ∈ Θ such that hθ∗ = h∗ (again, note that we identify each label with its corresponding one-hot vector)

  55. [66]

    We have a loss function L : P(Y) × Y →[0, ∞] such that L(p, y) = 0 if and only if p = y

  56. [67]

    Equip Y with the discrete metric, dY (y1, y2) := 1 y1̸=y2 and assume that Lθ(x, y) := L(hθ(x), y) is lower semicontinuous (LSC) on X × Yfor all θ ∈ Θ

  57. [68]

    Let δ := rf (0) + (1 − r)f (1/(1 − r))

    We have a strictly convex f : (0, ∞) → R such that f (1) = 0 , f (0) := lim t→0+ f (t) is finite, and f ∗(y) is finite for all y ∈ R. Let δ := rf (0) + (1 − r)f (1/(1 − r)). Then for all ˜θ ∈ argmin θ∈Θ inf Q:Df (Q∥PT )≤δ EQ[Lθ] (24) we have h˜θ = h∗ almost surely under PX (ab...

  58. [69]

    0 ≤ a <1 < b≤ ∞and a convex f : (a, b) → R that satisfies f (1) = 0

  59. [70]

    A measurable function Lθ : Z →R, depending on parameters θ ∈ Θ. Then for all δ >0, κ ≥ 0 we have inf θ∈Θ inf Q:Df (Q∥Pn)≤δ {EQ[Lθ] + κDf (Q∥Pn)} (39) = inf θ∈Θ sup λ>0, ρ∈R −λδ − ρ − (λ + κ)EPn f ∗ − Lθ + ρ λ + κ and in the KL case, we have the further simplification inf θ∈Θ i...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.