Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Realistic Evaluation of Deep Partial-Label Learning Algorithms

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that model selection for deep partial-label learning can be done reliably using only a partial-label validation set, supported by two new criteria, a 27-algorithm benchmark, and a human-annotated image dataset.

desk verdict A genuinely useful benchmark and human-annotated dataset for PLL, with practical model selection criteria whose theory is scoped more narrowly than the abstract suggests and one fixable proof typo. read the letter →

arxiv 2502.10184 v1 pith:N62U6OAP submitted 2025-02-14 cs.LG

Wei Wang , Dong-Dong Wu , Jindong Wang , Gang Niu , Min-Ling Zhang , Masashi Sugiyama This is my paper
classification cs.LG
keywords partial-labellearningmodelselectionbenchmarkCoveringRateApproximatedAccuracycandidatelabelsetshuman-annotateddatasetweaklysupervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Partial-label learning trains classifiers when each example comes with a set of candidate labels and only one is correct. This paper argues that the field's experiments have been hard to trust: model selection usually sneaks in clean ordinary labels, experimental settings vary across papers, and real-world image data compatible with deep networks barely exists. To fix this, it proposes PLENCH, a standardized benchmark comparing twenty-seven deep PLL algorithms on eleven datasets, and introduces two model selection criteria, Covering Rate and Approximated Accuracy, that need only a partial-label validation set. A new human-annotated image dataset, PLCIFAR10, is collected from crowdsourced workers to test algorithms under realistic, non-synthetic partial-label noise. The central message is that simple early algorithms often match or beat complicated newer ones, and that fair evaluation requires specifying and standardizing the model selection protocol.

What carries the argument

The load-bearing objects are the two validation metrics. Covering Rate $\mathrm{CR}(f)$ is the fraction of validation examples whose predicted label falls inside the candidate label set; under uniform or constant-probability flipping partial-label generation, expected CR is an affine function of expected accuracy, so maximizing CR selects the same classifier as maximizing accuracy. Approximated Accuracy $\mathrm{AA}(f)$ replaces each candidate label with a probability-weighted surrogate and is unbiased for accuracy when the classifier matches the true class-posterior and partial labels are generated independently of the true label. The benchmark's third metric, Oracle Accuracy, uses true labels but is restricted to a single query per hyperparameter configuration so that it serves as an upper-bound reference rather than a practical selection rule. Together these metrics, plus the ambiguity degree $\gamma$ bounding the CR-accuracy gap, are what let the paper claim model selection without ordinary labels.

What would settle it

Generate a suite of partial-label datasets with class-dependent or feature-dependent non-constant flipping probabilities, run the same hyperparameter search over the paper's algorithm pool, and compare test accuracy of configurations selected by CR against configurations selected by oracle accuracy; if CR systematically picks worse configurations on some class or feature groups, the guarantee's scope is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that model selection in partial-label learning does not require an ordinary-label validation set. The Covering Rate $\mathrm{CR}(f)$ -- the fraction of validation examples whose predicted class lies inside the candidate label set -- is shown (Theorem 1) to rank classifiers by expected accuracy whenever partial labels come from the uniform sampling strategy or from the flipping-probability strategy with a constant flipping probability, because then $\mathrm{ACC}(f) = (\mathbb{E}[\mathrm{CR}(f)] - c)/(1-c)$. The Approximated Accuracy $\mathrm{AA}(f)$, which averages the predictive probability of the selected class over candidate labels, is shown (Theorem 2) to be statistically consistent with accuracy when candidate-label generation is label-independent and the classifier is calibrated. The paper also builds PLENCH, a benchmark of twenty-seven algorithms on eleven datasets, and PLCIFAR10, a crowdsourced CIFAR-10 version whose flipping probabilities are visibly non-constant and whose 'Vaguest' split has a 17.56% noise rate. In this realistic setting, simple early methods such as PRODEN remain among the strongest, and no single algorithm wins everywhere. The conclusion is that the evaluation protocol, not only the algorithm, needs to be reported and standardized.

Load-bearing premise

The consistency guarantee for the Covering Rate criterion assumes partial labels are generated by the uniform sampling strategy or by a flipping-probability strategy with a single constant probability, and the paper's own PLCIFAR10 dataset violates that assumption.

Editorial extensions

If this is right

  • Practitioners can select PLL hyperparameters and checkpoints using only the partial-label validation set, without needing a clean labeled set that the PLL definition forbids.
  • Under USS or constant-probability FPS generation, CR is a safe substitute for accuracy: the best CR configuration is the best accuracy configuration.
  • AA is a usable alternative when the model's outputs are well-calibrated posterior estimates, though the paper's experiments show it degrades when early-training outputs are unreliable.
  • PLENCH gives a standardized protocol so future algorithm claims can be compared against the same baselines and search procedures.
  • PLCIFAR10, especially the noisy Vaguest split, provides a testbed where synthetic-generation assumptions fail and noisy-PLL methods can be evaluated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to stress CR on instance-dependent partial-label generation, where Theorem 1's conditions fail and CR rankings may diverge from accuracy rankings.
  • Theorem 1's explicit affine link suggests a practical calibration step: estimate the constant $c$ from a validation set and convert CR into an accuracy estimate, which the paper does not do.
  • Because the benchmark fixes training at 60,000 iterations, combining CR with training dynamics (selecting by CR trajectory rather than a single checkpoint) is an open variant that could recover some of the early-stopping gains the oracle metric shows.
  • PLCIFAR10's per-annotator candidate sets would support annotation-aggregation studies, since the noise-rate curve shows aggregation reduces noise; the paper only treats aggregated and vaguest splits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that deep partial-label learning (PLL) evaluation suffers from three overlooked problems: model selection is typically done with clean ordinary-label validation sets even though the PLL problem definition forbids them; experimental settings across papers are inconsistent; and there is no real-world image dataset compatible with modern deep architectures. To address these, the authors propose PLENCH, a benchmark with 27 PLL algorithms and 11 datasets, including PLCIFAR10, a new human-annotated partial-label image dataset collected via Amazon Mechanical Turk. They also introduce three model-selection criteria for use with only a partial-label validation set: Covering Rate (CR), Approximated Accuracy (AA), and Oracle Accuracy (OA). The main theoretical claims are that CR ranks classifiers consistently with accuracy under the uniform sampling strategy (USS) or the flipping probability strategy (FPS) with constant flipping probability (Theorem 1), and that AA is statistically consistent with expected accuracy under a proper-partial-label condition (Theorem 2). Extensive experiments on tabular and image datasets show that simple algorithms such as PRODEN are very competitive, no algorithm dominates across all settings, noisy partial labels degrade performance, and CR is often close to oracle-based selection.

Significance. If the empirical results hold, PLENCH would be a valuable standardized benchmark for deep PLL, and PLCIFAR10 would be a useful real-world resource. The paper's main empirical findings—simple methods are strong, no single algorithm dominates, and model selection matters—are clearly supported by a large set of experiments with reproducible code and released data. The model-selection criteria address a real gap in the PLL literature, since practitioners often lack clean-label validation sets. The theoretical results are secondary to the benchmark contribution, but Theorem 1 and Theorem 2, if properly stated and proved, would provide useful guidance about when CR and AA can be trusted. The work is significant for the PLL community as a standardization effort, and the authors should be credited for shipping code, releasing the dataset, and documenting hyperparameter search pools and data splits.

major comments (2)
  1. [Appendix A.3] The proof of Theorem 2 as printed sets L(f(x),k) = I(argmax_j f_j(x) = y) with an indicator on y rather than on k. This is not the loss that appears in the definition of AA, so the derivation in the printed proof does not establish the claimed consistency. The intended argument is valid after replacing y with k in the indicator: Lemma 1 from Wu et al. (2023) then yields exactly E[AA(f)] in the conditional expectation, and if f is consistent with p(y|x), the indicator selects the true class with probability ACC(f). Please correct the proof and re-verify the statement.
  2. [Section 3.1 and Section 4] Theorem 1's guarantee applies only when partial labels are generated by USS or FPS with a constant flipping probability, as the paper itself notes at the end of Section 3.1. Section 4 shows that PLCIFAR10 has non-constant flipping probabilities and diagonals below 1, so Theorem 1 does not cover the paper's own realistic image dataset. The abstract's phrase 'model selection criteria with theoretical guarantees' should therefore be scoped to the conditions of Theorems 1 and 2, and the reported effectiveness of CR on PLCIFAR10 should be presented explicitly as empirical evidence, not as a consequence of Theorem 1. This distinction is important because practitioners may otherwise assume the consistency guarantee applies to the new benchmark.
minor comments (4)
  1. [Section 4, Figure 2] The caption says 'The flipping probability matrix computed on PLCIFAR10-Aggregate' but does not state whether the entry in row i, column j is p(j in S | y = i) or p(i in S | y = j). Please clarify the row/column convention.
  2. [Appendix B.1] The sentence 'We set the thresholds to 6 for a HIT of 10 images' is awkward; a single threshold value of 6 is described, so 'threshold' should be singular.
  3. [Section 5.1] The term 'holisitic' in 'holisitic PLL algorithms' is a typo for 'holistic.'
  4. [Tables 7-16] In several tables, the marker for the best result is placed inconsistently with the surrounding text (e.g., Table 9 marks 53.41 with a dagger-like superscript while the caption says bold). Please unify the formatting of the best-performance markers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model-selection criteria are derived from stated data-generation assumptions and are evaluated on held-out test accuracy, not on fitted values.

full rationale

The paper's central derivations are the CR ranking theorem (Theorem 1) and the AA consistency theorem (Theorem 2). Theorem 1 is a direct algebraic reduction: under USS or constant-FPS, E[CR]-ACC = c(1-ACC), so E[CR] orders classifiers exactly as ACC; no parameter is fitted to test data and no prediction is defined in terms of the benchmark outcome. Theorem 2 is an application of Lemma 1 from Wu et al. (2023), a published equivalence whose stated assumptions (p(S|x,y)=C(x,S)1[y in S]) do not include the target statement; although one author overlaps (Sugiyama), the lemma is parameter-free and independently stated, so this is independent support rather than circularity. CR's lack of theoretical coverage on PLCIFAR10 is explicitly acknowledged in the paper ('this conclusion may not hold when partial labels are not generated by either strategy'), making it a scope limitation, not a disguised fit. All benchmark results are test-set accuracies selected by validation criteria on held-out validation splits, so there is no fitted-input-as-prediction pattern. Separately, Appendix A.3 contains a proof typo: it writes L(f(x),k)=I(argmax_j f_j(x)=y) instead of '=k'; the corrected argument follows directly from Lemma 1, so this is a correctness issue, not circularity.

Assumptions & free parameters 1 free parameters · 6 assumptions · 2 invented entities

The central claims rest mainly on domain assumptions about the partial-label generation process and on the quality of the crowdsourced dataset. No free parameter is fitted to the benchmark results in the theoretical derivations. PLCIFAR10 and PLENCH are introduced artifacts with public release, so they carry independent evidence.

free parameters (1)
  • Crowd annotation thresholds = 6
    Section B.1 sets the same-label cap and the candidate-set-size cap to 6 per HIT. These hand-chosen thresholds shape PLCIFAR10's ambiguity and noise but are not fitted to the model selection theory.
assumptions (6)
  • domain assumption Partial labels are generated by USS or FPS with a constant flipping probability (Theorem 1).
    Stated in Section 3.1 as the condition for the Covering Rate consistency result. Figure 2(c,d) shows PLCIFAR10 violates this assumption.
  • domain assumption The distribution satisfies p(S|x,y) = C(x,S) I[y in S] for the Approximated Accuracy result (Theorem 2).
    Introduced in Section 3.2 and used to apply Lemma 1 from Wu et al. 2023. The paper notes this can hold for a wide range of partial-label mechanisms.
  • domain assumption The trained classifier f is consistent with the true posterior p(y|x) for AA consistency.
    Invoked in Theorem 2. The paper acknowledges this may fail early in training, which limits AA's practical reliability.
  • standard math The ambiguity degree gamma is less than 1, ensuring ERM learnability for PLL.
    Citing Cour et al. 2011 and Liu & Dietterich 2014 in Section 3.1, this is a known sufficient condition for PLL learnability.
  • domain assumption CIFAR-10 ground-truth labels are correct labels for the images in PLCIFAR10.
    Section 4 assumes the original CIFAR-10 labels are the true labels when measuring noise rates and constructing candidate sets from human annotations.
  • domain assumption MTurk workers followed the annotation instructions and the quality-control thresholds are sufficient.
    Section B.1 and the Ethics Statement describe the data collection process, but the reliability of the annotators is not independently quantified beyond the reported noise rates.
invented entities (2)
  • PLCIFAR10 dataset independent evidence
    purpose: Provides a real-world image PLL testbed with human-annotated partial labels, including a noisy 'Vaguest' version.
    The dataset is released on GitHub and can be independently inspected, downloaded, and used in benchmarks, so it has a falsifiable handle outside the paper.
  • PLENCH benchmark framework independent evidence
    purpose: Standardizes evaluation of deep PLL algorithms through code, datasets, and protocol definitions.
    The code is publicly available and the benchmark results can be reproduced or extended by other researchers, providing independent testability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Realistic Evaluation of Deep Partial-Label Learning Algorithms." pith.science (2026). https://pith.science/paper/N62U6OAP

@misc{pith2026250210184,
  author       = {Pith},
  title        = {Pith review of: Realistic Evaluation of Deep Partial-Label Learning Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N62U6OAP}},
  note         = {Machine review of arXiv:2502.10184}
}
read the original abstract

Partial-label learning (PLL) is a weakly supervised learning problem in which each example is associated with multiple candidate labels and only one is the true label. In recent years, many deep PLL algorithms have been developed to improve model performance. However, we find that some early developed algorithms are often underestimated and can outperform many later algorithms with complicated designs. In this paper, we delve into the empirical perspective of PLL and identify several critical but previously overlooked issues. First, model selection for PLL is non-trivial, but has never been systematically studied. Second, the experimental settings are highly inconsistent, making it difficult to evaluate the effectiveness of the algorithms. Third, there is a lack of real-world image datasets that can be compatible with modern network architectures. Based on these findings, we propose PLENCH, the first Partial-Label learning bENCHmark to systematically compare state-of-the-art deep PLL algorithms. We investigate the model selection problem for PLL for the first time, and propose novel model selection criteria with theoretical guarantees. We also create Partial-Label CIFAR-10 (PLCIFAR10), an image dataset of human-annotated partial labels collected from Amazon Mechanical Turk, to provide a testbed for evaluating the performance of PLL algorithms in more realistic scenarios. Researchers can quickly and conveniently perform a comprehensive and fair evaluation and verify the effectiveness of newly developed algorithms based on PLENCH. We hope that PLENCH will facilitate standardized, fair, and practical evaluation of PLL algorithms in the future.

Figures

Figures reproduced from arXiv: 2502.10184 by the authors.

Figure 1
Figure 1. The two left panels show the differences in using an ordinary-label dataset for valida [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a). The distribution of the collected partial labels of PLCIFAR10. (b) The noise rate with [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Experimental results of different algorithms on tabular datasets. The top, middle, and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Running time and GPU memory utilization for each running step of different PLL algo [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CLID-MU replaces the clean meta-dataset in meta-learning with an unsupervised cross-layer divergence metric, improving noisy-label and semi-supervised results on several benchmarks.

Reference graph

Works this paper leans on

126 extracted references · 79 canonical work pages · cited by 1 Pith paper

  1. [5]

    Weinberger

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708,

    2017

  2. [6]

    Kingma and Jimmy Ba

    12 Published as a conference paper at ICLR 2025 Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations,

    2025

  3. [9]

    Decompositional generation process for instance-dependent partial label learning

    13 Published as a conference paper at ICLR 2025 Congyu Qiao, Ning Xu, and Xin Geng. Decompositional generation process for instance-dependent partial label learning. In Proceedings of the 11th International Conference on Learning Repre- sentations, 2023a. Congyu Qiao, Ning Xu, Jiaqi Lv, Yi Ren, and Xin Geng. FREDIS: A fusion framework of refinement and di...

    arXiv 2025

  4. [10]

    AFET: Automatic fine- grained entity typing by hierarchical partial-label embedding

    Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. AFET: Automatic fine- grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pp. 1369–1378, 2016a. Xiang Ren, Wenqi He, Meng Qu, Clare R. V oss, Heng Ji, and Jiawei Han. Label noise reducti...

    2016

  5. [13]

    Here, the third equation can be obtained by traversing the cases of arg maxj fjpxq “ y, arg maxj fjpxq PSztyu, and arg maxj fjpxq RS

    15 Published as a conference paper at ICLR 2025 A P ROOFS A.1 P ROOF OF PROPOSITION 1 ErCRpfqs´ ACCpfq “Eppx,Sq „ I ˆ arg max j fjpxqP S ˙ȷ ´ Eppx,yq „ I ˆ arg max j fjpxq“ y ˙ȷ “Eppx,y,Sq „ I ˆ arg max j fjpxqP Sztyu ˙ȷ “Eppx,y,Sq „ I ˆ arg max j fjpxqP Sztyu ˙ I ˆ arg max j fjpxq‰ y ˙ȷ “p ˆ arg max j fjpxqP Sztyu, arg max j fjpxq‰ y ˙ “p ˆ arg max j fjp...

    2023

  6. [14]

    Dataset # Examples # Features # Classes Avg

    Table 5: Characteristics of real-world PLL datasets used in PLENCH . Dataset # Examples # Features # Classes Avg. # CLs Noise Rate Type Task Domain Lost 1,122 108 16 2.23 0% tabular automatic face naming (Cour et al., 2011)MSRCv2 1,758 48 23 3.16 0% tabular object classification (Liu & Dietterich, 2012)Mirflickr 2,780 1,536 14 2.76 0% tabular web image cl...

    2011

  7. [16]

    Holistic Deep PLL AlgorithmsV ALEN (Xu et al., 2021), PiCO (Wang et al., 2022b), ABLE (Xia et al., 2022), CRDPLL (Wu et al., 2022),DIRK (Wu et al.,

    2021

  8. [18]

    Then, we provide the hyperparameter configurations for all the algorithms

    E D ETAILS OF DEEP PLL A LGORITHMS In this section, we first give detailed descriptions of the PLL algorithms used in this paper. Then, we provide the hyperparameter configurations for all the algorithms. Table 6 shows the summary of the deep PLL algorithms included in this paper. E.1 D ESCRIPTIONS OF ALGORITHMS The vanilla deep PLL algorithms include: • ...

    2020

Show all 126 references
  1. [19]

    71.33 ˘ 2.45 70.09 ˘ 2.33 71.33 ˘ 2.41 71.33 ˘ 2.29 CA VL ICLR 2022 (Zhang et al.,

    2022

  2. [20]

    60.00 ˘ 3.30 58.41 ˘ 2.99 64.42 ˘ 2.80 61.95 ˘ 2.40 POP ICML 2023 (Xu et al., 2023b) 69.38 ˘ 1.96 69.38 ˘ 1.96 70.44 ˘ 2.45 73.10 ˘ 2.16 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 68.50 ˘ 2.70 67.43 ˘ 2.10 68.85 ˘ 3.07 69.03 ˘ 2.99 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 62.65 ˘ 2.64...

    2023

  3. [21]

    72.92 ˘ 3.59 71.33 ˘ 3.75 72.21 ˘ 2.41 74.34 ˘ 2.68 IDGP ICLR 2023 (Qiao et al., 2023a) 69.73 ˘ 2.87 70.97 ˘ 1.59 75.04 ˘ 1.61 71.68 ˘ 2.39 PC NeurIPS 2017 (Ishida et al.,

    2023

  4. [23]

    71.68 ˘ 3.70 71.50 ˘ 3.77 73.27 ˘ 2.91 72.74 ˘ 3.51 NN ICML 2019 (Ishida et al.,

    2019

  5. [24]

    26.73 ˘ 1.04 17.52 ˘ 1.49 22.48 ˘ 1.24 26.37 ˘ 1.04 GA ICML 2019 (Ishida et al.,

    2019

  6. [25]

    18.94 ˘ 3.40 13.63 ˘ 2.57 6.73 ˘ 1.14 20.00 ˘ 2.06 SCL-EXP ICML 2020 (Chou et al.,

    2020

  7. [26]

    69.73 ˘ 2.50 70.80 ˘ 2.66 73.81 ˘ 1.95 73.81 ˘ 1.85 SCL-NL ICML 2020 (Chou et al.,

    2020

  8. [27]

    71.68 ˘ 2.54 70.80 ˘ 3.03 74.69 ˘ 2.97 72.74 ˘ 1.93 L-W ICML 2021 (Gao & Zhang,

    2021

  9. [28]

    63.72 ˘ 2.36 63.36 ˘ 1.19 63.01 ˘ 2.81 65.31 ˘ 2.33 OP-W AISTATS 2023 (Liu et al.,

    2023

  10. [29]

    73.10 ˘ 3.14 62.30 ˘ 2.61 76.28 ˘ 2.78 76.99 ˘ 2.68 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  11. [30]

    • PiCO (Wang et al., 2022b): A PLL algorithm that uses the supervised contrastive learning module to improve model performance

    66.02 ˘ 2.43 65.31 ˘ 2.55 64.42 ˘ 2.98 66.19 ˘ 3.32 The holistic deep PLL algorithms: • V ALEN (Xu et al., 2021): An identification-based strategy that uses variational inference to esti- mate the true label distribution. • PiCO (Wang et al., 2022b): A PLL algorithm that uses ...

    2021

  12. [31]

    We used the Adam optimizer (Kingma & Ba, 2015)

    and all experiments were conducted with a single NVIDIA Tesla V100 GPU. We used the Adam optimizer (Kingma & Ba, 2015). We ran 60,000 iterations for the image datasets, 20,000 iterations for the Soccer Player, Ital- ian, Yahoo! News, and English datasets, and 10,000 iterations...

    2015

  13. [32]

    52.50 ˘0.83 53.41˘1.11 51.82˘0.65 52.39 ˘1.16 CA VL ICLR 2022 (Zhang et al.,

    2022

  14. [33]

    46.14 ˘0.84 45.00˘1.26 48.64˘1.56 46.14 ˘1.63 POP ICML 2023 (Xu et al., 2023b) 51.36 ˘0.92 52.05˘0.73 52.61˘1.10 54.09 ˘1.01 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 45.68˘1.82 44.09˘1.80 46.48˘1.91 48.18 ˘1.47 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 48.30 ˘1.80 46.93˘1.37 49.55˘1....

    2023

  15. [34]

    47.50 ˘0.80 46.59˘0.53 49.77˘1.35 49.32 ˘1.03 IDGP ICLR 2023 (Qiao et al., 2023a) 52.61 ˘0.81 50.45˘2.07 52.05˘1.14 52.84 ˘1.56 PC NeurIPS 2017 (Ishida et al.,

    2023

  16. [35]

    41.82 ˘1.33 43.98˘2.02 44.09˘0.73 44.89 ˘1.24 Forward ECCV 2018 (Yu et al.,

    2018

  17. [36]

    49.89 ˘1.71 51.59˘1.64 51.14˘0.53 50.57 ˘1.80 NN ICML 2019 (Ishida et al.,

    2019

  18. [37]

    18.86 ˘1.79 16.82˘1.33 16.70˘0.83 20.23 ˘1.14 GA ICML 2019 (Ishida et al.,

    2019

  19. [38]

    12.95 ˘1.01 13.52˘0.67 10.91˘0.74 14.55 ˘1.22 SCL-EXP ICML 2020 (Chou et al.,

    2020

  20. [39]

    48.86 ˘1.41 49.89˘1.20 48.98˘1.70 49.32 ˘1.67 SCL-NL ICML 2020 (Chou et al.,

    2020

  21. [40]

    51.25 ˘1.02 51.70˘1.41 49.32˘1.11 51.36 ˘1.24 L-W ICML 2021 (Gao & Zhang,

    2021

  22. [41]

    44.77 ˘1.63 41.93˘0.74 45.34˘0.90 44.55 ˘0.89 OP-W AISTATS 2023 (Liu et al.,

    2023

  23. [42]

    49.89 ˘0.98 27.50˘9.01 51.59˘1.55 51.93 ˘1.42 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  24. [43]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    48.30 ˘1.43 49.09˘1.29 48.41˘0.93 49.89 ˘0.81 Table 10: Classification accuracy (mean˘std) of each algorithm on Mirflickr with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    2020

  25. [44]

    66.19 ˘1.61 65.40˘1.00 66.47˘1.63 66.69 ˘1.29 CA VL ICLR 2022 (Zhang et al.,

    2022

  26. [45]

    63.88 ˘0.69 64.75˘1.40 62.73˘1.69 65.83 ˘1.17 POP ICML 2023 (Xu et al., 2023b) 65.40 ˘1.18 64.53˘0.78 66.62˘1.05 66.40 ˘1.28 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 63.60˘0.98 58.49˘1.92 63.81˘1.01 66.04 ˘1.07 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 53.96 ˘1.71 53.02˘0.78 54.17˘0....

    2023

  27. [46]

    64.96 ˘1.30 38.78˘5.29 65.04˘1.23 65.54 ˘1.77 IDGP ICLR 2023 (Qiao et al., 2023a) 66.12 ˘2.07 68.35˘1.22 68.13˘0.98 69.06 ˘0.91 PC NeurIPS 2017 (Ishida et al.,

    2023

  28. [47]

    62.45 ˘1.14 58.78˘1.41 62.37˘1.07 63.02 ˘1.61 Forward ECCV 2018 (Yu et al.,

    2018

  29. [48]

    65.32 ˘1.45 64.68˘2.16 64.96˘1.19 65.54 ˘1.33 NN ICML 2019 (Ishida et al.,

    2019

  30. [49]

    19.93 ˘1.78 9.28 ˘1.19 18.06 ˘1.22 24.96 ˘1.52 GA ICML 2019 (Ishida et al.,

    2019

  31. [50]

    21.37 ˘0.52 11.37˘4.01 17.84˘0.39 21.51 ˘1.86 SCL-EXP ICML 2020 (Chou et al.,

    2020

  32. [51]

    63.31 ˘1.29 59.42˘4.38 64.60˘1.03 65.32 ˘0.86 SCL-NL ICML 2020 (Chou et al.,

    2020

  33. [52]

    63.60 ˘1.13 63.74˘1.17 67.12˘1.15 66.83 ˘0.62 L-W ICML 2021 (Gao & Zhang,

    2021

  34. [53]

    57.55 ˘1.31 52.09˘3.57 57.34˘2.71 60.07 ˘2.18 OP-W AISTATS 2023 (Liu et al.,

    2023

  35. [54]

    64.32 ˘1.47 15.32˘9.24 64.53˘1.24 65.83 ˘1.21 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  36. [55]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    58.56 ˘1.69 59.42˘1.10 58.78˘1.70 60.58 ˘0.94 22 Published as a conference paper at ICLR 2025 Table 11: Classification accuracy (mean˘std) of each algorithm on Birdsong with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    2025

  37. [56]

    71.64 ˘0.61 70.16˘0.88 71.44˘0.65 72.32 ˘0.89 CA VL ICLR 2022 (Zhang et al.,

    2022

  38. [57]

    65.96 ˘0.74 66.04˘0.36 66.56˘0.41 67.08 ˘0.68 POP ICML 2023 (Xu et al., 2023b) 71.80 ˘0.98 69.96˘0.58 72.60˘0.92 72.24 ˘0.73 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 66.20˘0.83 63.48˘1.17 68.20˘0.70 68.28 ˘0.72 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 69.04 ˘0.55 67.96˘0.79 69.44˘0....

    2023

  39. [58]

    66.04 ˘0.43 56.48˘5.22 65.84˘0.61 66.04 ˘0.58 IDGP ICLR 2023 (Qiao et al., 2023a) 72.48 ˘0.61 69.72˘0.50 73.40˘0.64 73.24 ˘0.62 PC NeurIPS 2017 (Ishida et al.,

    2023

  40. [59]

    68.92 ˘0.33 66.56˘1.13 70.08˘0.52 69.88 ˘0.72 Forward ECCV 2018 (Yu et al.,

    2018

  41. [60]

    69.84 ˘0.77 69.80˘0.41 70.00˘0.50 69.88 ˘0.61 NN ICML 2019 (Ishida et al.,

    2019

  42. [61]

    18.68 ˘1.28 16.04˘0.80 18.00˘0.70 20.72 ˘0.91 GA ICML 2019 (Ishida et al.,

    2019

  43. [62]

    20.08 ˘1.24 20.04˘1.14 13.24˘0.73 19.84 ˘1.15 SCL-EXP ICML 2020 (Chou et al.,

    2020

  44. [63]

    70.12 ˘0.61 68.84˘0.93 70.72˘0.38 71.08 ˘0.61 SCL-NL ICML 2020 (Chou et al.,

    2020

  45. [64]

    70.32 ˘0.62 70.16˘0.95 70.28˘0.50 70.28 ˘0.78 L-W ICML 2021 (Gao & Zhang,

    2021

  46. [65]

    62.48 ˘0.71 61.80˘0.83 66.48˘0.82 67.60 ˘0.81 OP-W AISTATS 2023 (Liu et al.,

    2023

  47. [66]

    69.60 ˘0.66 51.72˘9.37 69.60˘0.74 71.80 ˘0.91 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  48. [67]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    66.76 ˘1.62 66.76˘1.07 66.76˘0.67 68.44 ˘1.00 Table 12: Classification accuracy (mean˘std) of each algorithm on Malagasy with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    2020

  49. [68]

    67.19 ˘0.69 66.93˘2.41 71.83˘0.81 71.15 ˘0.58 CA VL ICLR 2022 (Zhang et al.,

    2022

  50. [69]

    65.46 ˘1.97 64.71˘1.21 68.55˘0.98 69.34 ˘1.08 POP ICML 2023 (Xu et al., 2023b) 62.52 ˘1.89 65.39˘2.12 70.47˘0.65 70.43 ˘0.81 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 58.46˘3.15 54.39˘3.29 66.52˘0.62 66.06 ˘0.44 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 60.64 ˘1.38 59.70˘2.10 63.47˘1....

    2023

  51. [70]

    57.74 ˘1.33 45.50˘4.37 63.54˘1.56 65.08 ˘1.02 IDGP ICLR 2023 (Qiao et al., 2023a) 69.94 ˘0.70 67.50˘1.45 70.06˘0.62 70.70 ˘0.83 PC NeurIPS 2017 (Ishida et al.,

    2023

  52. [71]

    67.01 ˘1.36 62.52˘1.38 69.15˘1.08 69.38 ˘0.82 Forward ECCV 2018 (Yu et al.,

    2018

  53. [72]

    61.51 ˘2.55 62.79˘2.60 69.72˘0.44 69.38 ˘0.79 NN ICML 2019 (Ishida et al.,

    2019

  54. [73]

    19.89 ˘1.75 14.43˘2.04 20.34˘1.00 25.54 ˘0.75 GA ICML 2019 (Ishida et al.,

    2019

  55. [74]

    18.83 ˘1.51 19.47˘1.68 9.04 ˘0.65 19.55 ˘1.62 SCL-EXP ICML 2020 (Chou et al.,

    2020

  56. [75]

    61.73 ˘2.40 57.48˘1.38 69.76˘0.36 69.72 ˘0.87 SCL-NL ICML 2020 (Chou et al.,

    2020

  57. [76]

    57.36 ˘0.69 61.77˘2.34 68.66˘1.05 68.89 ˘0.62 L-W ICML 2021 (Gao & Zhang,

    2021

  58. [77]

    57.33 ˘1.74 58.38˘2.46 64.71˘1.11 66.70 ˘1.03 OP-W AISTATS 2023 (Liu et al.,

    2023

  59. [78]

    58.95 ˘0.77 0.64 ˘0.18 70.21 ˘0.39 69.34 ˘0.74 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  60. [79]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    65.76 ˘1.23 60.23˘2.71 68.78˘0.75 68.63 ˘0.64 23 Published as a conference paper at ICLR 2025 Table 13: Classification accuracy (mean ˘std) of each algorithm on Soccer Player with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020...

    2025

  61. [80]

    54.98 ˘0.41 54.65 ˘0.55 55.02 ˘0.52 55.38 ˘0.49 CA VL ICLR 2022 (Zhang et al.,

    2022

  62. [81]

    51.89 ˘1.08 51.52 ˘1.09 51.26 ˘1.04 51.67 ˘1.02 POP ICML 2023 (Xu et al., 2023b) 54.74 ˘0.55 54.57 ˘0.62 54.82 ˘0.56 55.02 ˘0.40 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 48.90˘0.62 48.75 ˘0.54 48.90 ˘0.62 48.90 ˘0.62 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 55.33˘0.48 48.95 ˘0.55 55...

    2023

  63. [82]

    52.55 ˘0.48 48.99 ˘0.51 52.60 ˘0.45 52.69 ˘0.50 IDGP ICLR 2023 (Qiao et al., 2023a) 54.39 ˘0.71 54.16 ˘0.58 54.97 ˘0.42 55.13 ˘0.68 PC NeurIPS 2017 (Ishida et al.,

    2023

  64. [83]

    54.85 ˘0.76 48.75 ˘0.54 54.70 ˘0.72 54.45 ˘0.64 Forward ECCV 2018 (Yu et al.,

    2018

  65. [84]

    50.35 ˘0.93 49.93 ˘1.01 50.57 ˘0.86 50.47 ˘0.95 NN ICML 2019 (Ishida et al.,

    2019

  66. [85]

    11.10 ˘0.38 3.04 ˘0.29 9.97 ˘0.35 11.10 ˘0.38 GA ICML 2019 (Ishida et al.,

    2019

  67. [86]

    6.06 ˘0.46 4.95 ˘1.17 5.06 ˘0.24 6.02 ˘0.50 SCL-EXP ICML 2020 (Chou et al.,

    2020

  68. [87]

    49.32 ˘0.87 49.35 ˘0.87 49.51 ˘0.89 49.42 ˘0.84 SCL-NL ICML 2020 (Chou et al.,

    2020

  69. [88]

    50.57 ˘0.83 49.98 ˘1.06 50.58 ˘0.91 50.55 ˘0.95 L-W ICML 2021 (Gao & Zhang,

    2021

  70. [89]

    49.41 ˘0.69 48.84 ˘0.59 49.60 ˘0.60 49.83 ˘0.66 OP-W AISTATS 2023 (Liu et al.,

    2023

  71. [90]

    50.77 ˘0.48 28.54˘10.31 51.12˘0.61 50.93 ˘0.51 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  72. [91]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    52.13 ˘0.54 51.97 ˘0.52 52.15 ˘0.54 52.30 ˘0.59 Table 14: Classification accuracy (mean ˘std) of each algorithm on Italian with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    2020

  73. [92]

    68.89 ˘0.44 69.32˘0.60 71.80˘0.95 72.98 ˘0.66 CA VL ICLR 2022 (Zhang et al.,

    2022

  74. [93]

    68.83 ˘0.96 68.12˘0.72 70.71˘0.45 70.62 ˘0.60 POP ICML 2023 (Xu et al., 2023b) 68.48 ˘0.67 70.19˘0.36 71.56˘1.03 71.78 ˘0.91 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 67.60˘0.32 66.92˘0.84 70.11˘1.31 70.07 ˘1.26 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 66.76 ˘0.64 65.36˘0.68 69.60˘0....

    2023

  75. [94]

    71.65 ˘0.45 58.70˘5.74 73.36˘0.50 73.68 ˘0.73 IDGP ICLR 2023 (Qiao et al., 2023a) 68.43 ˘0.50 67.43˘0.82 70.08˘0.19 70.73 ˘0.42 PC NeurIPS 2017 (Ishida et al.,

    2023

  76. [95]

    70.10 ˘0.69 67.41˘1.23 71.22˘0.33 71.73 ˘0.42 Forward ECCV 2018 (Yu et al.,

    2018

  77. [96]

    68.28 ˘1.29 68.28˘1.29 71.20˘0.86 71.63 ˘0.66 NN ICML 2019 (Ishida et al.,

    2019

  78. [97]

    32.92 ˘1.92 17.62˘5.35 23.78˘0.57 33.75 ˘1.28 GA ICML 2019 (Ishida et al.,

    2019

  79. [98]

    11.62 ˘1.61 11.98˘1.43 5.88 ˘0.25 11.98 ˘1.43 SCL-EXP ICML 2020 (Chou et al.,

    2020

  80. [99]

    67.34 ˘0.31 67.34˘0.31 69.03˘0.54 69.08 ˘0.69 SCL-NL ICML 2020 (Chou et al.,

    2020

  81. [100]

    67.59 ˘0.33 67.59˘0.33 68.82˘0.69 70.33 ˘1.22 L-W ICML 2021 (Gao & Zhang,

    2021

  82. [101]

    67.43 ˘0.45 68.78˘0.43 68.90˘0.20 69.56 ˘0.33 OP-W AISTATS 2023 (Liu et al.,

    2023

  83. [102]

    65.70 ˘1.10 1.11 ˘0.38 71.04 ˘1.06 71.30 ˘1.04 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  84. [103]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    67.97 ˘0.86 67.29˘0.42 69.31˘0.64 71.12 ˘0.80 24 Published as a conference paper at ICLR 2025 Table 15: Classification accuracy (mean ˘std) of each algorithm on Yahoo! News with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (...

    2025

  85. [104]

    66.83 ˘0.52 66.29˘0.54 66.71˘0.58 66.81 ˘0.51 CA VL ICLR 2022 (Zhang et al.,

    2022

  86. [105]

    57.77 ˘1.53 57.14˘1.64 57.30˘1.55 58.30 ˘1.57 POP ICML 2023 (Xu et al., 2023b) 66.42 ˘0.74 65.76˘0.46 66.02˘0.57 66.17 ˘0.64 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 55.70˘1.53 48.53˘3.62 56.11˘1.63 56.15 ˘1.61 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 58.03 ˘0.43 57.79˘0.43 58.54˘0....

    2023

  87. [106]

    67.64 ˘0.73 39.87˘9.64 67.18˘0.55 67.46 ˘0.32 IDGP ICLR 2023 (Qiao et al., 2023a) 66.10 ˘0.63 65.66˘0.69 62.55˘0.75 65.99 ˘0.31 PC NeurIPS 2017 (Ishida et al.,

    2023

  88. [107]

    58.19 ˘0.41 54.07˘0.79 58.22˘0.46 58.17 ˘0.39 Forward ECCV 2018 (Yu et al.,

    2018

  89. [108]

    50.70 ˘1.25 50.56˘1.33 50.62˘1.40 50.70 ˘1.25 NN ICML 2019 (Ishida et al.,

    2019

  90. [109]

    23.80 ˘0.24 1.77 ˘0.16 23.23 ˘0.27 24.19 ˘0.20 GA ICML 2019 (Ishida et al.,

    2019

  91. [110]

    13.30 ˘0.31 13.30˘0.31 12.91˘0.57 13.30 ˘0.31 SCL-EXP ICML 2020 (Chou et al.,

    2020

  92. [111]

    50.30 ˘1.26 50.35˘1.23 50.57˘1.28 50.70 ˘1.21 SCL-NL ICML 2020 (Chou et al.,

    2020

  93. [112]

    50.59 ˘1.22 50.50˘1.19 50.76˘1.01 50.85 ˘1.16 L-W ICML 2021 (Gao & Zhang,

    2021

  94. [113]

    43.95 ˘0.67 42.98˘0.66 42.71˘0.62 44.60 ˘0.70 OP-W AISTATS 2023 (Liu et al.,

    2023

  95. [114]

    57.37 ˘0.96 45.21˘9.93 57.35˘1.05 57.49 ˘0.92 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  96. [115]

    Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    56.97 ˘0.44 56.98˘0.45 56.02˘0.62 57.82 ˘0.50 Table 16: Classification accuracy (mean ˘std) of each algorithm on English with different model selection criteria. Algorithm Venue w/ CR w/ AA w/ OA w/ OA & ES PRODEN ICML 2020 (Lv et al.,

    2020

  97. [116]

    74.01 ˘0.32 73.78 ˘0.13 73.62 ˘0.15 73.70 ˘0.26 CA VL ICLR 2022 (Zhang et al.,

    2022

  98. [117]

    73.71 ˘0.35 73.58 ˘0.35 74.15 ˘0.50 74.19 ˘0.25 POP ICML 2023 (Xu et al., 2023b) 74.04 ˘0.32 73.99 ˘0.17 73.78 ˘0.31 74.44 ˘0.36 ABS-MAE TPAMI 2024 (Lv et al., 2024a) 73.07˘0.22 72.37 ˘0.44 73.20 ˘0.13 73.31 ˘0.28 ABS-GCE TPAMI 2024 (Lv et al., 2024a) 73.52˘0.30 73.12 ˘0.26 74...

    2023

  99. [118]

    73.31 ˘0.34 64.11 ˘3.99 73.98 ˘0.38 73.91 ˘0.48 IDGP ICLR 2023 (Qiao et al., 2023a) 74.01 ˘0.26 73.82 ˘0.24 73.90 ˘0.35 74.44 ˘0.23 PC NeurIPS 2017 (Ishida et al.,

    2023

  100. [119]

    73.47 ˘0.29 72.82 ˘0.30 73.38 ˘0.45 73.74 ˘0.35 Forward ECCV 2018 (Yu et al.,

    2018

  101. [120]

    73.40 ˘0.21 73.60 ˘0.37 73.88 ˘0.24 73.40 ˘0.48 NN ICML 2019 (Ishida et al.,

    2019

  102. [121]

    54.18 ˘0.89 53.82 ˘0.76 39.25 ˘1.43 54.18 ˘0.89 GA ICML 2019 (Ishida et al.,

    2019

  103. [122]

    34.78 ˘1.90 34.77 ˘1.91 23.19 ˘0.84 34.77 ˘1.91 SCL-EXP ICML 2020 (Chou et al.,

    2020

  104. [123]

    73.84 ˘0.24 73.44 ˘0.40 74.08 ˘0.26 73.84 ˘0.25 SCL-NL ICML 2020 (Chou et al.,

    2020

  105. [124]

    74.01 ˘0.22 73.78 ˘0.44 74.04 ˘0.31 74.17 ˘0.24 L-W ICML 2021 (Gao & Zhang,

    2021

  106. [125]

    73.84 ˘0.44 73.13 ˘0.54 74.02 ˘0.39 74.14 ˘0.29 OP-W AISTATS 2023 (Liu et al.,

    2023

  107. [126]

    73.88 ˘0.39 31.83˘15.01 73.97˘0.35 74.05 ˘0.36 V ALEN NeurIPS 2021 (Xu et al.,

    2021

  108. [2009]

    Each image was resized to 256ˆ 256 for easy annotation

    B.1 M ORE DETAILS OF PLCIFAR10 For each example, we have a list of lists, where each sublist contains partial labels given by a single annotator. Each image was resized to 256ˆ 256 for easy annotation. We also imposed several requirements on the annotation task to ensure the q...

    2025

  109. [2010]

    Is object localization for free? Weakly- supervised learning with convolutional neural networks

    Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Is object localization for free? Weakly- supervised learning with convolutional neural networks. In Proceedings of the 2015 IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 685–694,

    2015

  110. [2012]

    On focal loss for class-posterior probability estimation: A theoretical perspective

    Nontawat Charoenphakdee, Jayakorn V ongkulbhisal, Nuttapong Chairatanakul, and Masashi Sugiyama. On focal loss for class-posterior probability estimation: A theoretical perspective. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. ...

    2021

  111. [2017]

    66.37 ˘ 2.54 63.01 ˘ 2.89 66.37 ˘ 2.33 67.96 ˘ 2.29 Forward ECCV 2018 (Yu et al.,

    2018

  112. [2018]

    Unbiased risk estimators can mislead: A case study of learning with complementary labels

    Yu-Ting Chou, Gang Niu, Hsuan-Tien Lin, and Masashi Sugiyama. Unbiased risk estimators can mislead: A case study of learning with complementary labels. In Proceedings of the 37th Inter- national Conference on Machine Learning, pp. 1929–1938,

    1929

  113. [2019]

    Peterson, Ruairidh M

    Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. InProceedings of the 2019 IEEE/CVF International Conference on Computer Vision, pp. 9617–9626,

    2019

  114. [2020]

    Partial label learning with discrimination augmentation

    14 Published as a conference paper at ICLR 2025 Wei Wang and Min-Ling Zhang. Partial label learning with discrimination augmentation. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp. 1920–1928,

    2025

  115. [2021]

    Deep residual learning for image recog- nition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778,

    2016

  116. [2022]

    Learning a part-of-speech tagger from two hours of annotation

    11 Published as a conference paper at ICLR 2025 Dan Garrette and Jason Baldridge. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2025

  117. [2023]

    CroSel: Cross selection of confident pseudo labels for partial-label learning

    Shiyu Tian, Hongxin Wei, Yiqun Wang, and Lei Feng. CroSel: Cross selection of confident pseudo labels for partial-label learning. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19479–19488,

    2024

  118. [2024]

    cuda" if index.is_cuda else

    Deep Noisy PLL AlgorithmsFREDIS (Qiao et al., 2023b), ALIM (Xu et al., 2023a), PiCO+ (Wang et al., 2024a) example, to implement CA VL (Zhang et al., 2022), we can write the following code: class CAVL(Algorithm): def __init__(self, input_shape, train_givenY, hparams): super(CAV...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.