Pith. sign in

REVIEW 4 major objections 6 minor 161 references

MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Pairing high-difficulty images with low-difficulty ones inside each minibatch consistently improves training across four diverse-quality image tasks.

desk verdict A plausible, broadly evaluated minibatch-sampling idea with a real reproducibility hole: the algorithm is only defined for batch size 2, though all experiments use larger batches. read the letter →

arxiv 2505.18741 v1 pith:BS6UQPX3 submitted 2025-05-24 cs.CV cs.AI

classification cs.CVcs.AI
keywords mixed-orderminibatchsamplingsampledifficultyuncertaintyestimationlossandrankinglong-tailedclassificationnoisy-labeluniversallesiondetectionCOVID-19CTsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most training strategies for diverse-quality images either down-weight or over-sample high-loss samples, using loss alone as the measure of hardness. This paper argues that loss alone cannot distinguish samples that are hard because they are poorly labeled from samples that are hard because they are under-represented, and that mixing difficulties inside a minibatch is a more effective lever than reweighting. It proposes MoMBS, which ranks every training image by both its loss and its prediction uncertainty, adds the two ranks into a single difficulty score, and builds minibatches by pairing high-score images with low-score images. If the paper is right, this one change consistently improves four tasks, universal lesion detection, COVID-19 segmentation, long-tailed classification, and noisy-label classification, with no change to the network architecture.

What carries the argument

The central object is the difficulty rank score $d_i = \pi(i) + \tau(i)$, where $\pi(i)$ is the sample's rank by prediction uncertainty and $\tau(i)$ its rank by loss. The scheduler's aim is to minimize the variance of the per-minibatch sum of difficulty scores, which is achieved by pairing samples with high $d$ against samples with low $d$; this is the mixed-order sampling step. The four-way sample taxonomy (well represented, poorly labeled, under represented, overfitted) is the interpretive device that explains why the pairing works: under-represented samples become the main gradient contributors while poorly labeled or overfitted samples are kept from dominating. The uncertainty estimate is obtained by injecting random disturbances into feature maps and measuring the entropy of the averaged prediction.

What would settle it

Take a benchmark with ground-truth sample-quality labels, such as CIFAR-100N with known clean and noisy labels, and compute the paper's loss-uncertainty rank scores after the pivot epoch; if the noisy-labeled samples do not systematically receive medium scores and clean minority-class samples do not receive high scores, MoMBS's predicted gains should be checked. A direct test: train CIFAR-100-LT with MoMBS and report tail-class accuracy separately; if tail classes (the under-represented samples) do not improve while head-class accuracy stays flat, the central categorization claim is falsified.

Watch

Extended reading notes

Core claim

MoMBS replaces random minibatch construction with a schedule that equalizes total difficulty across minibatches. Samples are ranked by loss and by uncertainty, and the rank indices are summed, $d_i = \pi(i) + \tau(i)$, so that well-represented samples receive low scores, poorly labeled and overfitted samples receive medium scores, and under-represented samples receive high scores. The scheduler then pairs high-$d$ samples with low-$d$ samples, increasing the frequency of minibatches the paper calls positive (notably well-represented paired with under-represented) while reducing negative pairings (such as poorly labeled paired with under-represented). The empirical claim is that this ordering improves accuracy and sensitivity on ULD, COVID-19 segmentation, long-tailed classification, and noisy-label classification beyond SCL, OHEM, and random sampling, with the largest gains at small training-data sizes and small batch sizes.

Load-bearing premise

The method's benefit rests on the assumption that the sum of loss rank and uncertainty rank orders samples so that well-represented samples score lowest, poorly labeled and overfitted samples score in the middle, and under-represented samples score highest; if these ranks do not track true sample quality, the high-low pairing will not create the intended effective minibatches.

Editorial extensions

If this is right

  • Adding MoMBS to a fixed network improves average precision on DeepLesion ULD over random, SCL, and OHEM at 25% and 50% training data, with gains shrinking as data grows.
  • On Seg-C19 COVID-19 segmentation, Dice scores improve over both U-Net and nnU-Net baselines with p-values below 0.05.
  • On CIFAR-100-LT, MoMBS improves accuracy for ResNet-32 and for six existing long-tailed methods including focal loss, class-balanced loss, and LDAM.
  • On CIFAR-100-NL with human and symmetric noise, MoMBS improves ResNet-32 and methods such as GCE and NECE+RCE, with larger gains at batch size 16 than at batch size 32.
  • Because MoMBS only changes minibatch composition and not sample weights, it can be composed with loss-modification, logit-adjustment, and sample-selection methods without conflict.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sample-interaction story is general, MoMBS should transfer to other data-quality regimes where a low-cost difficulty signal exists, such as noisy text or speech, active learning, and semi-supervised learning; this is a direct but untested extension.
  • The method's own visualization concedes misclassification of samples at category boundaries (e.g., b1 in Fig. 8), so the practical ceiling may be set by the assessor rather than the pairing; a sharper uncertainty estimator could extend the gains.
  • The equal-total-difficulty principle suggests a curriculum over batch composition rather than over sample weights, which implies that batch-size sensitivity is not just an implementation detail but a scaling property: larger batches dilute the pairing effect, consistent with the reported results.
  • One could test whether the benefit is actually driven by the well-represented/under-represented positive pair alone: ablating to pair only those two categories, excluding poorly labeled and overfitted samples from training decisions, would isolate the mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MoMBS, a minibatch sampling strategy for training deep networks on datasets with diverse sample quality. The method computes, for each training sample, a difficulty score that is the sum of its loss rank and its uncertainty rank (where uncertainty is measured as the entropy of predictions under feature-map disturbances). The scheduler then constructs minibatches with roughly constant total difficulty by pairing high-difficulty samples with low-difficulty samples. The authors argue that this mixed-order pairing increases the proportion of 'positive' minibatches (e.g., well-represented samples paired with under-represented samples) and decreases 'negative' minibatches, improving gradient updates. They evaluate MoMBS on four tasks: universal lesion detection on DeepLesion, COVID-19 CT segmentation on Seg-C19, long-tailed classification on CIFAR100-LT, and noisy-label classification on CIFAR100-NL, reporting consistent improvements over baselines and ablations.

Significance. If the central empirical claim holds, the paper makes a useful contribution: it shows that minibatch composition itself, not just loss reweighting or sample selection, can be a lever for training with diverse-quality data, and that uncertainty information can refine the notion of sample difficulty. The breadth of evaluation across medical and natural image tasks, and the inclusion of ablations with loss-only, uncertainty-only, and combined difficulty measures, strengthen the empirical case. The paper also ships a clear intuitive taxonomy (poorly labeled, under-represented, well-represented, overfitted) that could transfer to other data-quality problems. However, the significance is tempered by the fact that the algorithm as stated is only defined for batch size 2, while all reported experiments use larger batch sizes, and by the heuristic nature of the theoretical explanation in Section 3.4. The reproducibility and the precision of the claimed mechanism therefore need to be addressed before the findings can be fully credited.

major comments (4)
  1. [Section 3.3, Eq. (7)] The scheduler is defined only for minibatch size b = 2. The paper states 'To simplify our explanation, we set the minibatch size b to b = 2,' but Table 1 and Table 4 report results with batch size 4, and Tables 2 and 3 report results with batch sizes 64, 32, and 16. No extension of Eq. (7) to b > 2 is given, neither in terms of the variance-minimization objective nor in terms of the pairing rule. This is not a cosmetic omission: for b = 4, minimizing the variance of the total difficulty score across minibatches does not imply high-low pairing, because d_i = pi(i) + tau(i) collapses the four hypothesized categories into a single scalar; a batch with two medium-difficulty samples can have the same total d as one high and one low sample. The claimed mechanism of increasing <sw, su> positive minibatches is therefore not guaranteed by the stated method under the actual experimental settings. The authors should specify the exact construction used for b > 2 (e.g., sorting by d and pairing extremes within each batch, or another extension), and should ideally verify on the reported settings that the distribution of minibatch types shifts in the intended direction.
  2. [Section 3.4, Eq. (10) and Fig. 3] The 'update efficacy' explanation rests on unproven assertions. The derivation in Eq. (10) is standard, but the step from 'prediction error often holds more influence' (due to normalization and chain-rule sensitivity) to 'a minibatch with a higher loss value should exhibit greater update efficiency' is not established; the claim that this dominance holds for deep networks with BN and softmax/sigmoid is plausible but not formalized. The subsequent claim that adding uncertainty and using the even-total-difficulty schedule alleviates the limitations of loss-only scheduling is supported only by the qualitative scatter plots of Fig. 3, without any quantitative measure of 'update efficacy' or a comparison to alternative schedules. To make the explanation load-bearing, the authors should define a quantitative update-efficacy metric and show that the MoMBS schedule produces better values than random sampling and anti-mixed-order pairing, not just higher total loss reductions on the same minibatches.
  3. [Section 3.2, Eq. (5), and Section 4.8] The positive/negative minibatch categorization is partly circular with the difficulty measure. The categories are defined by thresholds on the same loss and uncertainty ranks used for pairing, so the claim that MoMBS increases the number of positive minibatches is to some extent true by construction. More importantly, the paper itself concedes in Section 4.8 that CIFAR100-LT and CIFAR100-NL 'align less with our proposed sample categorization' than DeepLesion and Seg-C19. On these tasks, the mechanism is not independently verified: the scatter plots in Fig. 7 are qualitative, and no quantitative measure (e.g., precision/recall of detecting minority-class samples or noisy labels using the proposed categories) is provided. Since the headline claim is that MoMBS consistently improves all four tasks, the authors should quantify how well the taxonomy holds on each dataset, for example by evaluating d_i as a predictor of known minority-class membership (LT) or known label noise (NL).
  4. [Tables 2 and 3, and Section 4.4] The paper lacks error bars for most experiments, which matters for the consistency claim. Tables 2 and 3 report single accuracy numbers for each method and batch size; several improvements are small (e.g., Table 3, GCE+ours at BS=32 improves from 0.590 to 0.594, a 0.4% gain). Table 4 provides p-values, but the text does not state the number of runs, the statistical test used, or whether the p-values are corrected for multiple comparisons. To support the claim that MoMBS 'consistently improves' all four tasks, the authors should report mean and standard deviation over at least three independent runs with different random seeds, and should state the statistical test and number of runs for the p-values in Table 4.
minor comments (6)
  1. [Abstract and Introduction] The acronym 'MoBMS' appears twice in the Abstract and once in the Introduction; it should be 'MoMBS'. Also, 'taskes' in the Introduction should be 'tasks'.
  2. [Eq. (1) and Eq. (2)] The notation for uncertainty is unclear: \(\hat{y}_i^g = F(x_i|w, t_g)\) suggests a single prediction per disturbance, but the entropy in Eq. (1) is applied to the average prediction \(\frac{1}{|G|}\sum_g \hat{y}_i^g\). It would be clearer to define \(\hat{y}_i^g\) as the model output under disturbance \(t_g\) and then define the average over g before computing H.
  3. [Section 3.2, Eq. (3)-(4)] The symbols \(\pi\) and \(\tau\) are used for uncertainty and loss ranks, respectively, but the order of terms in Eq. (4) (d_i = pi(i) + tau(i)) is never explicitly tied back to these definitions in the text. Please clarify which rank corresponds to which measure immediately before Eq. (4).
  4. [Section 3.2, last paragraph] The sentence 'no sample reweighting is used' is misleading: the scheduler changes the composition of minibatches, which effectively changes the sampling distribution of samples per epoch. The authors likely mean that no loss reweighting or sample duplication is applied; please rephrase to avoid this ambiguity.
  5. [Table 1 caption] The abbreviations 'Mo+l', 'Mo+u', and 'Mo+l+u' are used in the table but only fully described in the caption for Table 1; consider defining them in the caption for readers who jump directly to the table.
  6. [Section 2.4 and Section 2.5] The statements 'we can apply our MoMBS to some of them without any conflict' and 'our proposed MoMBS works seamlessly with them' are vague. Please specify the compatibility conditions (e.g., whether the method is applied before or after loss modification, and whether it interacts with methods that change the effective batch composition).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoMBS's sampling objective is used to construct training batches, while all reported gains are measured on external test sets; the positive/negative taxonomy is definitional rather than a fitted prediction.

full rationale

No load-bearing circular reduction is present. The difficulty score d_i = pi(i) + tau(i) (Eq. 4) and the scheduler objective B = arg min_B Var(d(B)) (Eq. 7) are used only to reorder training minibatches; every headline claim is measured on held-out test sets (DeepLesion, Seg-C19, CIFAR-100-LT/NL) against fixed baselines, so no fitted parameter is renamed as a prediction. The positive/negative minibatch taxonomy is a definition, and the statement that high-low pairing increases positive minibatches follows mathematically from the ordering d(sw) < d(sp) or d(so) < d(su) (Eq. 5) rather than being an empirically predicted quantity. The update-efficacy argument in Sec. 3.4 is a direct gradient/CE-loss derivation, not an input assumption. The anti-mixed-order ablation (Mo with hi+hi) provides an independent contrast for the sampling mechanism. The only self-citations ([144], [150]) are used as baseline methods, not as justification for the central claim. The paper does concede in Sec. 4.8 that CIFAR100-LT and CIFAR100-NL 'align less with our proposed sample categorization', and the scheduler is formally defined only for b=2 while experiments use batch sizes 4-64 without stating the extension; these are generality/reproducibility limitations, not circularity by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on hand-chosen hyperparameters (pivot epoch, disturbance magnitude and count) and on assumptions about the validity of loss and uncertainty ranks and feature-map noise as difficulty and robustness signals. No parameters are fitted to test data; the central empirical claim is benchmark accuracy.

free parameters (3)
  • Pivot epoch e = chosen per dataset, e.g., 50 for SATr 25% DeepLesion, 20 for CIFAR100-LT, 30 for CIFAR100-NL
    The uncertainty estimator is activated after e epochs; the ablation in Fig. 5 shows strong sensitivity to e, and the main results appear to use the best-performing e per task.
  • Disturbance magnitude gamma = 0.3
    Uniform noise U[-gamma, +gamma] is added to feature maps; no sensitivity analysis is provided.
  • Number of disturbances G = 8
    Number of noisy forward passes averaged for uncertainty; chosen without sensitivity analysis.
assumptions (5)
  • domain assumption Loss and uncertainty ranks provide a valid difficulty ordering of training samples for diverse-quality data.
    The entire scheduler is built on di = pi(i) + tau(i) and the ordering d(sw) < d(sp/so) < d(su) in Eq. 5.
  • domain assumption Adding uniform noise to feature maps approximates parameter-level perturbations for uncertainty estimation.
    Section 3.2(i) states noise is added to key feature maps because input noise does not alter output; the equivalence to parameter changes is assumed.
  • ad hoc to paper In deep networks, prediction error (y_hat_i - y_i) dominates the gradient over the data term (dz/dw).
    Section 3.4 uses this to argue loss measures update efficacy; the paper cites normalization and chain-rule sensitivity but provides no proof.
  • ad hoc to paper A balanced total difficulty across minibatches yields better updates than random or extreme pairings.
    This is the design goal of Eq. 7; it is validated empirically through ablations but not derived.
  • domain assumption The sigmoid and softmax cross-entropy model approximately represents the ULD and segmentation networks used in experiments.
    Section 3.4's derivation applies to a simple binary classifier, but the method is applied to two-stage detectors and segmentation networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images." pith.science (2026). https://pith.science/paper/BS6UQPX3

@misc{pith2026250518741,
  author       = {Pith},
  title        = {Pith review of: MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BS6UQPX3}},
  note         = {Machine review of arXiv:2505.18741}
}
read the original abstract

Natural images exhibit label diversity (clean vs. noisy) in noisy-labeled image classification and prevalence diversity (abundant vs. sparse) in long-tailed image classification. Similarly, medical images in universal lesion detection (ULD) exhibit substantial variations in image quality, encompassing attributes such as clarity and label correctness. How to effectively leverage training images with diverse qualities becomes a problem in learning deep models. Conventional training mechanisms, such as self-paced curriculum learning (SCL) and online hard example mining (OHEM), relieve this problem by reweighting images with high loss values. Despite their success, these methods still confront two challenges: (i) the loss-based measure of sample hardness is imprecise, preventing optimum handling of different cases, and (ii) there exists under-utilization in SCL or over-utilization OHEM with the identified hard samples. To address these issues, this paper revisits the minibatch sampling (MBS), a technique widely used in deep network training but largely unexplored concerning the handling of diverse-quality training samples. We discover that the samples within a minibatch influence each other during training; thus, we propose a novel Mixed-order Minibatch Sampling (MoMBS) method to optimize the use of training samples with diverse qualities. MoMBS introduces a measure that takes both loss and uncertainty into account to surpass a sole reliance on loss and allows for a more refined categorization of high-loss samples by distinguishing them as either poorly labeled and under represented or well represented and overfitted. We prioritize under represented samples as the main gradient contributors in a minibatch and keep them from the negative influences of poorly labeled or overfitted samples with a mixed-order minibatch sampling design.

Figures

Figures reproduced from arXiv: 2505.18741 by the authors.

Figure 1
Figure 1. Left: Training samples are typically grouped into four types based on data loss and uncertainty: (sp) poorly labeled, (su) under represented, (sw) well represented and (so) overfitted samples. The loss-based sample quality measurer, employed in self-paced curriculum learning (SCL) and online hard example mining (OHEM), inaccurately treats both (sp) and (su) as low-quality samples. Right: The distribution comparison … view at source ↗
Figure 2
Figure 2. Our MoMBS method consists of three learning steps. In step A, we randomly sample data using the vanilla random sampler for initial network [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a1): Total loss reduction after one iteration backward for two samples in a minibatch vs. their individual loss values. (b1): Loss reduc￾tion after one iteration backward of the sample with the lower loss in a minibatch vs. their individual loss values. (a2): Total loss reduction after one iteration backward vs. the sum of their individual loss and uncertainty values. (b2): Loss reduction after one iteration backwa… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Performance improvements with different batchsizes settings [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Illustration of loss and uncertainty relationship based on [150]. Yellow (or cyan) denotes the sample whose absolute difference between [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: UPPER: Loss vs. uncertainty scatter plots of LT samples (a1) [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Eight samples from four data types. Based on loss-based dif [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

161 extracted references · 75 canonical work pages

  1. [1]

    Liu et al

    Z. Liu et al. Self-paced ensemble for highly imbalanced massive data classification. In ICDE, pages 841–852. IEEE, 2020

  2. [2]

    Wang et al

    X. Wang et al. A survey on curriculum learning. IEEE T rans. Pattern Anal. Mach. Intell. , 2021

  3. [3]

    Kumar et al

    M. Kumar et al. Self-paced learning for latent variable models. Proc. Adv. Neural Inf. Process. Syst. , 23, 2010

  4. [4]

    Liu et al

    J. Liu et al. Co-correcting: noise-tolerant medical image clas- sification via mutual label correction. IEEE T rans. Med. Imag. , 40(12):3580–3592, 2021

  5. [5]

    Ju et al

    L. Ju et al. Improving medical images classification with label noise using dual-uncertainty estimation. IEEE T rans. Med. Imag., 41(6):1533–1546, 2022

  6. [6]

    Jiang et al

    L. Jiang et al. Easy samples first: Self-paced reranking for zero- example multimedia search. In ACM MM, pages 547–556, 2014

  7. [7]

    Zhao et al

    Q. Zhao et al. Self-paced learning for matrix factorization. In IEEE/CVF AAAI, 2015

  8. [8]

    Xu et al

    C. Xu et al. Multi-view self-paced learning for clustering. In IJCAI, 2015

Show all 161 references
  1. [9]

    Gong et al

    M. Gong et al. Decomposition-based evolutionary multiobjec- tive optimization to self-paced learning. IEEE T rans. on Evolut. Computa., 23(2):288–302, 2018

  2. [10]

    Warburg et al

    F. Warburg et al. Bayesian triplet loss: Uncertainty quantification in image retrieval. In IEEE/CVF ICCV, pages 12158–12168, 2021

  3. [11]

    Mao et al

    Y. Mao et al. Uasnet: Uncertainty adaptive sampling network for deep stereo matching. In IEEE/CVF ICCV, pages 6311–6319, 2021

  4. [12]

    Dokuz et al

    Y. Dokuz et al. Mini-batch sample selection strategies for deep learning based speech recognition. Applied Acoustics, 171:107573, 2021. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, 2023 14

  5. [13]

    Liu et al

    F. Liu et al. Acpl: Anti-curriculum pseudo-labelling for semi- supervised medical image classification. In IEEE/CVF CVPR , pages 20697–20706, 2022

  6. [14]

    Morerio et al

    P . Morerio et al. Curriculum dropout. In IEEE/CVF ICCV, pages 3544–3552, 2017

  7. [15]

    Binkowski et al

    M. Binkowski et al. Batch weight for domain adaptation with mass shift. In IEEE/CVF ICCV, pages 1844–1853, 2019

  8. [16]

    Kong et al

    Y. Kong et al. Adaptive curriculum learning. In IEEE/CVF ICCV, pages 5067–5076, 2021

  9. [17]

    Wang et al

    Y. Wang et al. Dynamic curriculum learning for imbalanced data classification. In IEEE/CVF ICCV, pages 5017–5026, 2019

  10. [18]

    Meng et al

    D. Meng et al. A theoretical understanding of self-paced learning. Information Sciences, 414:319–328, 2017

  11. [19]

    Y. J. Lee et al. Learning the easy things first: Self-paced visual category discovery. In IEEE/CVF CVPR , pages 1721–1728. IEEE, 2011

  12. [20]

    M. P . Kumar et al. Learning specific-class segmentation from diverse data. In IEEE/CVF ICCV, pages 1800–1807. IEEE, 2011

  13. [21]

    Yang et al

    M. Yang et al. Su-micl: Severity-guided multiple instance cur- riculum learning for histopathology image interpretable classifi- cation. IEEE T rans. Med. Imag., 41(12):3533–3543, 2022

  14. [22]

    Tang et al

    Y. Tang et al. Self-paced dictionary learning for image classifica- tion. In ACM MM, pages 833–836, 2012

  15. [23]

    Tang et al

    K. Tang et al. Shifting weights: Adapting object detectors from image to video. Proc. Adv. Neural Inf. Process. Syst. , 25, 2012

  16. [24]

    Zhang et al

    D. Zhang et al. Leveraging prior-knowledge for weakly super- vised object detection under a collaborative self-paced curricu- lum learning framework. Int. J. Comput. Vis., 127(4):363–380, 2019

  17. [25]

    Zhou et al

    S. Zhou et al. Deep self-paced learning for person re- identification. Pattern Recognition, 76:739–751, 2018

  18. [26]

    Han et al

    J. Han et al. Weakly-supervised learning of category-specific 3d object shapes. IEEE T rans. Pattern Anal. Mach. Intell. , 43(4):1423– 1437, 2019

  19. [27]

    Ghasedi et al

    K. Ghasedi et al. Balanced self-paced learning for generative adversarial clustering network. In IEEE/CVF CVPR, pages 4391– 4400, 2019

  20. [28]

    Gong et al

    C. Gong et al. Multi-modal curriculum learning for semi- supervised image classification. IEEE T rans. on Imag. Proces. , 25(7):3249–3260, 2016

  21. [29]

    Zhang et al

    D. Zhang et al. A self-paced multiple-instance learning frame- work for co-saliency detection. In IEEE/CVF ICCV , pages 594– 602, 2015

  22. [30]

    Li et al

    C. Li et al. Self-paced multi-task learning. In IEEE/CVF AAAI , 2017

  23. [31]

    Lin et al

    L. Lin et al. Active self-paced learning for cost-effective and progressive face identification. IEEE T rans. Pattern Anal. Mach. Intell., 40(1):7–19, 2017

  24. [32]

    Tang et al

    Y. Tang et al. Self-paced active learning: Query the right thing at the right time. In IEEE/CVF AAAI, volume 33, pages 5117–5124, 2019

  25. [33]

    D. MacKay. A practical bayesian framework for backpropagation networks. Neural Comput, 4(3):448–472, 1992

  26. [34]

    Gal and Z

    Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InICML, pages 1050–1059. PMLR, 2016

  27. [35]

    Teye et al

    M. Teye et al. Bayesian uncertainty estimation for batch normal- ized deep networks. In ICML, pages 4907–4916. PMLR, 2018

  28. [36]

    Lakshminarayanan et al

    B. Lakshminarayanan et al. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv:1612.01474, 2016

  29. [37]

    Corbi `ere et al

    C. Corbi `ere et al. Confidence estimation via auxiliary models. IEEE T rans. Pattern Anal. Mach. Intell., 44(10):6043–6055, 2022

  30. [38]

    Yang et al

    W. Yang et al. Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localiza- tion. IEEE T rans. Pattern Anal. Mach. Intell., 45(4):5252–5267, 2023

  31. [39]

    Zhou et al

    T. Zhou et al. Consistency and diversity induced human motion segmentation. IEEE T rans. Pattern Anal. Mach. Intell. , 45(1):197– 210, 2023

  32. [40]

    Tang et al

    B. Tang et al. Collaborative uncertainty benefits multi-agent multi-modal trajectory forecasting. IEEE T rans. Pattern Anal. Mach. Intell., 45(11):13297–13313, 2023

  33. [41]

    Xia et al

    J. Xia et al. Robust face alignment via inherent relation learning and uncertainty estimation. IEEE T rans. Pattern Anal. Mach. Intell., 45(8):10358–10375, 2023

  34. [42]

    Yan et al

    X. Yan et al. Ensemble multi-quantiles: Adaptively flexible distribution prediction for uncertainty quantification. IEEE T rans. Pattern Anal. Mach. Intell. , 45(11):13068–13082, 2023

  35. [43]

    Shen et al

    Z. Shen et al. Digging into uncertainty-based pseudo-label for robust stereo matching. IEEE T rans. Pattern Anal. Mach. Intell. , 45(12):14301–14320, 2023

  36. [44]

    Peng et al

    X. Peng et al. Out-of-domain generalization from a single source: An uncertainty quantification approach. IEEE T rans. Pattern Anal. Mach. Intell., pages 1–13, 2022

  37. [45]

    Franchi et al

    G. Franchi et al. Encoding the latent posterior of bayesian neural networks for uncertainty quantification. IEEE T rans. Pattern Anal. Mach. Intell., pages 1–13, 2023

  38. [46]

    Won et al

    C. Won et al. End-to-end learning for omnidirectional stereo matching with uncertainty prior. IEEE T rans. Pattern Anal. Mach. Intell., 43(11):3850–3862, 2021

  39. [47]

    Yang et al

    G. Yang et al. Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEEE T rans. Pattern Anal. Mach. Intell., 45(2):2567–2581, 2023

  40. [48]

    Training region-based object detectors with online hard example mining

    Abhinav Shrivastava et al. Training region-based object detectors with online hard example mining. In IEEE/CVF CVPR , pages 761–769, 2016

  41. [49]

    Dong et al

    Q. Dong et al. Class rectification hard mining for imbalanced deep learning. In IEEE/CVF ICCV, 2017

  42. [50]

    Sun et al

    H. Sun et al. Mvp matching: A maximum-value perfect match- ing for mining hard samples, with application to person re- identification. In IEEE/CVF ICCV, 2019

  43. [51]

    He et al

    J. He et al. Online hard patch mining using shape models and bandit algorithm for multi-organ segmentation. IEEE J. Biomed. Health Inform., 26(6):2648–2659, 2021

  44. [52]

    Xu et al

    H. Xu et al. Two-stream region convolutional 3d network for temporal activity detection. IEEE T rans. Pattern Anal. Mach. Intell., 41(10):2319–2332, 2019

  45. [53]

    Zhang et al

    Y. Zhang et al. Deep long-tailed learning: A survey. IEEE T rans. Pattern Anal. Mach. Intell. , 45(9):10795–10816, 2023

  46. [54]

    Li et al

    M. Li et al. Key point sensitive loss for long-tailed visual recognition. IEEE T rans. Pattern Anal. Mach. Intell. , 45(4):4812– 4825, 2023

  47. [55]

    Tan et al

    J. Tan et al. The equalization losses: Gradient-driven training for long-tailed object recognition. IEEE T rans. Pattern Anal. Mach. Intell., 45(11):13876–13892, 2023

  48. [56]

    Cui et al

    J. Cui et al. Reslt: Residual learning for long-tailed recognition. IEEE T rans. Pattern Anal. Mach. Intell., 45(3):3695–3706, 2023

  49. [57]

    Hou et al

    R. Hou et al. Dual compensation residual networks for class imbalanced learning. IEEE T rans. Pattern Anal. Mach. Intell. , 45(10):11733–11752, 2023

  50. [58]

    Zhou et al

    H. Zhou et al. Debiased scene graph generation for dual imbal- ance learning. IEEE T rans. Pattern Anal. Mach. Intell. , 45(4):4274– 4288, 2023

  51. [59]

    Jiang et al

    S. Jiang et al. Dynamic loss for robust learning. IEEE T rans. Pattern Anal. Mach. Intell. , 45(12):14420–14434, 2023

  52. [60]

    Lin et al

    T. Lin et al. Focal loss for dense object detection. In IEEE/CVF ICCV, pages 2980–2988, 2017

  53. [61]

    Ren et al

    M. Ren et al. Learning to reweight examples for robust deep learning. In ICML, pages 4334–4343. PMLR, 2018

  54. [62]

    Cui et al

    Y. Cui et al. Class-balanced loss based on effective number of samples. In IEEE/CVF CVPR, pages 9268–9277, 2019

  55. [63]

    Huang et al

    C. Huang et al. Learning deep representation for imbalanced classification. In IEEE/CVF CVPR, pages 5375–5384, 2016

  56. [64]

    Khan et al

    S. Khan et al. Cost-sensitive learning of deep feature represen- tations from imbalanced data. IEEE T rans Neural Networks Learn. Syst., 29(8):3573–3587, 2017

  57. [65]

    Tan et al

    J. Tan et al. Equalization loss for long-tailed object recognition. In IEEE/CVF CVPR, pages 11662–11671, 2020

  58. [66]

    Park et al

    S. Park et al. Influence-balanced loss for imbalanced visual classification. In IEEE/CVF ICCV, pages 735–744, 2021

  59. [67]

    Li et al

    M. Li et al. Long-tailed visual recognition via gaussian clouded logit adjustment. In IEEE/CVF CVPR, pages 6929–6938, 2022

  60. [68]

    Menon et al

    A. Menon et al. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314, 2020

  61. [69]

    Hong et al

    Y. Hong et al. Disentangling label distribution for long-tailed visual recognition. In IEEE/CVF CVPR, pages 6626–6636, 2021

  62. [70]

    Tang et al

    K. Tang et al. Long-tailed classification by keeping the good and removing the bad momentum causal effect. 33:1513–1524, 2020

  63. [71]

    Zhang et al

    S. Zhang et al. Distribution alignment: A unified framework for long-tail visual recognition. In IEEE/CVF CVPR, pages 2361–2370, 2021

  64. [72]

    Cao et al

    D. Cao et al. Domain balancing: Face recognition on long-tailed domains. In IEEE/CVF CVPR, pages 5671–5679, 2020

  65. [73]

    Cao et al

    K. Cao et al. Learning imbalanced datasets with label- distribution-aware margin loss. 32, 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, 2023 15

  66. [74]

    Wang et al

    T. Wang et al. The devil is in classification: A simple framework for long-tail instance segmentation. In IEEE/CVF ECCV , pages 728–744, 2020

  67. [75]

    Zhong et al

    Z. Zhong et al. Improving calibration for long-tailed recognition. In IEEE/CVF CVPR, pages 16489–16498, 2021

  68. [76]

    Zhou et al

    B. Zhou et al. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In IEEE/CVF CVPR , pages 9719–9728, 2020

  69. [77]

    Wang et al

    P . Wang et al. Contrastive learning based hybrid networks for long-tailed image classification. In IEEE/CVF CVPR , pages 943– 952, 2021

  70. [78]

    Song et al

    H. Song et al. Learning from noisy labels with deep neural networks: A survey. IEEE T rans. Neural Networks Learn. Syst. , 34(11):8135–8153, 2023

  71. [79]

    Xiao et al

    T. Xiao et al. Learning from massive noisy labeled data for image classification. In IEEE/CVF CVPR, pages 2691–2699, 2015

  72. [80]

    Chen et al

    X. Chen et al. Webly supervised learning of convolutional networks. In IEEE/CVF ICCV, pages 1431–1439, 2015

  73. [81]

    Goldberger et al

    J. Goldberger et al. Training deep neural-networks using a noise adaptation layer. In ICLR, 2016

  74. [82]

    Han et al

    B. Han et al. Masking: A new perspective of noisy supervision. 31, 2018

  75. [83]

    Cheng et al

    L. Cheng et al. Weakly supervised learning with side information for noisy labeled images. In IEEE/CVF ECCV , pages 306–321. Springer, 2020

  76. [84]

    Jindal et al

    I. Jindal et al. Learning deep networks from noisy labels with dropout regularization. In IEEE ICDM, pages 967–972, 2016

  77. [85]

    Lee et al

    K. Lee et al. Robust inference via generative classifiers for handling noisy labels. In ICML, pages 3763–3772. PMLR, 2019

  78. [86]

    Zhou et al

    X. Zhou et al. Asymmetric loss functions for noise-tolerant learning: Theory and applications. IEEE T rans. Pattern Anal. Mach. Intell., 45(7):8094–8109, 2023

  79. [87]

    Xia et al

    X. Xia et al. Extended tt: Learning with mixed closed-set and open-set noisy labels. IEEE T rans. Pattern Anal. Mach. Intell. , 45(3):3047–3058, 2023

  80. [88]

    Xie et al

    M. Xie et al. Ccmn: A general framework for learning with class- conditional multi-label noise. IEEE T rans. Pattern Anal. Mach. Intell., 45(1):154–166, 2023

  81. [89]

    Gong et al

    C. Gong et al. Class-wise denoising for robust learning under label noise. IEEE T rans. Pattern Anal. Mach. Intell. , 45(3):2835– 2848, 2023

  82. [90]

    Fatras et al

    K. Fatras et al. Wasserstein adversarial regularization for learn- ing with label noise. IEEE T rans. Pattern Anal. Mach. Intell. , 44(10):7296–7306, 2022

  83. [91]

    Yang et al

    S. Yang et al. A parametrical model for instance-dependent label noise. IEEE T rans. Pattern Anal. Mach. Intell. , 45(12):14055–14068, 2023

  84. [92]

    Tanno et al

    R. Tanno et al. Learning from noisy labels by regularized estimation of annotator confusion. In IEEE/CVF CVPR , pages 11244–11253, 2019

  85. [93]

    Menon et al

    A. Menon et al. Can gradient clipping mitigate label noise? In ICLR, 2019

  86. [94]

    Xia et al

    X. Xia et al. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2020

  87. [95]

    Wei et al

    H. Wei et al. Open-set label noise can improve robustness against inherent label noise. 34:7978–7992, 2021

  88. [96]

    Pereyra et al

    G. Pereyra et al. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017

  89. [97]

    Lukasik et al

    M. Lukasik et al. Does label smoothing mitigate label noise? In ICML, pages 6448–6458. PMLR, 2020

  90. [98]

    Wang et al

    Y. Wang et al. Symmetric cross entropy for robust learning with noisy labels. In IEEE/CVF ICCV, pages 322–330, 2019

  91. [99]

    Feng et al

    L. Feng et al. Can cross entropy loss be robust to label noise? In IJCAI, pages 2206–2212, 2021

  92. [100]

    Liu et al

    Y. Liu et al. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, pages 6226–6236. PMLR, 2020

  93. [101]

    Amid et al

    E. Amid et al. Robust bi-tempered logistic loss based on bregman divergences. 32, 2019

  94. [102]

    Ma et al

    X. Ma et al. Normalized loss functions for deep learning with noisy labels. In ICML, pages 6543–6553. PMLR, 2020

  95. [103]

    Arazo et al

    E. Arazo et al. Unsupervised label noise modeling and loss correction. In ICML, pages 312–321. PMLR, 2019

  96. [104]

    Yao et al

    Y. Yao et al. Dual t: Reducing estimation error for transition matrix in label-noise learning. 33:7260–7271, 2020

  97. [105]

    Liu et al

    T. Liu et al. Classification with noisy labels by importance reweighting. IEEE T rans. Pattern Anal. Mach. Intell., 38(3):447–461, 2015

  98. [106]

    Zhang et al

    H. Zhang et al. Dualgraph: A graph-based method for reasoning about label noise. In IEEE/CVF CVPR, pages 9654–9663, 2021

  99. [107]

    Zheng et al

    S. Zheng et al. Error-bounded correction of noisy labels. In IEEE/CVF CVPR, pages 2751–2760, 2020

  100. [108]

    Chen et al

    P . Chen et al. Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise. In IEEE/CVF AAAI, volume 35, pages 11442–11450, 2021

  101. [109]

    Shu et al

    J. Shu et al. Meta-weight-net: Learning an explicit mapping for sample weighting. 32, 2019

  102. [110]

    Zhang et al

    Z. Zhang et al. Distilling effective supervision from severe label noise. In IEEE/CVF CVPR, pages 9294–9303, 2020

  103. [111]

    Li et al

    Y. Li et al. Learning from noisy labels with distillation. In IEEE/CVF ICCV, pages 1910–1918, 2017

  104. [112]

    Zheng et al

    G. Zheng et al. Meta label correction for noisy label learning. In IEEE/CVF AAAI, volume 35, pages 11053–11061, 2021

  105. [113]

    Song et al

    H. Song et al. Selfie: Refurbishing unclean samples for robust deep learning. In ICML, pages 5907–5915, 2019

  106. [114]

    Chen et al

    P . Chen et al. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pages 1062–1070, 2019

  107. [115]

    Song et al

    H. Song et al. Robust learning by self-transition for handling noisy labels. In KDD, pages 1490–1500, 2021

  108. [116]

    Nguyen et al

    D. Nguyen et al. Self: Learning to filter noisy labels with self- ensembling. arXiv preprint arXiv:1910.01842 , 2019

  109. [117]

    Li et al

    J. Li et al. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2019

  110. [118]

    Wei et al

    H. Wei et al. Combating noisy labels by agreement: A joint training method with co-regularization. In IEEE/CVF CVPR , pages 13726–13735, 2020

  111. [119]

    Huang et al

    J. Huang et al. O2u-net: A simple noisy label detection approach for deep neural networks. In IEEE/CVF ICCV, pages 3326–3334, 2019

  112. [120]

    Wu et al

    P . Wu et al. A topological filter for learning with label noise. 33:21382–21393, 2020

  113. [121]

    Wu et al

    Z. Wu et al. Ngc: A unified framework for learning with open- world noisy data. In IEEE/CVF ICCV, pages 62–71, 2021

  114. [122]

    Zhou et al

    T. Zhou et al. Robust curriculum learning: From clean label detection to noisy label self-correction. In ICLR, 2020

  115. [123]

    Zeng et al

    L. Zeng et al. Ss-tbn: A semi-supervised tri-branch network for covid-19 screening and lesion segmentation. IEEE T rans. Pattern Anal. Mach. Intell. , 45(8):10427–10442, 2023

  116. [124]

    Liu et al

    Y. Liu et al. Structural attention graph neural network for diagnosis and prediction of covid-19 severity. IEEE T rans. Med. Imag., 42(2):557–567, 2023

  117. [125]

    Lyu et al

    F. Lyu et al. Pseudo-label guided image synthesis for semi- supervised covid-19 pneumonia infection segmentation. IEEE T rans. Med. Imag., 42(3):797–809, 2023

  118. [126]

    Cohen Hochberg et al

    D. Cohen Hochberg et al. A self supervised stylegan for image annotation and classification with extremely limited labels. IEEE T rans. Med. Imag., 41(12):3509–3519, 2022

  119. [127]

    Cao et al

    Y. Cao et al. Longitudinal assessment of covid-19 using a deep learning–based quantitative ct pipeline: illustration of two cases. Radiol.: Cardiothorac. Imaging , 2(2):e200082, 2020

  120. [128]

    Huang et al

    L. Huang et al. Serial quantitative chest ct assessment of covid-19: Deep-learning approach. Radiol.: Cardiothorac. Imaging , 2(2):e200075, 2020

  121. [129]

    Wang et al

    G. Wang et al. A noise-robust framework for automatic segmen- tation of covid-19 pneumonia lesions from ct images. IEEE T rans. Med. Imag., 39(8):2653–2663, 2020

  122. [130]

    Fan et al

    D. Fan et al. Inf-net: Automatic covid-19 lung infection segmen- tation from ct images. IEEE T rans. Med. Imag., 2020

  123. [131]

    Zhou et al

    L. Zhou et al. A rapid, accurate and machine-agnostic segmenta- tion and quantification method for ct-based covid-19 diagnosis. IEEE T rans. Med. Imag., 39(8):2638–2652, 2020

  124. [132]

    Wang et al

    X. Wang et al. A weakly-supervised framework for covid-19 classification and lesion localization from chest ct. IEEE T rans. Med. Imag., 39(8):2615–2625, 2020

  125. [133]

    Wu et al

    Y. Wu et al. Jcs: An explainable covid-19 diagnosis system by joint classification and zsegmentation. IEEE T rans Image Process, 30:3113–3126, 2021

  126. [134]

    Liu et al

    J. Liu et al. Covid-19 lung infection segmentation with a novel two-stage cross-domain transfer learning framework. Med Image Anal, 74:102205, 2021

  127. [135]

    Yao et al

    Q. Yao et al. Label-free segmentation of covid-19 lesions in lung ct. IEEE T rans. Med. Imag., 2021. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, 2023 16

  128. [136]

    Zhang et al

    N. Zhang et al. 3d aggregated faster R-CNN for general lesion detection. arXiv:2001.11071, 2020

  129. [137]

    Yan et al

    K. Yan et al. 3d context enhanced region-based convolutional neural network for end-to-end lesion detection. In MICCAI, pages 511–519. Springer, 2018

  130. [138]

    Li et al

    Z. Li et al. Mvp-net: Multi-view fpn with position-aware atten- tion for deep universal lesion detection. In MICCAI, pages 13–21, 2019

  131. [139]

    Yan et al

    K. Yan et al. Mulan: Multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation. In MICCAI, pages 194–202. Springer, 2019

  132. [140]

    Yang et al

    J. Yang et al. Alignshift: bridging the gap of imaging thickness in 3d anisotropic volumes. In MICCAI, pages 562–572. Springer, 2020

  133. [141]

    Zhang et al

    S. Zhang et al. Revisiting 3d context modeling with supervised pre-training for universal lesion detection in ct slices. In MICCAI, pages 542–551, 2020

  134. [142]

    Tang et al

    Y. Tang et al. Weakly-supervised universal lesion segmentation with regional level set loss. In MICCAI, pages 515–525. Springer, 2021

  135. [143]

    Yang et al

    J. Yang et al. Asymmetric 3d context fusion for universal lesion detection. In MICCAI, pages 571–580. Springer, 2021

  136. [144]

    Li et al

    H. Li et al. Conditional training with bounding map for universal lesion detection. In MICCAI, pages 141–152. Springer, 2021

  137. [145]

    Lyu et al

    F. Lyu et al. A segmentation-assisted model for universal lesion detection with partial labels. In MICCAI, pages 117–127. Springer, 2021

  138. [146]

    Cai et al

    J. Cai et al. Deep volumetric universal lesion detection using light-weight pseudo 3d convolution and surface point regression. In MICCAI, pages 3–13. Springer, 2020

  139. [147]

    Ouali et al

    Y. Ouali et al. Semi-supervised semantic segmentation with cross- consistency training. In IEEE/CVF CVPR , pages 12674–12684, 2020

  140. [148]

    Yan et al

    K. Yan et al. Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In IEEE/CVF CVPR , pages 9261–9270, 2018

  141. [149]

    Cai et al

    J. Cai et al. Lesion-harvester: Iteratively mining unlabeled lesions and hard-negative examples at scale. IEEE T rans. Med. Imag. , 40(1):59–70, 2020

  142. [150]

    Li et al

    H. Li et al. Satr: Slice attention with transformer for universal lesion detection. In MICCAI, pages 163–174. Springer, 2022

  143. [151]

    Qiao et al

    P . Qiao et al. Semi-supervised ct lesion segmentation using uncertainty-based data pairing and swapmix. IEEE T rans. Med. Imag., 2022

  144. [152]

    Wei et al

    J. Wei et al. Learning with noisy labels revisited: A study using real-world human annotations. In ICLR, 2021

  145. [153]

    Krizhevsky et al

    A. Krizhevsky et al. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of T oronto, 2009

  146. [154]

    Targ et al

    S. Targ et al. Resnet in resnet: Generalizing residual architectures. arXiv:1603.08029, 2016

  147. [155]

    Han et al

    X. Han et al. General greedy de-bias learning. IEEE T rans. Pattern Anal. Mach. Intell. , 45(8):9789–9805, 2023

  148. [156]

    Kim et al

    Y. Kim et al. Nlnl: Negative learning for noisy labels. In IEEE/CVF ICCV, pages 101–110, 2019

  149. [157]

    Zhang et al

    Z. Zhang et al. Generalized cross entropy loss for training deep neural networks with noisy labels. 31, 2018

  150. [158]

    Li et al

    X. Li et al. H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE T rans. Med. Imag., 37(12):2663–2674, 2018

  151. [159]

    Ronneberger et al

    O. Ronneberger et al. U-net: Convolutional networks for biomed- ical image segmentation. In MICCAI, pages 234–241. Springer, 2015

  152. [160]

    Isensee et al

    F. Isensee et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods , 18(2):203–211, 2021. Han Li received the B.E. degree from Henan Polytechnic University (HPU) in 2016, and the M.S. degree in computer science from the In- ...

  153. [2011]

    Hand- book of Medical Image Computing and Computer Assisted Intervention, SK Zhou, D Rueckert, G Fichtinger (Eds.)

    He is a Professor with ICT, CAS. He was a Research Associate with PRIP Lab, Michigan State University, East Lansing, MI, USA, and a Visiting Researcher with Google, Mountain View, CA, USA, from 2011 to 2015. He has pub- lished more than 70 papers in journals and con- ferences,...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.