The reviewed record of science sign in
Pith

arxiv: 2606.29720 · v1 · pith:TSBTQFEV · submitted 2026-06-29 · cs.LG · cs.CL

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

Zewen Liu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-30 07:51 UTCgrok-4.3pith:TSBTQFEVrecord.jsonopen to challenge →

classification cs.LG cs.CL
keywords imbalanced classificationprobability calibrationSMOTErandom undersamplingtree ensemblesexpected calibration errorresampling methodsPlatt scaling
0
0 comments X

The pith

Resampling for class imbalance harms probability calibration in tree ensembles, with random undersampling causing far more damage than SMOTE.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests how SMOTE and random undersampling change the reliability of probability estimates from random forests and gradient boosting on five imbalanced datasets. It finds SMOTE raises expected calibration error by only 0.009 on average across imbalance ratios from 1.9 to 70, while undersampling can raise it from 0.008 to 0.395 at the highest ratios because the reduced training sets become too small for stable probability estimates. A single post-hoc recalibration step largely removes the damage in both cases. The prior-shift correction that fixes undersampling does not work for SMOTE because SMOTE also changes the class-conditional densities.

Core claim

Across the studied datasets and models, SMOTE's calibration cost is real but small (ECE +0.009; Cliff's delta = +0.27) and its discrimination gains typically outweigh the penalty, while random undersampling's damage grows sharply with imbalance and inflates ECE up to 0.395; a single post-hoc recalibration step eliminates most of the damage with negligible ranking loss, and the analytic prior-shift correction does not transfer to SMOTE.

What carries the argument

Expected calibration error (ECE) tracked before and after each resampling method on tree ensembles, paired with AUC to separate calibration from discrimination effects.

If this is right

  • SMOTE's discrimination gains usually exceed its small calibration penalty across the tested imbalance range.
  • Post-hoc recalibration (Platt or isotonic) reduces ECE by up to 66 percent with AUC drop of only 0.002.
  • The prior-shift correction that repairs undersampling fails for SMOTE because SMOTE distorts class-conditional densities, not only the prior.
  • Imbalanced-learning studies should report calibration metrics alongside discrimination metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Applications that use predicted probabilities for decisions should apply recalibration after any resampling step as routine practice.
  • The pattern suggests that at extreme imbalance ratios, undersampling may need to be avoided entirely if reliable probabilities are required.
  • Similar calibration comparisons could be run on non-tree models to check whether the resampling effects generalize.

Load-bearing premise

The five public datasets and two ensemble models are representative enough that the observed calibration effects are driven by the resampling mechanisms rather than dataset-specific artifacts.

What would settle it

Re-running the experiment on a new dataset with imbalance ratio near 70 and observing whether random undersampling still raises ECE from near 0.008 to near 0.395 while SMOTE stays near +0.009.

Figures

Figures reproduced from arXiv: 2606.29720 by Zewen Liu.

Figure 1
Figure 1. Figure 1: ECE across conditions (mean ± SEM). Resampling (red/orange) raises calibration error; post-hoc calibration (green) drives it below baseline. B. H1: Resampling degrades calibration All three resampling families significantly increase ECE versus baseline (Wilcoxon p < 10−3 for all three families; all five primary comparisons survive Holm-Bonferroni cor￾rection; [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Reliability diagram (credit-g). The uncalibrated SMOTE model (red) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ECE versus SMOTE oversampling ratio ρ (gradient boosting, ten seeds). Calibration error increases monotonically with the amount of synthetic minority data; the dashed line is the across-dataset mean. while HGB calibration worsens (baseline ECE 0.061 → 0.125; SMOTE 0.133 → 0.121). Boosting’s poor calibration is inher￾ent to the algorithm, not an underfitting issue—and resampling compounds it regardless of n… view at source ↗
Figure 5
Figure 5. Figure 5: ECE inflation over baseline versus imbalance ratio (log scale). [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean |SHAP| feature importance for the top-10 baseline features, baseline vs SMOTE (credit-g, gradient boosting). The attribution ranking is essentially preserved (Spearman ρ = 0.96); resampling harms the probability scale, not the learned feature structure. VI. A NEGATIVE RESULT: PRIOR CORRECTION DOES NOT TRANSFER TO SMOTE For undersampling, the calibration damage is essentially a known prior shift, and D… view at source ↗
read the original abstract

Resampling methods such as SMOTE and random under/over-sampling are standard tools for class-imbalanced classification, almost always evaluated by minority-class accuracy or F1. Prior work has established that undersampling degrades probability calibration by distorting the training prior [1]. We extend this lens to synthetic oversampling (SMOTE) and provide a practical, evidence-based guide to when calibration damage matters and how to fix it. Across five public datasets (imbalance ratio 1.9-70) and two ensemble models (random forest, gradient boosting), with ten seeds and paired statistics, we find: (1) SMOTE's calibration cost is real but small (ECE +0.009; Cliff's delta = +0.27, small-to-moderate) across the studied imbalance range (IR 1.9-70) and its discrimination gains typically outweigh the calibration penalty; (2) random undersampling is the genuine danger -- its damage grows sharply with imbalance, inflating ECE from 0.008 to 0.395 on a dataset with ratio 70, largely because the resulting training sets are too small to estimate probabilities reliably; (3) a single post-hoc recalibration step (Platt or isotonic) eliminates the damage, reducing ECE by up to 66% at a negligible ranking-power cost (AUC -0.002, Cliff's delta = -0.07); and (4) the analytic prior-shift correction that repairs undersampling does not transfer to SMOTE, because SMOTE distorts the class-conditional density rather than only the prior -- so data-driven recalibration remains necessary. We recommend that imbalanced-learning studies report calibration alongside discrimination, and that practitioners recalibrate after resampling whenever predicted probabilities drive decisions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript empirically studies the effects of common resampling methods (SMOTE and random undersampling) on probability calibration of tree ensemble models (random forest, gradient boosting) for class-imbalanced classification. Across five public datasets (imbalance ratios 1.9–70), ten seeds, and paired statistics, it reports that SMOTE produces a small ECE increase (+0.009, Cliff’s delta +0.27) whose discrimination gains usually outweigh the penalty; random undersampling produces large, IR-dependent ECE degradation (up to 0.395 at IR=70) mainly from reduced training-set size; post-hoc recalibration (Platt or isotonic) removes most of the damage at negligible AUC cost; and the analytic prior-shift correction that repairs undersampling does not transfer to SMOTE because SMOTE alters class-conditional densities rather than only the prior. The paper recommends that imbalanced-learning studies report calibration alongside discrimination metrics and that practitioners recalibrate after resampling.

Significance. If the reported patterns hold, the work supplies a practical, evidence-based guide for when calibration damage from resampling is negligible versus severe and demonstrates that a cheap post-hoc fix restores calibration. This is valuable for any application in which predicted probabilities are used for decision-making or risk assessment. The multi-dataset, multi-model, multi-seed design with effect-size reporting (Cliff’s delta) and explicit comparison of analytic versus data-driven correction constitutes a clear strength.

major comments (2)
  1. [Abstract, point (4)] Abstract, point (4): the claim that the analytic prior-shift correction fails for SMOTE specifically because SMOTE distorts class-conditional densities (rather than only the prior) is load-bearing for the mechanistic distinction drawn between the two resampling families. This attribution rests on the observed patterns across the five chosen datasets; without controlled synthetic data in which the true class-conditional densities are known, alternative explanations (dataset-specific overlap, feature distributions, or sample-size artifacts) cannot be ruled out.
  2. [Results / Discussion] The central attribution of ECE changes to the resampling mechanisms themselves (prior shift for undersampling versus density distortion for SMOTE) assumes the five public datasets are free of unmeasured confounders that interact with the resampling step. A concrete test would be to repeat the protocol on synthetic data with known generative distributions; the current design leaves open whether the reported IR-dependent patterns are general properties of the methods or artifacts of the particular datasets.
minor comments (2)
  1. [Methods] The abstract states that “paired statistics” and Cliff’s delta values are reported, yet the precise tests (e.g., Wilcoxon signed-rank, exact p-value thresholds) and any multiple-comparison corrections are not described; these details belong in the methods section.
  2. [Experimental setup] The five datasets and their exact imbalance ratios are referenced only by range; listing the dataset names together with their IR values and sample sizes in a table would improve reproducibility and allow readers to assess representativeness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on strengthening the mechanistic claims. We address both major comments below by agreeing to expand the discussion of limitations while maintaining the empirical focus on real-world datasets.

read point-by-point responses

  1. Referee: [Abstract, point (4)] Abstract, point (4): the claim that the analytic prior-shift correction fails for SMOTE specifically because SMOTE distorts class-conditional densities (rather than only the prior) is load-bearing for the mechanistic distinction drawn between the two resampling families. This attribution rests on the observed patterns across the five chosen datasets; without controlled synthetic data in which the true class-conditional densities are known, alternative explanations (dataset-specific overlap, feature distributions, or sample-size artifacts) cannot be ruled out.

    Authors: We agree that the mechanistic attribution would be strengthened by synthetic data experiments with known generative distributions. Our current evidence rests on the consistent empirical pattern that the analytic prior-shift correction (which assumes only a change in class prior) restores calibration for undersampling but provides no benefit for SMOTE, across five datasets with varying characteristics. This differential behavior supports density distortion as the distinguishing factor. We will revise the abstract and discussion to explicitly note this as a limitation and recommend synthetic validation as future work. revision: partial

  2. Referee: [Results / Discussion] The central attribution of ECE changes to the resampling mechanisms themselves (prior shift for undersampling versus density distortion for SMOTE) assumes the five public datasets are free of unmeasured confounders that interact with the resampling step. A concrete test would be to repeat the protocol on synthetic data with known generative distributions; the current design leaves open whether the reported IR-dependent patterns are general properties of the methods or artifacts of the particular datasets.

    Authors: We acknowledge that dataset-specific factors could interact with resampling and that synthetic data would isolate the mechanisms more cleanly. The reported patterns (IR-dependent ECE degradation under undersampling, small and stable cost under SMOTE) are nevertheless observed consistently across five public datasets spanning different domains, feature types, and imbalance ratios (1.9–70), which reduces the likelihood of idiosyncratic artifacts. In the revised manuscript we will add an explicit limitations paragraph in the Discussion section acknowledging the absence of synthetic controls and outlining how such experiments could confirm generality. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical measurements with no derivations or self-referential claims

full rationale

The manuscript is an empirical study reporting measured effects of resampling (SMOTE, undersampling) on ECE, AUC, and related metrics across five public datasets and two tree ensembles. No equations, analytic derivations, fitted parameters renamed as predictions, or uniqueness theorems appear. Prior work on undersampling is cited externally; the central results (ECE deltas, recalibration benefits) are direct observations from the experiments rather than reductions to inputs by construction. This matches the default non-circular case for data-driven reporting.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard empirical ML evaluation practices and the assumption that ECE and AUC are appropriate metrics; no new free parameters, axioms beyond domain standards, or invented entities are introduced.

axioms (1)
  • domain assumption Standard assumptions that i.i.d. test data and ECE/AUC are valid measures of calibration and discrimination.
    Invoked implicitly when interpreting the reported ECE and AUC changes as meaningful.

pith-pipeline@v0.9.1-grok · 5851 in / 1260 out tokens · 36835 ms · 2026-06-30T07:51:59.672857+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Cal- ibrating probability with undersampling for unbalanced classification,

    A. Dal Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi, “Cal- ibrating probability with undersampling for unbalanced classification,” inProc. IEEE Symp. Series on Computational Intelligence (SSCI), 2015, pp. 159–166

    2015

  2. [2]

    SMOTE: Synthetic minority over-sampling technique,

    N. V . Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,”Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002

    2002

  3. [3]

    SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary,

    A. Fern ´andez, S. Garc ´ıa, F. Herrera, and N. V . Chawla, “SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary,”Journal of Artificial Intelligence Research, vol. 61, pp. 863–905, 2018

    2018

  4. [4]

    Random forests,

    L. Breiman, “Random forests,”Machine Learning, vol. 45, no. 1, pp. 5–32, 2001

    2001

  5. [5]

    XGBoost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” inProc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794

    2016

  6. [6]

    Predicting good probabilities with supervised learning,

    A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” inProc. 22nd Int. Conf. Machine Learning (ICML), 2005, pp. 625–632

    2005

  7. [7]

    Beta calibration: A well-founded and easily implemented improvement on logistic calibration for binary classifiers,

    M. Kull, T. Silva Filho, and P. Flach, “Beta calibration: A well-founded and easily implemented improvement on logistic calibration for binary classifiers,” inProc. 20th Int. Conf. Artificial Intelligence and Statistics (AISTATS), ser. PMLR, vol. 54, 2017, pp. 623–631

    2017

  8. [8]

    On Calibration of Modern Neural Networks

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inProc. 34th Int. Conf. Machine Learning (ICML), ser. PMLR, vol. 70, 2017, pp. 1321–1330, arXiv:1706.04599

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  9. [9]

    Verified uncertainty calibration,

    A. Kumar, P. Liang, and T. Ma, “Verified uncertainty calibration,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019, pp. 3791–3802

    2019

  10. [10]

    Measur- ing calibration in deep learning,

    J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measur- ing calibration in deep learning,” inProc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR) Workshops, 2019

    2019

  11. [11]

    An experimental investigation of calibration techniques for imbalanced data,

    Y . Huang, W. Li, F. Macheret, R. A. Gabriel, and L. Ohno-Machado, “An experimental investigation of calibration techniques for imbalanced data,”IEEE Access, vol. 8, pp. 127 343–127 352, 2020

    2020

  12. [12]

    ReMix: Calibrated resam- pling for class imbalance in deep learning,

    C. Bellinger, R. Corizzo, and N. Japkowicz, “ReMix: Calibrated resam- pling for class imbalance in deep learning,” inProc. IEEE Int. Conf. Data Mining (ICDM), 2020

    2020

  13. [13]

    Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure,

    M. Saerens, P. Latinne, and C. Decaestecker, “Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure,”Neural Computation, vol. 14, no. 1, pp. 21–41, 2002

    2002

  14. [14]

    The foundations of cost-sensitive learning,

    C. Elkan, “The foundations of cost-sensitive learning,” inProc. 17th Int. Joint Conf. Artificial Intelligence (IJCAI), 2001, pp. 973–978

    2001