REVIEW 4 major objections 5 minor 1 cited by
Statistical Undersampling with Mutual Information and Support Points
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Information-aware undersampling methods improve balanced accuracy by preserving the structure of the majority class.
desk verdict Support points for undersampling is a principled new application, but the post-hoc split invalidates the reported gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the empirical energy distance $\hat{E}(X,Z)$ between the full majority dataset $X$ and the candidate subset $Z$, defined as $2\cdot\frac{1}{Nm}\sum_{i,j}\|x_i-z_j\| - \frac{1}{N^2}\sum_{i,i'}\|x_i-x_{i'}\| - \frac{1}{m^2}\sum_{j,j'}\|z_j-z_{j'}\|$. Minimizing it with the gradient update pushes the selected points toward the majority distribution while keeping them from collapsing, which is the mechanism the paper claims preserves feature-wise means and standard deviations. The mutual-information branch uses the pairwise matrix $I(X,Y)$ computed by brute force, converted into a dissimilarity matrix and clustered into strata; stratified simple random sampling then draws points from each stratum. The paper validates the support-points branch with feature-wise statistics and Kolmogorov-Smirnov tests compared against the original majority class.
What would settle it
Re-run the breast-cancer experiment with a held-out test set taken from the original 272 samples before any undersampling, then compare logistic regression balanced accuracy under random undersampling versus the mutual-information method; if random undersampling matches or beats the proposed method on that held-out set, the paper's central claim fails.
Extended reading notes
Core claim
On its own terms, the paper establishes that both proposed undersampling methods improve or match the balanced accuracy of random undersampling because they keep more of the majority class's distributional structure. The mutual-information branch computes pairwise mutual information among majority points, clusters the resulting dissimilarity matrix into four strata, and applies stratified simple random sampling with Neyman or optimal allocation. The support-points branch clusters the majority class into 50 groups, extracts 5,000 representative points, then refines them by gradient descent on the empirical energy distance, reaching a final energy distance near 0.0102 and mapping the optimized points back to nearest neighbors in the original data. The reported balanced accuracies on the breast-cancer dataset are 0.8485 versus 0.5455 for logistic regression, 0.6970 versus 0.4848 for random forest, and 0.7273 versus 0.6061 for SVC, all against random undersampling. On the credit-card fraud dataset, support points give balanced accuracy at or slightly above random undersampling for logistic regression, random forest, XGBoost, and SVC.
Load-bearing premise
The load-bearing assumption is that an 80-20 train-test split performed after undersampling measures how well the classifier generalizes; because the test set is also artificially balanced, the reported balanced accuracy describes performance on a balanced subset rather than on the original imbalanced population.
Editorial extensions
If this is right
- If the breast-cancer results replicate, imbalanced medical datasets with only a few hundred samples can be undersampled in under a minute while gaining more than 30 percentage points of balanced accuracy for simple linear models.
- If the support-points results hold at scale, a majority class of hundreds of thousands of points can be summarized by a few thousand representative points with minimal loss of distributional fidelity.
- The MI branch's brute-force pairwise computation is $O(n^2)$, so its practical use is limited to small datasets unless a cheaper mutual-information estimator is substituted.
- The support-points branch's $O(T(N'md + m^2d))$ cost makes clustering-based pre-subsampling necessary for very large datasets, as the paper itself does for the 284,807-sample fraud dataset.
Reading between the lines
- A natural next test, not reported in the paper, is to hold out a test set from the original imbalanced distribution before any undersampling; the current 80-20 split after undersampling measures accuracy on a balanced subset and likely overstates real-world performance.
- The MI stratification idea could be paired with subquadratic or upper-bound MI estimators to escape the $O(n^2)$ bottleneck, potentially making the method applicable to mid-sized datasets.
- The support-points branch's final energy distance and KS-test results suggest the optimized points retain global distributional shape; a testable extension would be to compare support points against other stratified sampling methods on the same energy distance, not only against random undersampling.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two undersampling strategies for imbalanced classification: a mutual-information-based stratified simple random sampling method and a support-points optimization method. The MI method computes a purported pairwise 'element-wise' mutual information matrix among majority-class samples, converts it into a dissimilarity matrix, clusters via K-means, and then applies stratified sampling. The support-points method minimizes the empirical energy distance between a reduced majority subset and the original majority distribution, mapping optimized points to nearest neighbors. Experiments compare balanced accuracy against random undersampling on a 272-sample breast cancer dataset (MI) and a 284,807-sample credit card fraud dataset (support points). The paper reports large MI gains for logistic regression, random forest, and SVC, and small gains or ties for support points across four classifiers.
Significance. If the empirical claims were valid, the support-points application would be a modest but reasonable extension of Mak and Joseph (2018) to class-imbalance problems, and the paper's quantitative representativeness checks (feature-wise statistics and KS tests) are appropriate. However, the central evaluation protocol is flawed: the train/test split occurs after undersampling, so the test sets are method-dependent and artificially balanced. The MI estimator is also not mathematically well-defined, as mutual information is a distributional quantity rather than a per-point score. The reported gains, especially the >30% logistic-regression improvement in Table 1, are therefore not interpretable as generalization improvements. The current manuscript does not establish its headline claim that the proposed methods outperform traditional undersampling.
major comments (4)
- [§5.1, Step 3; §6.1.2, Table 1; §6.2.3, Table 5] The evaluation protocol is invalid for the central claim. The 80-20 train/test split is applied after the majority class has been undersampled, so the test set contains the same method-selected majority points used for training, is balanced, and differs across methods. Consequently, the reported balanced accuracies do not measure performance on the original imbalanced distribution; a sampling method that preferentially selects majority points that are easy to separate from the minority class can inflate test accuracy without improving genuine generalization. For example, the logistic-regression improvement from 0.5455 to 0.8485 in Table 1 is an artifact of this protocol. A correct protocol would split the original data into train and test first, then undersample only the training fold, and evaluate on the untouched test set.
- [§4.1.1 and §4.1.2] The proposed 'element-wise' mutual information computation is not well defined. Mutual information I(X;Y) is a functional of the joint and marginal distributions of two random variables, not a per-data-point quantity. The text says the authors calculate MI between each x and y pair and store the values in a symmetric matrix, but for fixed observations x_i and y_j there is no joint distribution PXY(x_i, y_j) to estimate in the manner described. The algorithm therefore does not implement the formula in §4.1.1, and the stratification based on this heuristic is not justified by the stated information-theoretic rationale.
- [§6.1.2, Table 1] The MI experiments use a single split with no repeated seeds, no cross-validation, and no uncertainty quantification. The balanced test set for the breast cancer dataset is only about 32 points after the 80-20 split, so the reported 'over 30%' improvement for logistic regression and 'over 20%' for random forest could be within split-to-split variability. The paper needs repeated trials or confidence intervals before such claims can be taken as evidence.
- [§6.2.3, Table 5; §6.2.2] The support-points classification results show at most a 1.4-percentage-point improvement over random undersampling and one exact tie (XGBoost). Given that the same post-undersampling split protocol is used and no error bars are provided, the abstract's claim that support points 'outperform' traditional undersampling is not supported. The representativeness analysis in §6.2.2 is informative but reports only selected features and does not by itself establish classifier-level superiority.
minor comments (5)
- [Abstract and §1] There are numerous grammatical errors that impede readability, e.g., 'classification tasks machine learning' in the abstract and 'it impacted machine learning models' in §3; these should be corrected.
- [§6.1.1] The text says the resampled data will be 'spitted' into an 80-20 split; this should be 'split'.
- [§2.2] The citation to Bhagat and Patil (2015) is incomplete and does not appear in the reference list; the authors should add the full reference.
- [§4.2.1, Eq. (1)] The energy distance formula is written without explicit expectation notation and the norm is undefined; specifying the norm and the underlying probability spaces would improve clarity.
- [§6.2.1] The claim that direct pairwise computations require over 600GB of memory is not explained; the complexity analysis in §4.2.4 also omits the cost of the clustering-based subsampling stage.
Circularity Check
No circular dependency: the reported accuracies are empirical outcomes, not encoded in the sampling objectives; the train/test-split concern is a validity issue, not circularity.
full rationale
I walked the paper's derivation chain and found no step where a claimed prediction or first-principles result reduces by construction to its inputs. The mutual-information method computes pairwise MI values, clusters them, and performs stratified SRS; this is a heuristic sampling procedure with no fitted parameter that forces the reported balanced accuracies. The support-points method minimizes the empirical energy distance (Eq. 1) against the majority-class distribution; again, the classification accuracies in Tables 1 and 5 are produced after training four classifiers on the undersampled data and evaluating on the held-out 20% split. No accuracy value is used as an input to the sampling objective, and no classifier output is claimed to be predicted from the method's defining equations. The representativeness checks in Section 6.2.2 (feature-wise means, standard deviations, KS tests) are consequences of the energy-distance objective, but the paper does not present them as an independent prediction; they are sanity checks on the optimization. The only external method invoked is support points from Mak & Joseph (2018), which is a published, externally checkable technique, and it is not used to justify the empirical superiority claim. The evaluative concern that undersampling before the 80-20 split makes test sets method-dependent and artificially balanced is a legitimate correctness/validity critique, but it is not circularity under the definitions used here. Therefore, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (6)
- Number of MI strata =
4
- Number of MiniBatchKMeans clusters =
50
- Subsample size from clusters =
5000
- Support points learning rate
- Support points iterations =
2000
- Optimal allocation cost function =
Not specified
assumptions (4)
- ad hoc to paper Pairwise mutual information between individual data points is a valid similarity measure for stratification
- domain assumption The elbow method reliably determines the true number of strata/clusters
- domain assumption Minimizing energy distance between the reduced set and the original data preserves enough information for classification
- domain assumption The clustering-based subsampling retains the heterogeneity of the original majority class
Cite this review
Pith. "Pith review of Statistical Undersampling with Mutual Information and Support Points." pith.science (2026). https://pith.science/paper/J3TSDHL2
@misc{pith2026241214527,
author = {Pith},
title = {Pith review of: Statistical Undersampling with Mutual Information and Support Points},
year = {2026},
howpublished = {\url{https://pith.science/paper/J3TSDHL2}},
note = {Machine review of arXiv:2412.14527}
}
read the original abstract
Class imbalance and distributional differences in large datasets present significant challenges for classification tasks machine learning, often leading to biased models and poor predictive performance for minority classes. This work introduces two novel undersampling approaches: mutual information-based stratified simple random sampling and support points optimization. These methods prioritize representative data selection, effectively minimizing information loss. Empirical results across multiple classification tasks demonstrate that our methods outperform traditional undersampling techniques, achieving higher balanced classification accuracy. These findings highlight the potential of combining statistical concepts with machine learning to address class imbalance in practical applications.
Figures
Forward citations
Cited by 1 Pith paper
-
Detecting Mislabeled and Corrupted Data via Pointwise Mutual Information
Samples with low pointwise mutual information between image and label are mostly mislabeled or corrupted, and dropping them before training improves MNIST accuracy by up to 15%.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Devi, D., Biswas, S. K., and Purkayastha, B. A review on solution to class imbalance problem: Undersampling approaches. In 2020 International Conference on Computational Performance Evaluation (ComPE), pp.\ 626--631, 2020. doi:10.1109/ComPE49325.2020.9200087
-
[3]
Measuring mutual information between all pairs of variables in subquadratic complexity
Ferdosi, M., Gholamidavoodi, A., and Mohimani, H. Measuring mutual information between all pairs of variables in subquadratic complexity. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp.\ 4399--4409. PMLR, ...
work page 2020
-
[4]
Fernandes, A. D. and Gloor, G. B. Mutual information is critically dependent on prior assumptions: would the correct estimate of mutual information please identify itself? Bioinformatics, 26 0 (9): 0 1135--1139, 2010. doi:10.1093/bioinformatics/btq111. URL https://doi.org/10.1093/bioinformatics/btq111
-
[5]
An overview of classification algorithms for imbalanced datasets
Ganganwar, V. An overview of classification algorithms for imbalanced datasets. International Journal of Emerging Technology and Advanced Engineering, 2: 0 42--47, 01 2012
work page 2012
-
[6]
Mak, S. and Joseph, V. R. Support points. The Annals of Statistics, 46 0 (6A): 0 2562--2592, 2018. ISSN 00905364, 21688966. URL https://www.jstor.org/stable/26542875
-
[7]
Mim4dd: mutual information maximization for dataset distillation
Shang, Y., Yuan, Z., and Yan, Y. Mim4dd: mutual information maximization for dataset distillation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc
work page 2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.