Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Neural Network Based Undersampling Techniques

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training a neural network on the minority class and keeping the farthest majority points improves imbalanced classification.

desk verdict A simple, clearly described undersampling idea whose central experimental claim is invalidated by resampling the whole dataset before classification. read the letter →

arxiv 1908.06487 v1 pith:3OPSZSYJ submitted 2019-08-18 cs.LG stat.ML

Md. Adnan Arefeen , Sumaiya Tabassum Nimi , M Sohel Rahman This is my paper · ORCID
classification cs.LGstat.ML
keywords undersamplingclassimbalanceautoencoderneuralnetworkresamplingimbalancedclassificationAUCG-mean
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes two undersampling algorithms, NUS1 ('hard') and NUS2 ('soft'), that use a neural network trained only on the minority class to score every majority-class example. The score is the Euclidean distance between the example's actual features and the features the minority-trained model predicts for it; the algorithms then keep majority examples with large reconstruction distance. The authors claim that, across four public datasets and five classifiers, these selections yield higher AUC, F1, and G-mean than most established undersampling techniques. If the claim holds, practitioners gain a resampling method that turns imbalanced data into a more separable balanced set without discarding all informative majority structure.

What carries the argument

The load-bearing object is the minority-trained reconstruction model: an autoencoder when the input has more than 30 attributes, and a simple feedforward network otherwise. The model is not used to classify; it is used to generate, for every majority example $x$, a predicted feature vector $x'$, and the selection score is the squared Euclidean distance $\|x - x'\|_2^2$. Sorting majority samples by this score and keeping the largest distances is what creates the balanced, more separable dataset. The threshold 30 is the switch that picks the architecture, and NUS2's extra threshold—the maximum or half-average minority distance—is what softens selection for overlapping data.

What would settle it

Run the same four-dataset comparison with a strict protocol: split each dataset into training and test sets first, apply each undersampler only to the training portion, train the five classifiers on the resampled training data, and evaluate on the untouched test set. If NUS1 and NUS2 no longer produce the reported AUC, F1, and G-mean advantages, the central claim is an artifact of the undersampler selecting the test examples.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a minority-class reconstruction model can act as an effective undersampling oracle. NUS1 trains an autoencoder (for datasets with more than 30 attributes) or a simple feedforward network on the minority samples, predicts each majority sample's features with that model, computes the squared Euclidean distance $\|x - x'\|_2^2$ between prediction and actual values, and retains the $n_1$ majority samples with the largest distances, where $n_1$ is the minority count. NUS2 uses the same reconstruction distances but selects majority samples whose distance exceeds the maximum minority reconstruction distance or the average distance of the farthest half of minority samples, which the paper says handles overlapping classes. The paper reports that these methods outperform most other undersamplers in almost all classifier-dataset combinations and that NUS1 excels on non-overlapping data while NUS2 performs better on overlapping data.

Load-bearing premise

The load-bearing premise is that classifying a dataset that was resampled before any train/test split tells you how the undersampler will perform on the original imbalanced distribution; the paper never states that resampling is restricted to training folds.

Editorial extensions

If this is right

  • If NUS1 and NUS2 work as reported, a practitioner can replace distance-to-minority heuristics like NearMiss with a learned reconstruction distance, which adapts to feature correlations.
  • The paper's own experiments imply NUS1 should be preferred when classes are well separated, and NUS2 when classes overlap; the choice can be guided by visualizing resampled data or by validating on a holdout.
  • Because the method only needs the minority class to train the scoring model, it can be applied before any classifier, matching the paper's claim that gains are not classifier-specific.
  • The paper states that for very high imbalance or majority outliers, the greedy selection of farthest samples may discard useful majority information; the proposed future direction is to use prediction accuracy on majority samples to choose batches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the reconstruction-error score is essentially a one-class novelty measure, so comparing NUS1 and NUS2 against one-class classifiers or isolation-forest-based undersampling would test whether the specific neural-network training step is what drives the gain.
  • My inference: the hard/soft distinction suggests a continuum; a tunable quantile of the reconstruction-error distribution could interpolate between NUS1 and NUS2 and might remove the need to choose one algorithm up front.
  • My inference: if resampling is applied before the train/test split, as the paper's experimental description suggests, test examples are selected by the same scores that define the method; I would expect the reported margins to shrink under a protocol that resamples only training folds. This is my reading of the pipeline, not a claim the paper makes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two neural-network-based undersampling algorithms, NUS-1 (hard) and NUS-2 (soft), for imbalanced classification. Both train an autoencoder or simple feedforward network on the minority class, compute Euclidean distances between actual and predicted feature values for majority samples, and retain those majority samples that are most distant from the minority class (or outside an estimated minority perimeter). The authors evaluate these and several standard undersamplers on four UCI datasets with five classifiers, reporting AUC, G-mean, and F1 scores, and claim that NUS-1 and NUS-2 outperform other undersamplers in most settings. They also include visualizations on two artificial datasets.

Significance. If the reported evaluation were valid, the idea of using reconstruction error from a minority-class model to select informative majority samples would be a plausible and interesting contribution to imbalanced learning. The algorithms are simple to describe and compare across multiple classifiers. However, the central empirical claim is not supported by the experiments as described: because the entire dataset is resampled before classification and no train/test separation is specified, the reported scores appear to be computed on the very samples each undersampler selected, which is a circular evaluation. The paper also lacks machine-checked proofs, reproducible code, or a precise experimental protocol, and the significance therefore rests entirely on an unverified and likely invalid comparison.

major comments (4)
  1. [Section IV.A] The experimental protocol never describes a train/test split or cross-validation. The text states only that the dataset is undersampled and then 'fed to a number of classifiers'. If, as written, the whole dataset is resampled before classification, then the test examples are the very majority samples selected by the undersampler being scored. For NUS-1 those are the majority points farthest from the minority model, so the classifier is evaluated on an artificially separable subset; this circularity inflates the metrics and invalidates the comparison with other undersamplers. This is the load-bearing flaw of the paper.
  2. [Section IV.C] The paper explicitly ties the reported higher AUC to the separability of the resampled data: 'NUS1 and NUS2 resample the data in such a way that they become more separable... This leads to higher AUC'. This confirms that the evaluation measures separability of the selected samples, not generalization to the original imbalanced distribution. A valid evaluation must restrict undersampling to the training folds and evaluate on an untouched test set; the current design cannot support the claim of outperformance.
  3. [Tables V-XVI] The reported means plus/minus standard deviations are unexplained. No repetition count, fold structure, random seed, or resampling protocol is given, so the variance estimates cannot be interpreted. Additionally, Table IV shows that several baselines select almost the entire majority class on some datasets (e.g., TLL selects 571 of 576 majority samples on Balance and 225 of 225 on Ionosphere), meaning those comparisons are not even against balanced data, further undermining the comparative claim.
  4. [Section IV.B, Eq. (1)] The G-mean definition is inconsistent: the equation correctly gives sqrt(TP*TN), but the text says 'square root of the product of true positives (TP) and false positives (FP)'. If the textual description were used, the metric would be incorrect. The definitions and the equation should be aligned.
minor comments (5)
  1. [Section III] The threshold value of 30 for choosing between autoencoder and simple neural network is mentioned, but no justification or sensitivity analysis is provided, and the neural network architectures, training epochs, learning rates, and other hyperparameters are unspecified, making the experiments hard to reproduce.
  2. [Section V] There is a typo in the first sentence: 'Th datasets' should be 'The datasets'.
  3. [Section VII] 'loosing information' should be 'losing information'.
  4. [Section VI] The claim that the choice between maximum and average distance is a tunable parameter is not accompanied by any experiments on that parameter, so it is unclear how it affects performance.
  5. [Section IV.C] The statement that NUS1 and NUS2 'outperform all other undersamplers in case of almost all training algorithms' is stronger than the data show: for example, on the Ionosphere dataset, NCR has comparable or better AUC and G-mean for several classifiers (Tables XIV and XV).

Circularity Check

2 steps flagged · score 7.0 of 10

Reported NUS1/NUS2 superiority is computed on the very majority points the undersamplers select to be farthest from the minority class, so the headline result reduces by construction to the selection criterion.

  1. fitted input called prediction [Section IV.A (Overview of the experiments), Section IV.C, with Algorithm 1 (lines 18-19)]
    "We undersample the dataset under consideration using different undersampling algorithms. Subsequently, the under-sampled dataset is fed to a number of classifiers and we evaluate the classification results thereof. ... In almost all cases, we found that, our proposed undersamplers, NUS1 and NUS2 outperform all other undersamplers in case of almost all training algorithms. NUS1 and NUS2 resample the data in such a way that they become more separable ... This leads to higher AUC, G-mean and F1 values and hence better performance."

    No train/test split, cross-validation, or restriction of resampling to training folds is ever described; the 'under-sampled dataset' is the only data classified and scored. For NUS1, Algorithm 1 sorts all majority samples by descending Euclidean distance between each sample and the value predicted by a model trained on minority samples and 'choose[s] first n1' (lines 18-19), i.e., the majority points farthest from the minority class. AUC, F1, and G-mean are then measured on this exact subset.

  2. self definitional [Section III.A.1, Hard Neural Network Based Undersampling]
    "In other words, our under-sampling approach actually removes the majority class samples which are present in the vicinity of the minority class samples and retains the majority class samples which are located further from the minority class samples. Hence the decision boundary becomes more defined and the resulting balanced dataset becomes more separable. As a consequence, this algorithm outperforms most other undersampling algorithms for most datasets."

    The paper derives the conclusion 'this algorithm outperforms' directly from the premise that the algorithm makes the balanced dataset 'more separable' ('As a consequence'). But making the dataset more separable is precisely the algorithm's selection objective, since it discards near-boundary majority samples and keeps far ones. When performance is scored on that same self-selected dataset (Section IV.A), 'outperforms' is the selection rule restated as an empirical result rather than an independent finding.

full rationale

The paper's central claim (NUS1 and NUS2 outperform other undersamplers in AUC, F1, and G-mean) rests entirely on the Section IV evaluation, which as written feeds the undersampled dataset directly to classifiers with no train/test split, cross-validation, or resampling-restricted-to-train protocol. NUS1 by construction retains only the majority points farthest from a minority-fitted model; NUS2 retains only points outside the minority 'perimeter.' Scoring classifiers on these same points measures separability of the chosen subset, not performance on the original imbalanced distribution, so the reported superiority reduces by construction to the selection criterion. The paper itself states this reduction: the algorithm makes the data 'more separable' and 'As a consequence, this algorithm outperforms' (Section III.A.1), and Section IV.C attributes higher AUC to the resampling making data 'more separable.' The missing test protocol is an omitted support that must be flagged per the review rule; the limitation passage in Section VII concerns only information loss at high imbalance ratios and does not address this evaluation leakage. There are no load-bearing self-citations, uniqueness theorems, or renamed known results in the paper, so the circularity is confined to the empirical evaluation; nevertheless, because that evaluation is the entire basis of the headline claim, the score is 7 rather than a minor citation-level score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. It relies on two domain assumptions about autoencoder reconstruction error and about the validity of evaluating on resampled data, plus arbitrary hyperparameter choices.

free parameters (2)
  • threshold value for model selection = 30
    Section III: if number of input attributes > 30 use autoencoder, else simple NN. This value is arbitrary and not tuned or justified.
  • neural network architecture and hyperparameters
    Section III and Figures 1-2 specify only '2/3 hidden layers' with no units, activation, epochs, optimizer, learning rate, or batch size; these choices affect reconstruction error and thus sample selection.
assumptions (3)
  • domain assumption Reconstruction error of a minority-trained autoencoder/network on majority samples is a valid measure of distance to the minority class distribution.
    The entire selection mechanism in Algorithms 1 and 2 relies on this; no theoretical or empirical validation is provided.
  • domain assumption Evaluating classifiers on data that has been undersampled before splitting into train/test is a valid measure of real-world performance.
    Section IV.A describes resampling the dataset and then classifying it; this assumes the resampled distribution represents the deployment distribution, which is generally false and biases results.
  • domain assumption Linear separability of the resampled training data transfers to better performance on the original imbalanced distribution.
    Section V and VI claim that making the data separable leads to higher AUC; the link between separability of resampled data and generalization to original data is assumed, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Network Based Undersampling Techniques." pith.science (2026). https://pith.science/paper/3OPSZSYJ

@misc{pith2026190806487,
  author       = {Pith},
  title        = {Pith review of: Neural Network Based Undersampling Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OPSZSYJ}},
  note         = {Machine review of arXiv:1908.06487}
}
read the original abstract

Class imbalance problem is commonly faced while developing machine learning models for real-life issues. Due to this problem, the fitted model tends to be biased towards the majority class data, which leads to lower precision, recall, AUC, F1, G-mean score. Several researches have been done to tackle this problem, most of which employed resampling, i.e. oversampling and undersampling techniques to bring the required balance in the data. In this paper, we propose neural network based algorithms for undersampling. Then we resampled several class imbalanced data using our algorithms and also some other popular resampling techniques. Afterwards we classified these undersampled data using some common classifier. We found out that our resampling approaches outperform most other resampling techniques in terms of both AUC, F1 and G-mean score.

Figures

Figures reproduced from arXiv: 1908.06487 by the authors.

Figure 1
Figure 1. Simple Neural Network to generate input. The nodes shown in green color are inputs. Two hidden layers are shown [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An autoencoder neural network model. The inputs are regenerated by first encoding and then by decoding the encoded [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Artificial dataset resampling with various undersamplers [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Artificial overlapped dataset resampled by various undersamplers [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: NUS-1 and NUS-2 both are trying to choose far [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages

  1. [1]

    Evolutionary undersampling boosting for imbalanced classification of breast cancer malignancy

    Bartosz Krawczyk, Mikel Galar, Łukasz Jele ´n, and Francisco Herrera. Evolutionary undersampling boosting for imbalanced classification of breast cancer malignancy. Applied Soft Computing , 38:714–726, 2016

    work page 2016

  2. [2]

    Prediction of weather-induced airline delays based on machine learning algorithms

    Sun Choi, Young Jin Kim, Simon Briceno, and Dimitri Mavris. Prediction of weather-induced airline delays based on machine learning algorithms. In 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) , pages 1–6. IEEE, 2016

    work page 2016

  3. [3]

    Effective detection of sophisticated online banking fraud on extremely imbalanced data

    Wei Wei, Jinjiu Li, Longbing Cao, Yuming Ou, and Jiahang Chen. Effective detection of sophisticated online banking fraud on extremely imbalanced data. World Wide Web, 16(4):449–475, 2013

    work page 2013

  4. [4]

    Information retrieval 2nd edition butterworths.London available on internet , 1979

    CJ Van Rijsbergen. Information retrieval 2nd edition butterworths.London available on internet , 1979

    work page 1979

  5. [5]

    Using auc and accuracy in evaluating learning algorithms

    Jin Huang and Charles X Ling. Using auc and accuracy in evaluating learning algorithms. IEEE Transactions on knowledge and Data Engineering, 17(3):299–310, 2005

    2005

  6. [6]

    Addressing the curse of imbalanced training sets: one-sided selection

    Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Icml, volume 97, pages 179–186. Nashville, USA, 1997

    1997

  7. [7]

    Classification of imbalanced data: A review

    Yanmin Sun, Andrew KC Wong, and Mohamed S Kamel. Classification of imbalanced data: A review. International Journal of Pattern Recognition and Artificial Intelligence , 23(04):687–719, 2009

    work page 2009

  8. [8]

    Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research , 18(17):1–5, 2017

    work page 2017

Show all 28 references
  1. [9]

    A generalization of the k-nn rule

    Ivan Tomek. A generalization of the k-nn rule. IEEE Transactions on Systems, Man, and Cybernetics , (2):121–126, 1976

    1976

  2. [10]

    Improving identification of difficult small classes by balancing class distribution

    Jorma Laurikkala. Improving identification of difficult small classes by balancing class distribution. In Conference on Artificial Intelligence in Medicine in Europe , pages 63–66. Springer, 2001

    2001

  3. [11]

    Smote: synthetic minority over-sampling technique

    Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321–357, 2002

    2002

  4. [12]

    Borderline-smote: a new over-sampling method in imbalanced data sets learning

    Hui Han, Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing , pages 878–887. Springer, 2005

    2005

  5. [13]

    A study of the behavior of several methods for balancing machine learning training data

    Gustavo EAPA Batista, Ronaldo C Prati, and Maria Carolina Monard. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD explorations newsletter , 6(1):20–29, 2004

    2004

  6. [14]

    Adasyn: Adaptive synthetic sampling approach for imbalanced learning

    Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) , pages 1322–1328. IEEE, 2008

    2008

  7. [15]

    Asymptotic properties of nearest neighbor rules using edited data

    Dennis L Wilson. Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics , (3):408–421, 1972

    1972

  8. [16]

    knn approach to unbalanced data distribu- tions: a case study involving information extraction

    Inderjeet Mani and I Zhang. knn approach to unbalanced data distribu- tions: a case study involving information extraction. In Proceedings of workshop on learning from imbalanced datasets , volume 126, 2003

    2003

  9. [17]

    Random forests

    Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001

    2001

  10. [18]

    Greedy function approximation: a gradient boosting machine

    Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189–1232, 2001

    2001

  11. [19]

    Pattern classification and scene analysis–john wiley & sons

    RO Duda and PE Hart. Pattern classification and scene analysis–john wiley & sons. New York, NY, 1973

    1973

  12. [20]

    Solving large scale linear prediction problems using stochastic gradient descent algorithms

    Tong Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning , page 116. ACM, 2004

    2004

  13. [21]

    Logistic regression

    Raymond E Wright. Logistic regression. 1995

    1995

  14. [22]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...

    2011

  15. [23]

    The numpy array: a structure for efficient numerical computation

    Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in Science & Engineering , 13(2):22, 2011

    2011

  16. [24]

    J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering , 9(3):90–95, 2007

    2007

  17. [25]

    François Chollet et al. Keras. https://keras.io, 2015

    2015

  18. [26]

    An introduction to roc analysis

    Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874, 2006

    2006

  19. [27]

    Diversified ensemble classifiers for highly imbalanced data learning and their application in bioinformatics

    Zejin Ding. Diversified ensemble classifiers for highly imbalanced data learning and their application in bioinformatics. 2011

    2011

  20. [28]

    UCI machine learning repository, 2017

    Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. Md. Adnan Arefeen Md. Adnan Arefeen is currently working as a lecturer at United Intenational Univer- sity, Bangladesh. He completed his graduation from Bangldesh University of Engneering and Technology. He is ...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.