{"id":"c5b644a2-1394-42d3-b3de-7d92db48103d","arxiv_id":"1908.06487","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A neural-network-based undersampling method selects majority samples with the largest reconstruction error from a minority-trained model and reports higher AUC, F1, and G-mean, but the evaluation protocol likely biases the results.","lead":"This paper proposes two undersampling algorithms that use a neural network trained on the minority class to pick which majority-class samples to keep, then compares them to older resampling methods on four datasets. The reported improvements are undermined by a likely flaw in how the tests were set up: the data appears to be resampled before splitting into training and test sets, which can inflate the scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split or cross-validation is described; reported scores may be computed on the very majority samples each undersampler selected, so the claimed superiority is unverified.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation protocol is not described and appears to resample the entire dataset before classification. My reading of Section IV.A confirms that no split or cross-validation is mentioned, and the language 'we undersample the dataset under consideration' suggests whole-dataset resampling. The reported mean and standard deviation values make the absence of a protocol description even more problematic, since the source of the variance is never stated. This is not merely a reproducibility detail; it determines whether the central claim is supported. If the test set is constructed by the undersampler, high AUC/F1/G-mean values are expected and uninformative. I also note that the paper's own tables contradict the Section IV.C claim in some cases (e.g., Pima AUC: AKNN 0.880 vs NUS1 0.836), but the protocol issue is the primary reason the central claim is unsupported. The artificial-data figures are illustrative but do not provide quantitative evidence. Since the same concern was raised by the reader and I do not find an independent counterargument, the verdict remains REJECT; no change to the reader's decision is warranted.","tokens_in":14197,"tokens_out":4762,"duration_ms":55519,"concrete_test":"Re-run the Table V experiment on the Balance dataset (or obtain the authors' code) with a described 5-fold stratified cross-validation: for each fold, fit each undersampler using only the training fold, resample only that training fold, train the classifier, and evaluate on the untouched original test fold. If NUS1's AUC no longer exceeds the baselines (e.g., CC's reported 0.964) by a meaningful margin, or if the ranking changes materially, the claimed superiority in the paper is an artifact of evaluating on the resampled data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that NUS1 and NUS2 outperform other undersamplers, rests entirely on the experimental evaluation in Section IV. However, Section IV.A never describes a train/test split, a cross-validation procedure, or any separation between the data used to fit the undersampler and the data used to evaluate the classifier. It says only: 'We undersample the dataset under consideration using different undersampling algorithms. Subsequently, the under-sampled dataset is fed to a number of classifiers and we evaluate the classification results thereof.' The tables report mean plus/minus standard deviation, but the source of that variation is unexplained. If the entire dataset is resampled first and then classified, as the text suggests, then the test examples are exactly the majority samples that each undersampler was designed to select. For NUS1, those are the majority points with the largest reconstruction error from a model trained on all minority points, so they are by construction far from the minority class; for NUS2, they are outside the minority 'perimeter.' Evaluating a classifier on such a selected subset measures class separability of the chosen samples, not generalization to the original imbalanced distribution. This leakage would systematically inflate the metrics for every undersampler, but especially for NUS1/NUS2, and would invalidate the comparative claim in Section IV.C. Even if the authors intended to resample within training folds, that intent is absent from the manuscript, so the reported results cannot be interpreted as evidence for the abstract's assertion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two neural-network-based undersampling algorithms, NUS-1 (hard) and NUS-2 (soft), for imbalanced classification. Both train an autoencoder or simple feedforward network on the minority class, compute Euclidean distances between actual and predicted feature values for majority samples, and retain those majority samples that are most distant from the minority class (or outside an estimated minority perimeter). The authors evaluate these and several standard undersamplers on four UCI datasets with five classifiers, reporting AUC, G-mean, and F1 scores, and claim that NUS-1 and NUS-2 outperform other undersamplers in most settings. They also include visualizations on two artificial datasets.","tokens_in":14469,"tokens_out":2102,"duration_ms":23167,"significance":"If the reported evaluation were valid, the idea of using reconstruction error from a minority-class model to select informative majority samples would be a plausible and interesting contribution to imbalanced learning. The algorithms are simple to describe and compare across multiple classifiers. However, the central empirical claim is not supported by the experiments as described: because the entire dataset is resampled before classification and no train/test separation is specified, the reported scores appear to be computed on the very samples each undersampler selected, which is a circular evaluation. The paper also lacks machine-checked proofs, reproducible code, or a precise experimental protocol, and the significance therefore rests entirely on an unverified and likely invalid comparison.","major_comments":[{"comment":"The experimental protocol never describes a train/test split or cross-validation. The text states only that the dataset is undersampled and then 'fed to a number of classifiers'. If, as written, the whole dataset is resampled before classification, then the test examples are the very majority samples selected by the undersampler being scored. For NUS-1 those are the majority points farthest from the minority model, so the classifier is evaluated on an artificially separable subset; this circularity inflates the metrics and invalidates the comparison with other undersamplers. This is the load-bearing flaw of the paper.","section":"Section IV.A"},{"comment":"The paper explicitly ties the reported higher AUC to the separability of the resampled data: 'NUS1 and NUS2 resample the data in such a way that they become more separable... This leads to higher AUC'. This confirms that the evaluation measures separability of the selected samples, not generalization to the original imbalanced distribution. A valid evaluation must restrict undersampling to the training folds and evaluate on an untouched test set; the current design cannot support the claim of outperformance.","section":"Section IV.C"},{"comment":"The reported means plus/minus standard deviations are unexplained. No repetition count, fold structure, random seed, or resampling protocol is given, so the variance estimates cannot be interpreted. Additionally, Table IV shows that several baselines select almost the entire majority class on some datasets (e.g., TLL selects 571 of 576 majority samples on Balance and 225 of 225 on Ionosphere), meaning those comparisons are not even against balanced data, further undermining the comparative claim.","section":"Tables V-XVI"},{"comment":"The G-mean definition is inconsistent: the equation correctly gives sqrt(TP*TN), but the text says 'square root of the product of true positives (TP) and false positives (FP)'. If the textual description were used, the metric would be incorrect. The definitions and the equation should be aligned.","section":"Section IV.B, Eq. (1)"}],"minor_comments":[{"comment":"The threshold value of 30 for choosing between autoencoder and simple neural network is mentioned, but no justification or sensitivity analysis is provided, and the neural network architectures, training epochs, learning rates, and other hyperparameters are unspecified, making the experiments hard to reproduce.","section":"Section III"},{"comment":"There is a typo in the first sentence: 'Th datasets' should be 'The datasets'.","section":"Section V"},{"comment":"'loosing information' should be 'losing information'.","section":"Section VII"},{"comment":"The claim that the choice between maximum and average distance is a tunable parameter is not accompanied by any experiments on that parameter, so it is unclear how it affects performance.","section":"Section VI"},{"comment":"The statement that NUS1 and NUS2 'outperform all other undersamplers in case of almost all training algorithms' is stronger than the data show: for example, on the Ionosphere dataset, NCR has comparable or better AUC and G-mean for several classifiers (Tables XIV and XV).","section":"Section IV.C"}],"recommendation":"reject","confidential_remarks":"The paper's central evaluation is circular as written, and the authors do not describe a train/test split or cross-validation. Even a major revision would require re-running all experiments with proper nested resampling, which is a substantial undertaking. The manuscript also lacks reproducibility details (hyperparameters, seeds, code). I do not see a path to acceptance within the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The algorithms are simple and clearly described: train a neural net on the minority class, reconstruct or predict the majority features, and keep the majority points farthest from the model's reconstruction. That specific recipe is not in the cited prior work, and the pseudocode plus the artificial-data plots make the mechanics easy to follow. The problem is that the evaluation cannot support the headline claim. The paper never describes a train/test split or cross-validation. Section IV.A says only that the dataset is undersampled and then fed to classifiers; the tables report mean plus/minus standard deviation without saying where that variation comes from. If the whole dataset is resampled before classification, as written, then the test set consists exactly of the majority points each undersampler was designed to select. For NUS1 those are points with the largest reconstruction error from a model trained on all minority points, so they are by construction far from the minority class. Scoring a classifier on such a set measures separability of chosen samples, not generalization to the original distribution. Section IV.C essentially admits this: 'NUS1 and NUS2 resample the data in such a way that they become more separable... This leads to higher AUC.' That is circular. The stress-test concern holds up.\n\nWhat the paper does well: the idea is easy to state, the two variants have a sensible motivation (hard selection versus a soft threshold for overlap), and the authors do flag an outlier caveat and a limitation about information loss at high imbalance ratios. Those are honest touches. The related-work citations are standard and relevant.\n\nThe soft spots beyond the leakage: the G-mean definition is inconsistent (the prose says square root of TP times FP, while the displayed equation correctly uses TN). The threshold of 30 attributes for choosing an autoencoder versus a feedforward net is arbitrary and never varied. Network architecture and hyperparameters are not reported, and no code is shipped. There is also an internal slip in Section III where the NUS2 description repeats the NUS1 selection rule, and NUS2's thresholding does not obviously enforce a 1:1 balance; Table IV shows it keeps 105, 161, 204, and 3045 majority samples depending on the dataset. None of these would matter if the main comparison were valid, but they make the paper hard to reproduce.\n\nMy take: this is a small, plausible addition to the undersampling toolbox, but the reported evidence is not evidence. It is useful mainly as a cautionary example of resampling leakage. If the authors reran the experiments with resampling strictly inside training folds and reported all hyperparameters, a serious referee could engage with it. As submitted, I would not send it to peer review; it needs to be reworked first.","headline":"A simple, clearly described undersampling idea whose central experimental claim is invalidated by resampling the whole dataset before classification.","tokens_in":14993,"tokens_out":4566,"would_cite":false,"duration_ms":48841,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a neural network on the minority class and keeping the farthest majority points improves imbalanced classification.","keywords":["undersampling","class imbalance","autoencoder","neural network","resampling","imbalanced classification","AUC","G-mean"],"falsifier":"Run the same four-dataset comparison with a strict protocol: split each dataset into training and test sets first, apply each undersampler only to the training portion, train the five classifiers on the resampled training data, and evaluate on the untouched test set. If NUS1 and NUS2 no longer produce the reported AUC, F1, and G-mean advantages, the central claim is an artifact of the undersampler selecting the test examples.","tokens_in":14005,"feed_emoji":"🧠","tokens_out":6798,"duration_ms":62420,"temperature":0.7,"pith_summary":"This paper proposes two undersampling algorithms, NUS1 ('hard') and NUS2 ('soft'), that use a neural network trained only on the minority class to score every majority-class example. The score is the Euclidean distance between the example's actual features and the features the minority-trained model predicts for it; the algorithms then keep majority examples with large reconstruction distance. The authors claim that, across four public datasets and five classifiers, these selections yield higher AUC, F1, and G-mean than most established undersampling techniques. If the claim holds, practitioners gain a resampling method that turns imbalanced data into a more separable balanced set without discarding all informative majority structure.","feed_headline":"Neural net undersampling beats classic resampling on imbalanced data","feed_subtitle":"Train on the minority class, keep majority points it fails to reconstruct, and classifiers separate better, per four public datasets.","key_machinery":"The load-bearing object is the minority-trained reconstruction model: an autoencoder when the input has more than 30 attributes, and a simple feedforward network otherwise. The model is not used to classify; it is used to generate, for every majority example $x$, a predicted feature vector $x'$, and the selection score is the squared Euclidean distance $\\|x - x'\\|_2^2$. Sorting majority samples by this score and keeping the largest distances is what creates the balanced, more separable dataset. The threshold 30 is the switch that picks the architecture, and NUS2's extra threshold—the maximum or half-average minority distance—is what softens selection for overlapping data.","core_discovery":"On the paper's own terms, the central discovery is that a minority-class reconstruction model can act as an effective undersampling oracle. NUS1 trains an autoencoder (for datasets with more than 30 attributes) or a simple feedforward network on the minority samples, predicts each majority sample's features with that model, computes the squared Euclidean distance $\\|x - x'\\|_2^2$ between prediction and actual values, and retains the $n_1$ majority samples with the largest distances, where $n_1$ is the minority count. NUS2 uses the same reconstruction distances but selects majority samples whose distance exceeds the maximum minority reconstruction distance or the average distance of the farthest half of minority samples, which the paper says handles overlapping classes. The paper reports that these methods outperform most other undersamplers in almost all classifier-dataset combinations and that NUS1 excels on non-overlapping data while NUS2 performs better on overlapping data.","pith_inferences":["My inference: the reconstruction-error score is essentially a one-class novelty measure, so comparing NUS1 and NUS2 against one-class classifiers or isolation-forest-based undersampling would test whether the specific neural-network training step is what drives the gain.","My inference: the hard/soft distinction suggests a continuum; a tunable quantile of the reconstruction-error distribution could interpolate between NUS1 and NUS2 and might remove the need to choose one algorithm up front.","My inference: if resampling is applied before the train/test split, as the paper's experimental description suggests, test examples are selected by the same scores that define the method; I would expect the reported margins to shrink under a protocol that resamples only training folds. This is my reading of the pipeline, not a claim the paper makes."],"forward_implications":["If NUS1 and NUS2 work as reported, a practitioner can replace distance-to-minority heuristics like NearMiss with a learned reconstruction distance, which adapts to feature correlations.","The paper's own experiments imply NUS1 should be preferred when classes are well separated, and NUS2 when classes overlap; the choice can be guided by visualizing resampled data or by validating on a holdout.","Because the method only needs the minority class to train the scoring model, it can be applied before any classifier, matching the paper's claim that gains are not classifier-specific.","The paper states that for very high imbalance or majority outliers, the greedy selection of farthest samples may discard useful majority information; the proposed future direction is to use prediction accuracy on majority samples to choose batches."],"supporting_citations":[{"why":"Supplies the Edited Nearest Neighbour baseline that the proposed methods are compared against.","marker":"[15]"},{"why":"Supplies the Tomek-links and All-KNN baselines, both distance-based undersamplers used in the comparison.","marker":"[9]"},{"why":"Supplies the NearMiss-1/2/3 baselines that select majority points by distance to the minority class.","marker":"[16]"},{"why":"Supplies the Neighbourhood Cleaning Rule baseline used in the comparisons.","marker":"[10]"},{"why":"Provides the toolbox implementation of baseline resamplers and the cluster-centroids undersampler.","marker":"[8]"},{"why":"Provides the classifiers and data utilities used to evaluate the resampled datasets.","marker":"[22]"},{"why":"Provides the neural-network and autoencoder implementation used by the proposed algorithms.","marker":"[25]"},{"why":"Supplies the four datasets on which the experimental comparison is run.","marker":"[28]"}],"fun_headline_variants":["Autoencoder-guided undersampling lifts AUC, F1, G-mean","Neural net keeps majority points it can't reconstruct, wins on imbalance","Undersample by reconstruction error: better than classic resamplers","Minority-mimic networks select majority outliers for balanced training","Reconstruction-based undersampling outperforms popular resampling techniques"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that classifying a dataset that was resampled before any train/test split tells you how the undersampler will perform on the original imbalanced distribution; the paper never states that resampling is restricted to training folds.","fun_headline_variants_meta":{"raw":{"variants":["Autoencoder-guided undersampling lifts AUC, F1, G-mean","Neural net keeps majority points it can't reconstruct, wins on imbalance","Undersample by reconstruction error: better than classic resamplers","Minority-mimic networks select majority outliers for balanced training","Reconstruction-based undersampling outperforms popular resampling techniques"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":3954,"prompt_tokens":866,"completion_tokens":3088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":482,"tokens_out":3088,"duration_ms":23469,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:43:43.395839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four-dataset comparison with a strict protocol: split each dataset into training and test sets first, apply each undersampler only to the training portion, train the five classifiers on the resampled training data, and evaluate on the untouched test set. If NUS1 and NUS2 no longer produce the reported AUC, F1, and G-mean advantages, the central claim is an artifact of the undersampler selecting the test examples.","supporting_citations":[{"cited_title":"Asymptotic properties of nearest neighbor rules using edited data","cited_arxiv_id":null,"evidence_quote":"Supplies the Edited Nearest Neighbour baseline that the proposed methods are compared against."},{"cited_title":"A generalization of the k-nn rule","cited_arxiv_id":null,"evidence_quote":"Supplies the Tomek-links and All-KNN baselines, both distance-based undersamplers used in the comparison."},{"cited_title":"knn approach to unbalanced data distribu- tions: a case study involving information extraction","cited_arxiv_id":null,"evidence_quote":"Supplies the NearMiss-1/2/3 baselines that select majority points by distance to the minority class."},{"cited_title":"Improving identification of difficult small classes by balancing class distribution","cited_arxiv_id":null,"evidence_quote":"Supplies the Neighbourhood Cleaning Rule baseline used in the comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the toolbox implementation of baseline resamplers and the cluster-centroids undersampler."},{"cited_title":"UCI machine learning repository, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the four datasets on which the experimental comparison is run."}],"review_version":1}