REVIEW 3 major objections 5 minor 48 references
Meta-Instance Selection. Instance Selection as a Classification Problem with Meta-Features
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Instance selection can be reframed as binary classification over nearest-neighbor graph statistics, so a trained meta-classifier prunes a new dataset in one pass with accuracy comparable to five reference instance-selection methods and…
desk verdict The per-instance meta-classifier idea is real and the constrained comparison is honest; the headline 'outperforms almost all cases' is overstated because it relies on an unrestricted area metric that rewards extra compression even when accuracy drops. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the meta-feature space built from the nearest-neighbor graph (NNG). For each vertex (training sample), the method records the average and minimum squared Euclidean distance to neighbors in the same class, the opposite class, and any class, plus the counts of same-class and opposite-class neighbors among the k nearest, computed for k = 3, 5, 9, 15, 23, 33, with -1 inserted for missing opposite-class values. These statistics are normalized by feature count so datasets of different dimensionality become comparable, and the same fixed-size descriptor vector is extracted for every dataset. The meta-classifier, Balanced Random Forest in the recommended setup, maps these descriptors to a removal probability, and a threshold converts that probability into a keep/remove decision; the AUARR metric, area under the accuracy-versus-reduction-rate curve, is introduced to compare the resulting continuum of operating points against the single point returned by a reference algorithm.
What would settle it
Train the meta-classifier on one set of tabular benchmarks and evaluate it on held-out image or text collections with the same reference labels; if the unrestricted AUARR on those collections is no better than always keeping everything, or is worse than a threshold tuned per dataset, the claimed generality of NNG meta-features fails.
Extended reading notes
Core claim
On its own terms, the paper claims that any instance-selection algorithm that works through the nearest-neighbor graph leaves a learnable signature: the local geometry around each training vector, summarized by distance and neighbor-count statistics, determines whether the algorithm keeps or removes that vector. By extracting the same fixed set of these statistics for every dataset and labeling each vector with keep/remove decisions from a reference algorithm, the authors build a single meta-dataset on which a Balanced Random Forest classifier is trained; applying that classifier to a new dataset's nearest-neighbor graph yields a removal probability per instance, and thresholding it reproduces reference-method behavior. In leave-one-dataset-out experiments over 17 tabular datasets and five reference methods (ENN, Drop3, ICF, HMN-EI, CCIS), the resulting meta-instance selection is statistically indistinguishable from ENN, HMN-EI, and CCIS on the compression-limited area-under-curve metric, significantly better than ICF, and significantly worse than Drop3; when the comparison area is not capped by the reference reduction rate, the meta-model wins in all but the Drop3 comparison. The advertised payoff is complexity: selection is a single pass over the dataset, with runtime governed only by nearest-neighbor finding and a constant-cost meta-classifier, yielding measured speedups up to 219x over Drop3.
Load-bearing premise
The load-bearing premise is that statistics of the nearest-neighbor graph are shared across domains, so that a classifier trained on meta-features from some datasets can decide keep/remove on a dataset it has never seen.
Editorial extensions
If this is right
- A new dataset can be pruned in a single pass: compute nearest neighbors, extract meta-features, and apply the meta-classifier, without iterating through candidate removals.
- For large datasets and slow reference algorithms, the measured speedup grows with dataset size, reaching about 219x over Drop3 in the reported experiments.
- Users can trade compression against accuracy after the fact by setting a threshold on the removal probability, instead of rerunning the selection algorithm.
- The approach is generic across NNG-based instance-selection methods: the same pipeline reproduces or improves the behavior of ENN, HMN-EI, CCIS, and ICF, and in the unrestricted AUARR comparison it significantly beats four of the five reference methods, with Drop3 the only exception.
- With nearest-neighbor index structures, the selection phase can run in log-linear or near-linear time, independent of the reference algorithm's iterative cost.
Reading between the lines
- If transferability holds beyond the 17 tabular benchmarks, the method turns any expensive iterative instance-selection algorithm into a one-time training cost plus per-dataset nearest-neighbor construction, making nearest-neighbor search the practical bottleneck.
- The continuous removal probability provides a built-in compression-accuracy curve, so a natural extension is to use that probability as a sample-importance score for active learning, noisy-label detection, or budget-constrained training, though the paper does not test those uses.
- Because the feature-importance analysis shows each reference method relies on different NNG descriptors, an ensemble of reference-label-specific meta-classifiers could carry complementary information; the paper mentions diversity as a motivation but does not build the ensemble.
- A direct test of the core assumption would be applying the same pipeline to non-tabular data such as image patches or text embeddings; the paper's experiments are limited to tabular datasets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MetaIS, a method that recasts instance selection as a per-instance binary classification problem. Each training sample is described by meta-features extracted from the nearest-neighbor graph, and a meta-classifier predicts whether the sample should be kept or removed. The meta-classifier is trained on labels produced by five reference instance selection algorithms (ENN, Drop3, ICF, HMN-EI, CCIS) applied to multiple source datasets, and is evaluated on held-out datasets using a leave-one-dataset-out protocol. The authors report accuracy-reduction tradeoffs on tabular benchmarks, execution-time comparisons, an analysis of meta-classifier choice, and a feature-importance study. The central claims are that MetaIS achieves results comparable to reference methods, that it 'outperforms the reference IS in almost all cases' under one metric, and that its selection cost is dominated by nearest-neighbor graph construction plus a single classifier pass.
Significance. The core idea of amortizing iterative instance selection into a trained meta-classifier is interesting and potentially useful: if the meta-learner transfers across datasets, expensive multi-pass pruning could be replaced by a single-pass prediction. The paper has real strengths: it provides a public implementation (MetaIS library), uses a methodologically appropriate leave-one-dataset-out protocol, reports statistical tests, and introduces both an unrestricted and a reduction-limited area metric. However, the strongest advertised advantage—outperforming reference methods 'in almost all cases'—is an artifact of the unrestricted AUARR metric. The matched-reduction AUARR_L results support only comparability, with mixed individual outcomes. The computational-complexity claim is plausible algorithmically but the empirical speedups are not measured under controlled implementation conditions. With the evaluation framing corrected, the paper would present a useful, though more modest, contribution.
major comments (3)
- [§4.4, Table 3, §5.1]
- [§5.2, Tables 4 and 5]
- [§3, first paragraph]
minor comments (5)
- [Throughout]
- [Tables 1-3]
- [Figure 8 caption]
- [Table 4]
- [References]
Circularity Check
MetaIS is a genuine meta-classifier trained on external reference labels and evaluated on held-out datasets; no load-bearing self-citation or definitional reduction found.
full rationale
The paper's central chain is not circular. The meta-classifier is trained on k-NN meta-features extracted from NNGs and labeled by external reference instance selection algorithms (ENN, Drop3, ICF, HMN-EI, CCIS), then evaluated on datasets excluded by leave-one-dataset-out. The predicted keep/remove decisions are not the same quantity used to fit any parameter: the reference algorithms define the training labels, but the meta-features are fixed statistics of the NNG, and the held-out evaluation uses independent 5-fold test data. The computational-complexity claim follows directly from the single-pass design (NNG construction plus random forest prediction) and is not derived from the experimental results. Self-citations [6, 20, 23, 44] appear only as related work or as the implementation library used in experiments; they do not carry the central claim, so they are not load-bearing. The unrestricted AUARR comparison (Table 3 versus Table 2) is a legitimate evaluation-fairness concern because extending the MetaIS curve to higher reduction adds area even when accuracy falls, but this is a metric artifact rather than circularity: it does not make the prediction equivalent to the training labels or to the reference output by construction. Under the reduction-capped AUARR_L in Table 2, the results are more moderate, but that only weakens the empirical claim; it does not create a circular derivation.
Assumptions & free parameters
free parameters (3)
- Neighbor counts k grid =
{3, 5, 9, 15, 23, 33}
- Meta-feature design =
8 descriptors per k value
- Acceptance threshold Theta =
0.1 to 0.9 grid
assumptions (3)
- domain assumption Local nearest neighbor graph features contain enough information to predict instance selection decisions across datasets
- domain assumption Squared Euclidean distance normalized by feature count makes distances comparable across datasets with different dimensionality
- domain assumption Reference instance selection algorithms with k=3 provide reliable labels for meta-training
Cite this review
Pith. "Pith review of Meta-Instance Selection. Instance Selection as a Classification Problem with Meta-Features." pith.science (2026). https://pith.science/paper/HCOWUEI4
@misc{pith2026250111526,
author = {Pith},
title = {Pith review of: Meta-Instance Selection. Instance Selection as a Classification Problem with Meta-Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/HCOWUEI4}},
note = {Machine review of arXiv:2501.11526}
}
read the original abstract
Data pruning, or instance selection, is an important problem in machine learning especially in terms of nearest neighbour classifier. However, in data pruning which speeds up the prediction phase, there is an issue related to the speed and efficiency of the process itself. In response, the study proposes an approach involving transforming the instance selection process into a classification task conducted in a unified meta-feature space where each instance can be classified and assigned to either the "to keep" or "to remove" class. This approach requires training an appropriate meta-classifier, which can be developed based on historical instance selection results from other datasets using reference instance selection methods as a labeling tool. This work proposes constructing the meta-feature space based on properties extracted from the nearest neighbor graph. Experiments conducted on 17 datasets of varying sizes and five reference instance selection methods (ENN, Drop3, ICF, HMN-EI, and CCIS) demonstrate that the proposed solution achieves results comparable to reference instance selection methods while significantly reducing computational complexity. In the proposed approach, the computational complexity of the system depends only on identifying the k-nearest neighbors for each data sample and running the meta-classifier. Additionally, the study discusses the choice of meta-classifier, recommending the use of Balanced Random Forest.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, A. Morcos, Beyond neu- ral scaling laws: beating power law scaling via data pruning, Advances in Neural Information Processing Systems 35 (2022) 19523–19536
work page 2022
-
[2]
M. H. Jarrahi, A. Memariani, S. Guha, The principles of data-centric ai, Communications of the ACM 66 (8) (2023) 84–92
work page 2023
- [3]
-
[4]
H. Brighton, C. Mellish, Advances in instance selection for instance- based learning algorithms, Data mining and knowledge discovery 6 (2) (2002) 153–172
work page 2002
- [5]
-
[6]
M. Blachnik, M. Kordos, Comparison of instance selection and construc- tion methods with various classifiers, Applied Sciences 10 (11) (2020) 3933
work page 2020
- [7]
-
[8]
B. Saha, D. Srivastava, Data quality: The other face of big data, in: 2014 IEEE 30th international conference on data engineering, IEEE, 2014, pp. 1294–1297
work page 2014
Show all 48 references
-
[9]
A. Jain, H. Patel, L. Nagalapatti, N. Gupta, S. Mehta, S. Guttula, S. Mujumdar, S. Afzal, R. Sharma Mittal, V. Munigala, Overview and importance of data quality for machine learning tasks, in: Proceedings of the 26th ACM SIGKDD international conference on knowledge dis- covery...
2020
-
[10]
Singh, Systematic review of data-centric approaches in artificial in- telligence and machine learning, Data Science and Management 6 (3) (2023) 144–157
P. Singh, Systematic review of data-centric approaches in artificial in- telligence and machine learning, Data Science and Management 6 (3) (2023) 144–157
2023
-
[11]
D. Zha, Z. P. Bhat, K.-H. Lai, F. Yang, Z. Jiang, S. Zhong, X. Hu, Data- centric artificial intelligence: A survey, arXiv preprint arXiv:2303.10158 (2023)
2023 arXiv
-
[12]
Hart, The condensed nearest neighbor rule., IEEE Trans
P. Hart, The condensed nearest neighbor rule., IEEE Trans. on Infor- mation Theory 16 (1968) 515–516
1968
-
[13]
Wilson, Assymptotic properties of nearest neighbour rules using edited data., IEEE Trans
D. Wilson, Assymptotic properties of nearest neighbour rules using edited data., IEEE Trans. on Systems, Man, and Cybernetics SMC-2 (1972) 408–421
1972
-
[14]
Jankowski, M
N. Jankowski, M. Grochowski, Comparison of instance selection algo- rithms. i. algorithms survey, LNCS 3070 (2004) 598–603
2004
-
[15]
García, J
S. García, J. Derrac, J. R. Cano, F. Herrera, Prototype selection for nearest neighbor classification: Taxonomy and empirical study, Pattern Analysis and Machine Intelligence, IEEE Transactions on 34 (3) (2012) 417–435
2012
-
[16]
Marchiori, Hit miss networks with applications to instance selection, Journal of Machine Learning Research 9 (Jun) (2008) 997–1017
E. Marchiori, Hit miss networks with applications to instance selection, Journal of Machine Learning Research 9 (Jun) (2008) 997–1017
2008
-
[17]
Cunha, F
W. Cunha, F. Viegas, C. França, T. Rosa, L. Rocha, M. A. Gonçalves, A comparative survey of instance selection methods applied to non- neural and transformer-based text classification, ACM Computing Sur- veys 55 (13s) (2023) 1–52
2023
-
[18]
Leyva, A
E. Leyva, A. González, R. Pérez, Three new instance selection methods based on local sets: A comparative study with several approaches from a bi-objective perspective, Pattern Recognition 48 (4) (2015) 1523–1537
2015
-
[19]
Malhat, M
M. Malhat, M. El Menshawy, H. Mousa, A. El Sisi, A new approach for instance selection: Algorithms, evaluation, and comparisons, Expert Systems with Applications 149 (2020) 113297
2020
-
[20]
Blachnik, Ensembles of instance selection methods
M. Blachnik, Ensembles of instance selection methods. a comparative study., International Journal of Applied Mathematics and Computer Science 29 (1) (2019). 35
2019
-
[21]
de Haro-García, G
A. de Haro-García, G. Cerruela-García, N. García-Pedrajas, Instance selection based on boosting for instance-based learners, Pattern Recog- nition 96 (2019) 106959
2019
-
[22]
S. M. H. Mousavi, V. Charles, T. Gherman, An evolutionary pentagon support vector finder method, Expert Systems with Applications 150 (2020) 113284
2020
-
[23]
Kordos, M
M. Kordos, M. Blachnik, R. Scherer, Fuzzy clustering decomposition of genetic algorithm-based instance selection for regression problems, Information Sciences 587 (2022) 23–40
2022
-
[24]
J. N. van Rijn, S. M. Abdulrahman, P. Brazdil, J. Vanschoren, Fast algorithm selection using learning curves, in: Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings 14, Springer, 2015, pp. 298–309
2015
-
[25]
S. M. Abdulrahman, P. Brazdil, J. N. van Rijn, J. Vanschoren, Speed- ing up algorithm selection using average ranking and active testing by introducing runtime, Machine learning 107 (2018) 79–108
2018
-
[26]
Y. Peng, P. A. Flach, C. Soares, P. Brazdil, Improved dataset charac- terisation for meta-learning, in: International Conference on Discovery Science, Springer, 2002, pp. 141–152
2002
-
[27]
Castiello, G
C. Castiello, G. Castellano, A. Fanelli, Meta-data: Characterization of input features for meta-learning, in: Proceedings of the Second Inter- national Conference on Modeling Decisions for Artificial Intelligence, LNAI 3558, 2005, pp. 457–468
2005
-
[28]
M.Reif, F.Shafait, A.Dengel, Meta-learningforevolutionaryparameter optimization of classifiers, Machine Learning 87 (3) (2012) 357–380
2012
-
[29]
Leite, P
R. Leite, P. Brazdil, Exploiting performance-based similarity between datasets in metalearning, in: AAAI Workshop on Meta-Learning and MetaDL Challenge, PMLR, 2021, pp. 90–99
2021
-
[30]
Rivolli, L
A. Rivolli, L. P. Garcia, C. Soares, J. Vanschoren, A. C. de Carvalho, Meta-features for meta-learning, Knowledge-Based Systems 240 (2022) 108101. 36
2022
-
[31]
T. Wei, Z. Wu, R. Li, Z. Hu, F. Feng, X. He, Y. Sun, W. Wang, Fast adaptation for cold-start collaborative filtering with meta-learning, in: 2020 IEEE International Conference on Data Mining (ICDM), IEEE, 2020, pp. 661–670
2020
-
[32]
Leyva, Y
E. Leyva, Y. Caises, A. González, R. Pérez, On the use of meta-learning for instance selection: An architecture and an experimental study, In- formation Sciences 266 (2014) 16–30
2014
-
[33]
Pinto, C
F. Pinto, C. Soares, J. Mendes-Moreira, Towards automatic generation of metafeatures, in: Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer, 2016, pp. 215–226
2016
-
[34]
Chubet, P
O. Chubet, P. Parikh, D. R. Sheehy, S. Sheth, Proximity search in the greedy tree, in: Symposium on Simplicity in Algorithms (SOSA), SIAM, 2023, pp. 332–342
2023
-
[35]
P. Ram, K. Sinha, Revisiting kd-tree for nearest neighbor search, in: Proceedings of the 25th acm sigkdd international conference on knowl- edge discovery & data mining, 2019, pp. 1378–1388
2019
-
[36]
Y. Chen, L. Zhou, Y. Tang, J. P. Singh, N. Bouguila, C. Wang, H. Wang, J. Du, Fast neighbor search by using revised kd tree, Information Sci- ences 472 (2019) 145–162
2019
-
[37]
Cheng, J
D. Cheng, J. Huang, S. Zhang, Q. Wu, A robust method based on lo- cality sensitive hashing for k-nearest neighbors searching, Wireless Net- works 30 (5) (2024) 4195–4208
2024
-
[38]
Jafari, P
O. Jafari, P. Maurya, P. Nagarkar, K. M. Islam, C. Crushev, A survey on locality sensitive hashing algorithms and their applications, arXiv preprint arXiv:2102.08942 (2021)
2021 arXiv
-
[40]
Gates, The reduced nearest neighbor rule (corresp.), IEEE transac- tions on information theory 18 (3) (1972) 431–433
G. Gates, The reduced nearest neighbor rule (corresp.), IEEE transac- tions on information theory 18 (3) (1972) 431–433. 37
1972
-
[41]
D. R. Wilson, T. R. Martinez, Instance pruning techniques, in: ICML, Vol. 97, 1997, pp. 403–411
1997
-
[42]
Marchiori, Class conditional nearest neighbor for large margin in- stance selection, IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (2) (2010) 364–370
E. Marchiori, Class conditional nearest neighbor for large margin in- stance selection, IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (2) (2010) 364–370
2010
-
[43]
Triguero, S
I. Triguero, S. González, J. M. Moyano, S. García, J. Alcalá-Fdez, J. Lu- engo, A. Fernández, M. J. del Jesús, L. Sánchez, F. Herrera, Keel 3.0: an open source software for multi-stage analysis in data mining, Inter- national Journal of Computational Intelligence Systems 10 (1...
2017
-
[44]
Blachnik, M
M. Blachnik, M. Kordos, Information selection and data compression rapidminer library, in: Machine Intelligence and Big Data in Industry, Springer, 2016, pp. 135–145
2016
-
[45]
Hofmann, R
M. Hofmann, R. Klinkenberg, RapidMiner: Data mining use cases and business analytics applications, CRC Press, 2016
2016
-
[46]
Feurer, J
M. Feurer, J. N. Van Rijn, A. Kadra, P. Gijsbers, N. Mallik, S. Ravi, A. Müller, J. Vanschoren, F. Hutter, Openml-python: an extensible python api for openml, Journal of Machine Learning Research 22 (100) (2021) 1–5
2021
-
[47]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O.Grisel, M.Blondel, P.Prettenhofer, R.Weiss, V.Dubourg, J.Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Researc...
2011
-
[48]
Lemaître, F
G. Lemaître, F. Nogueira, C. K. Aridas, Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning, Journal of Machine Learning Research 18 (17) (2017) 1–5. URL http://jmlr.org/papers/v18/16-365.html
2017
-
[49]
Louppe, L
G. Louppe, L. Wehenkel, A. Sutera, P. Geurts, Understanding variable importances in forests of randomized trees, Advances in neural informa- tion processing systems 26 (2013). 38
2013
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.