REVIEW 4 major objections 5 minor 37 references
PIPES: A Meta-dataset of Machine Learning Pipelines
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read The paper proposes PIPES, a meta-dataset built from all 9,408 combinations of five pipeline blocks applied to hundreds of datasets, claiming this completeness removes the preprocessing bias that distorts meta-learning on public repositories
desk verdict A useful new meta-dataset of 9,408 logged pipelines, but the paper's own evaluation (oracle best-pipeline comparison, no trained meta-model) doesn't support the 'overcomes OpenML' claim—still worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the pipeline space itself: a five-block tuple (imputation, encoding, scaling, feature preprocessing, classifier) whose blocks contain respectively 2, 3, 7, 14, and 16 techniques, including a 'None' option in the preprocessing blocks. Taking the Cartesian product yields 9,408 pipelines, and the design's work is to enforce equal frequency—each technique appears in exactly the same number of pipelines, so a meta-model cannot latch onto the popularity of a technique instead of its effect. The second essential mechanism is the record schema: for every pipeline and dataset the authors store the performance score, training time, testing time, and error text, plus the full predicti
What would settle it
Train a meta-learning recommender on the 9,408 pipeline results for a training split of datasets, then use it to rank pipelines on held-out datasets, and compare its top-1 accuracy to a recommender trained on records from the existing public repository restricted to the same datasets; if the repository-trained recommender matches or exceeds the PIPES-trained one, the claim that balanced completeness improves meta-learning is falsified.
Extended reading notes
Core claim
The central discovery claim is that a completeness-oriented experiment collection can overcome the selection bias of community-contributed repository records. Rather than sampling pipelines from what users happened to log, the authors construct the pipeline space as the Cartesian product T×E×P×A×C of imputation, encoding, scaling, feature-preprocessing, and classification techniques, and run every resulting pipeline on every dataset. The collection therefore stores all outcomes—not just the best one—so a meta-model can learn the joint effect of preprocessing and classifier choices. In a comparison on 192 datasets shared with the existing public repository, the best pipeline selected from PIP
Load-bearing premise
The load-bearing premise is that giving every pipeline technique equal representation in an exhaustive experiment collection yields meta-learning recommendations that transfer to new datasets; the paper's own comparison only checks the best logged pipeline per dataset rather than training a meta-model, and the authors list fixed hyperparameters and 280-of-300 dataset completion as limitations.
Editorial extensions
If this is right
- Meta-models trained on PIPES can recommend complete preprocessing-plus-classifier pipelines, not only a classifier, because all preprocessing blocks are represented in every combination.
- Because full predictions are stored, researchers can change evaluation metrics (e.g., from accuracy to balanced accuracy or F1) without re-running any experiment.
- The inclusion of training and testing times makes cost-sensitive algorithm selection feasible at scale: a user can trade performance against runtime for the same pipeline.
- Error messages recorded for invalid combinations (e.g., a missing-value dataset paired with a pipeline lacking imputation) provide training signal for meta-models to predict when a pipeline will fail.
- The repository's API allows the same exhaustive design to be extended with new datasets, new techniques, or additional blocks such as resampling.
Reading between the lines
- Inference: the most immediate high-value extension is to vary hyperparameters, since the current collection fixes them at defaults; doing so would turn PIPES from a pipeline-combination benchmark into a full algorithm-configuration benchmark.
- Inference: the equal-frequency design is transferable: the same 'run the full Cartesian product and log failures' template could be applied to regression, clustering, or multi-label tasks, where public records show the same preprocessing imbalance.
- Inference: the headline accuracy gap (0.84 vs 0.735) is best read as evidence of the larger search space rather than proof of transfer; a decisive test would train a meta-model on one source and evaluate its top recommendation on held-out datasets, which the paper does not do.
- Inference: because only 280 of 300 datasets were successfully processed, treating dataset completion status as a first-class field in the meta-dataset would let users filter and measure coverage-dependent bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PIPES, a meta-dataset of machine learning pipelines built by exhaustively applying 9,408 pipelines (combinations of imputation, encoding, scaling, feature preprocessing, and classification blocks) to OpenML datasets. The authors document OpenML's pipeline imbalance, describe the PIPES construction and API, and report two analyses: (RQ1) a comparison of best-performing SVC pipelines in PIPES versus OpenML on 192 common datasets, and (RQ2) a UMAP-based diversity assessment of the datasets. The paper claims that PIPES overcomes OpenML's limitations and enables less biased meta-learning.
Significance. If the dataset is delivered as described, PIPES is a potentially valuable community resource for meta-learning, especially for studying preprocessing effects and pipeline recommendation. The authors share code, data, predictions, timings, and error logs, and they provide an API for expansion. The formal description of the pipeline space and the exploratory analysis of OpenML are useful contributions. However, the paper's central validation claim—that PIPES improves meta-learning—rests on an oracle comparison that does not involve a trained meta-model, and the reported dataset size is inconsistent. These issues currently limit the strength of the contribution.
major comments (4)
- [Section V-B, Table II] The RQ1 evaluation compares, per dataset, the best SVC pipeline in OpenML (Eq. 3) with the best in PIPES (Eq. 5). Because the best pipeline is selected after observing test performance, this is an oracle comparison. PIPES evaluates a much larger and more varied candidate set, so the higher mean accuracy (0.840 vs. 0.735) and 180 wins are substantially entailed by construction; they do not demonstrate that a meta-model trained on M_PIPES outperforms one trained on M_OpenML. No meta-model is trained or evaluated on held-out datasets. The concluding sentence of Section V-B—that PIPES 'allows for less biased and more balanced meta-model learning'—is therefore not supported by the experiment as designed. The authors should either train and compare actual meta-models or reframe RQ1 as a claim about coverage rather than meta-learning superiority.
- [Section V-B] The paper does not state whether the OpenML accuracies used in Table II were recomputed under the same 5-fold cross-validation protocol as PIPES or taken from OpenML's stored runs. If the latter, differences in evaluation protocols, data splits, or preprocessing defaults could explain part of the accuracy gap. This is load-bearing for the win/loss comparison. Please specify the exact source of the OpenML accuracies and, if needed, rerun OpenML pipelines under the PIPES protocol.
- [Abstract, Section IV-A, Section V-C] The abstract and Section IV state that PIPES contains results for 300 datasets, and Eq. (2) quantifies over all D_i in D. However, Section V-C reports that 'only 280 datasets could be successfully processed and included in the analysis,' and Fig. 5 plots 280 datasets. This direct inconsistency affects a core contribution. The authors must clarify whether the released dataset contains 280 or 300 datasets, list the 20 excluded datasets, and explain the failure reasons. As written, users cannot determine the actual size of the resource.
- [Section V-C, Fig. 5] RQ2 is answered with a single UMAP scatter plot (k=3 neighbors) and the visual assertion that datasets 'cover the extent of the space.' No quantitative diversity metric, comparison to a baseline or OpenML dataset distribution, or sensitivity analysis for UMAP parameters is provided. The diversity claim is therefore not established. Please add quantitative measures (e.g., coverage, pairwise distances, or entropy) and compare against a relevant baseline.
minor comments (5)
- [Section III, Eq. (2)] The notation H, τ_train, τ_test, and ε is introduced informally after the equation. Define each symbol precisely (e.g., H is accuracy? which metric?) and specify the domain of ε.
- [Section III] Typo: 'feature preprocessing techinique' should be 'technique'.
- [Table II] The table title contains a typo: 'AANALYSIS' should be 'ANALYSIS'.
- [Fig. 4] The caption references 'Fig. 4c' twice for the scaling plots; the expected labels are probably 4b and 4d. Please correct.
- [Section IV-D] The software list reads 'Scikit-Learn (version 1.2.2) [34], (version 1.5.3)'; the second version is missing a package name.
Circularity Check
No significant circularity: PIPES is a constructed dataset artifact; the RQ1 comparison is a validity concern, not a circular reduction.
full rationale
The paper's central product is a meta-dataset, not a fitted prediction. The construction in Eq. 1 is an explicit Cartesian product of pipeline blocks, and the reported performances are measured outcomes of running those pipelines. The Section V-B comparison builds M_OpenML and M_PIPES by selecting the best pipeline per dataset (Eqs. 3-5) and comparing their accuracies. This is an oracle-style comparison, not a trained meta-model, so the conclusion that PIPES 'allows for less biased and more balanced meta-model learning' is a design claim inferred from the data rather than a derived result; that is a methodological limitation, not circularity. The win/loss margin in Table II is not forced by construction: OpenML contains pipelines outside PIPES's selected blocks (e.g., VarianceThreshold appears in Fig. 4a and is not in PIPES's feature-preprocessing list), and PIPES uses default hyperparameters, so OpenML could in principle have had the better best pipeline. The self-citations ([16], [27]) support motivation and related work only and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The paper's own Limitations section (hyperparameters not considered, missing blocks such as balancing) is acknowledged and does not conceal circularity.
Assumptions & free parameters
free parameters (2)
- Dataset selection thresholds =
instances [100, 100000], classes [2, 100], attributes [5, 100]
- UMAP neighbor count k =
3
assumptions (4)
- domain assumption OpenML records of pipeline blocks and techniques accurately reflect the experiments logged by users
- domain assumption The 280 successfully processed datasets, after family-based exclusion and computational failures, are representative of the broader OpenML population
- domain assumption Default scikit-learn hyperparameters provide a fair basis for comparing pipeline components
- ad hoc to paper UMAP projection with k=3 preserves enough structure to judge dataset diversity
Cite this review
Pith. "Pith review of PIPES: A Meta-dataset of Machine Learning Pipelines." pith.science (2026). https://pith.science/paper/SGOPC4ZW
@misc{pith2026250909512,
author = {Pith},
title = {Pith review of: PIPES: A Meta-dataset of Machine Learning Pipelines},
year = {2026},
howpublished = {\url{https://pith.science/paper/SGOPC4ZW}},
note = {Machine review of arXiv:2509.09512}
}
read the original abstract
Solutions to the Algorithm Selection Problem (ASP) in machine learning face the challenge of high computational costs associated with evaluating various algorithms' performances on a given dataset. To mitigate this cost, the meta-learning field can leverage previously executed experiments shared in online repositories such as OpenML. OpenML provides an extensive collection of machine learning experiments. However, an analysis of OpenML's records reveals limitations. It lacks diversity in pipelines, specifically when exploring data preprocessing steps/blocks, such as scaling or imputation, resulting in limited representation. Its experiments are often focused on a few popular techniques within each pipeline block, leading to an imbalanced sample. To overcome the observed limitations of OpenML, we propose PIPES, a collection of experiments involving multiple pipelines designed to represent all combinations of the selected sets of techniques, aiming at diversity and completeness. PIPES stores the results of experiments performed applying 9,408 pipelines to 300 datasets. It includes detailed information on the pipeline blocks, training and testing times, predictions, performances, and the eventual error messages. This comprehensive collection of results allows researchers to perform analyses across diverse and representative pipelines and datasets. PIPES also offers potential for expansion, as additional data and experiments can be incorporated to support the meta-learning community further. The data, code, supplementary material, and all experiments can be found at https://github.com/cynthiamaia/PIPES.git.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
The algorithm selection problem,
J. R. Rice, “The algorithm selection problem,” inAdvances in Comput- ers, Elsevier, vol. 15, pp. 65–118, 1976
1976
-
[2]
Cross-disciplinary perspectives on meta-learning for algorithm selection,
K. A. Smith-Miles, “Cross-disciplinary perspectives on meta-learning for algorithm selection,”ACM Computing Surveys (CSUR), vol. 41, no. 1, pp. 1–25, 2009
2009
-
[3]
Automatic recommendation of classification algorithms based on data set characteristics,
Q. Song, G. Wang, and C. Wang, “Automatic recommendation of classification algorithms based on data set characteristics,”Pattern Recognition, Elsevier, vol. 45, no. 7, pp. 2672–2689, 2012
2012
-
[4]
Ranking and selecting clustering algorithms using a meta-learning approach,
M. C. P. De Souto, R. B. C. Prudencio, R. G. F. Soares, D. S. A. De Araujo, I. G. Costa, T. B. Ludermir, and A. Schliep, “Ranking and selecting clustering algorithms using a meta-learning approach,” in IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 3729–3735, 2008
2008
-
[5]
A new data char- acterization for selecting clustering algorithms using meta-learning,
B. A. Pimentel and A. C. P. L. F. De Carvalho, “A new data char- acterization for selecting clustering algorithms using meta-learning,” Information Sciences, Elsevier, vol. 477, pp. 203–219, 2019
2019
-
[6]
Clustering algorithm selection by meta-learning systems: A new distance-based problem characterization and ranking combination methods,
D. G. Ferrari and L. N. De Castro, “Clustering algorithm selection by meta-learning systems: A new distance-based problem characterization and ranking combination methods,”Information Sciences, Elsevier, vol. 301, pp. 181–194, 2015
2015
-
[7]
A literature survey and empirical study of meta-learning for classifier selection,
I. Khan, X. Zhang, M. Rehman, and R. Ali, “A literature survey and empirical study of meta-learning for classifier selection,”IEEE Access, vol. 8, pp. 10262–10281, 2020
2020
-
[8]
Hutter, L
F. Hutter, L. Kotthoff, and J. Vanschoren,Automated Machine Learning: Methods, Systems, Challenges, Springer Nature, 2019
2019
Show all 37 references
-
[9]
Brazdil, J
P. Brazdil, J. N. Van Rijn, C. Soares, and J. Vanschoren,Metalearn- ing: Applications to Automated Machine Learning and Data Mining, Springer Nature, 2022
2022
-
[10]
OpenML bench- marking suites,
B. Bischl, G. Casalicchio, M. Feurer, P. Gijsbers, F. Hutter, M. Lang, R. G. Mantovani, J. N. Van Rijn, and J. Vanschoren, “OpenML bench- marking suites,” inarXiv preprint arXiv:1708.03731, 2017
2017 arXiv
-
[11]
J. N. Van Rijn, B. Bischl, L. Torgo, B. Gao, V . Umaashankar, S. Fischer, P. Winter, B. Wiswedel, M. R. Berthold, and J. Vanschoren,OpenML: A Collaborative Science Platform, Springer, inMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013,...
2013
-
[12]
K ¨uhn, P
D. K ¨uhn, P. Probst, J. Thomas, and B. Bischl,Automatic Explo- ration of Machine Learning Experiments on OpenML, arXiv preprint, arXiv:1806.10961, 2018
2018 arXiv
-
[13]
Perrone, R
V . Perrone, R. Jenatton, M. W. Seeger, and C. Archambeau,Scalable Hyperparameter Transfer Learning, inAdvances in Neural Information Processing Systems, vol. 31, 2018
2018
-
[14]
H. S. Obaid, S. A. Dheyab, and S. S. Sabry,The Impact of Data Pre- Processing Techniques and Dimensionality Reduction on the Accuracy of Machine Learning, IEEE, in2019 9th Annual Information Technol- ogy, Electromechanical Engineering and Microelectronics Conference (IEMECON),...
2019
-
[15]
V . N. G. Raju, K. P. Lakshmi, V . M. Jain, A. Kalidindi, and V . Padma, Study the Influence of Normalization/Transformation Process on the Accuracy of Supervised Classification, IEEE, in2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), pp...
2020
-
[16]
L. B. V . De Amorim, G. D. C. Cavalcanti, and R. M. O. Cruz,The Choice of Scaling Technique Matters for Classification Performance, Elsevier, inApplied Soft Computing, vol. 133, pp. 109924, 2023
2023
-
[17]
P. B. Pio, A. Rivolli, A. C. P. L. F. De Carvalho, and L. P. F. Garcia, A Review on Preprocessing Algorithm Selection with Meta-Learning, Springer, inKnowledge and Information Systems, vol. 66, no. 1, pp. 1–28, 2024
2024
-
[18]
Brazdil, C
P. Brazdil, C. G. Carrier, C. Soares, and R. Vilalta,Metalearning: Applications to Data Mining, Springer Science & Business Media, 2008
2008
-
[19]
Castiello, G
C. Castiello, G. Castellano, and A. M. Fanelli,Meta-Data: Character- ization of Input Features for Meta-Learning, Springer, inInternational Conference on Modeling Decisions for Artificial Intelligence, pp. 457– 468, 2005
2005
-
[20]
Rivolli, L
A. Rivolli, L. P. F. Garcia, C. Soares, J. Vanschoren, and A. C. P. L. F. De Carvalho,Meta-Features for Meta-Learning, Elsevier, inKnowledge- Based Systems, vol. 240, pp. 108101, 2022
2022
-
[21]
X. Zhu, X. Yang, C. Ying, and G. Wang,A New Classification Algo- rithm Recommendation Method Based on Link Prediction, Elsevier, in Knowledge-Based Systems, vol. 159, pp. 171–185, 2018
2018
-
[22]
Mustafa and M
M. Mustafa and M. Sebag,ALORS: An Algorithm Recommender System for Classifiers, inArtificial Intelligence, vol. 244, pp. 291–314, 2017
2017
-
[23]
G. Wang, Q. Song, X. Zhang, and K. Zhang,A Generic Multilabel Learning-Based Classification Algorithm Recommendation Method, in ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 9, no. 1, pp. 1–30, 2014
2014
-
[24]
A. L. Dantas and A. T. R. Pozo,Selecting Algorithms for the Quadratic Assignment Problem with a Multi-Label Meta-Learning Approach, in 2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pp. 175–180, 2018
2018
-
[25]
I. Khan, X. Zhang, R. K. Ayyasamy, and R. Ali,AutoFe-Sel: A Meta-Learning Based Methodology for Recommending Feature Subset Selection Algorithms, inKSII Transactions on Internet & Information Systems, vol. 17, no. 7, 2023
2023
-
[26]
Resampling strategies for imbalanced regression: a survey and empirical analysis,
J. G. Avelino, G. D. C. Cavalcanti, and R. M. O. Cruz, “Resampling strategies for imbalanced regression: a survey and empirical analysis,” inArtificial Intelligence Review, Springer, vol. 57, no. 4, pp. 82, 2024
2024
-
[27]
Meta-scaler: A meta-learning framework for the selection of scaling techniques,
L. B. V . de Amorim, G. D. C. Cavalcanti, and R. M. O. Cruz, “Meta-scaler: A meta-learning framework for the selection of scaling techniques,” inIEEE Transactions on Neural Networks and Learning Systems, IEEE, 2024
2024
-
[28]
Zhang and Q
X. Zhang and Q. Song,A Multi-Label Learning Based Kernel Automatic Recommendation Method for Support Vector Machine, inPloS One, vol. 10, no. 4, pp. e0120455, 2015
2015
-
[29]
Vanschoren, H
J. Vanschoren, H. Blockeel, B. Pfahringer, and G. Holmes,Experiment Databases: A New Way to Share, Organize and Learn from Experiments, inMachine Learning, vol. 87, pp. 127–158, 2012
2012
-
[30]
Vanschoren, J
J. Vanschoren, J. N. Van Rijn, B. Bischl, and L. Torgo,OpenML: Networked Science in Machine Learning, inACM SIGKDD Explorations Newsletter, vol. 15, no. 2, pp. 49–60, 2014
2014
-
[31]
C. S. Bojer and J. P. Meldgaard,Kaggle Forecasting Competitions: An Overlooked Learning Opportunity, inInternational Journal of Forecast- ing, vol. 37, no. 2, pp. 587–603, 2021
2021
-
[32]
C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, Nas-Bench-101: Towards Reproducible Neural Architecture Search, in International Conference on Machine Learning, pp. 7105–7114, 2019
2019
-
[33]
Feurer, A
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter,Efficient and Robust Automated Machine Learning, inAdvances in Neural Information Processing Systems, vol. 28, 2015
2015
-
[34]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, et al.,Scikit- learn: Machine Learning in Python, inJournal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011
2011
-
[35]
Feurer, J
M. Feurer, J. N. Van Rijn, A. Kadra, P. Gijsbers, N. Mallik, S. Ravi, A. M ¨uller, J. Vanschoren, and F. Hutter,Openml-Python: An Extensible Python API for OpenML, inJournal of Machine Learning Research, vol. 22, no. 100, pp. 1–5, 2021
2021
-
[36]
Alcobac ¸a, F
E. Alcobac ¸a, F. Siqueira, A. Rivolli, L. P. F. Garcia, J. T. Oliva, and A. C. P. L. F. De Carvalho,MFE: Towards Reproducible Meta-Feature Extraction, inJournal of Machine Learning Research, vol. 21, no. 111, pp. 1–5, 2020
2020
-
[37]
McInnes, J
L. McInnes, J. Healy, and J. Melville,UMAP: Uniform Manifold Ap- proximation and Projection for Dimension Reduction, inarXiv preprint arXiv:1802.03426, 2018
2018 arXiv
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.