Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that, across nearly 300 datasets and three supervised tasks, no AutoML tool consistently beats all others; H2O-Automl, Auto-keras, and Auto-sklearn outperform Ludwig, Darwin, TPOT, and Auto-ml, with H2O-Automl best at…

desk verdict A useful, honest AutoML benchmark whose headline ranking is really a 15-minute-budget ranking; worth refereeing, but the authors should scope the claim and add longer-budget stability evidence. read the letter →

arxiv 1908.05557 v2 pith:72DNKCTM submitted 2019-08-15 cs.LG stat.ML

Anh Truong , Austin Walters , Jeremy Goodsitt , Keegan Hines , C. Bayan Bruss , Reza Farivar This is my paper · ORCID
classification cs.LGstat.ML
keywords automatedmachinelearningAutoMLtoolsbenchmarkevaluationhyperparameteroptimizationmodelselectionbinaryclassificationmulticlassregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how well current AutoML tools automate the machine learning pipeline and compares seven tools on hundreds of datasets across three supervised tasks. It claims that no tool dominates: H2O-Automl, Auto-keras, and Auto-sklearn generally outperform Ludwig, Darwin, TPOT, and Auto-ml, but the winner depends on the task. H2O-Automl is best for binary classification and regression, Auto-keras is best for multiclass classification, and Auto-sklearn ties on all tasks but needs more time. The practical upshot is that AutoML is not yet a one-size-fits-all solution; choosing a tool means trading off task type, convergence speed, and robustness.

What carries the argument

The argument is carried by a controlled evaluation protocol built on the common three-stage AutoML pipeline: data preprocessing, model selection with hyperparameter optimization, and model interpretation. The tools are compared on data segments defined by sample size, feature dimension, categorical-feature ratio, missing proportion, and class imbalance, using accuracy (or F1 for imbalanced data) and mean squared error. A 15-minute time limit was chosen so that all tools complete at least 70 percent of datasets, and the same train/test splits are fed to every tool. This protocol turns 'which tool is better' into a measurable question, with completion rates, convergence curves, and robustness checks as supporting evidence.

What would settle it

Run the same tools on raw, uncleaned datasets with missing values, categorical text, and mixed data types, using both 15-minute and 3-hour budgets; if a tool ranked low in this paper (Ludwig, TPOT, or Darwin) reaches or beats the accuracy or mean squared error of H2O-Automl and Auto-keras on those raw tasks, the paper's conclusion that those three tools are generically better is falsified.

Watch

Extended reading notes

Core claim

The central claim is that, on nearly 300 benchmark datasets with a 15-minute time limit, the AutoML tools form a clear performance hierarchy even though none is universally best. H2O-Automl, Auto-keras, and Auto-sklearn outperformed Ludwig, Darwin, TPOT, and Auto-ml across binary classification, multiclass classification, and regression. H2O-Automl slightly leads binary classification and regression and converges quickly; Auto-keras slightly leads multiclass classification and is stable; Auto-sklearn ties on all tasks but needs longer runs. The paper also finds that most tools improve given more time, that H2O-Automl and Ludwig are the most robust across repeated runs, and that Darwin frequently fails to finish within the 15-minute limit.

Load-bearing premise

The evaluation feeds every tool the same already-preprocessed numerical datasets, so it measures only model selection and hyperparameter search, not data preprocessing or feature engineering; if those tasks matter in practice, the ranking may not hold on raw data.

Editorial extensions

If this is right

  • A data scientist choosing an AutoML tool for tabular data with a short time budget should expect H2O-Automl to be a strong default for binary classification and regression, and Auto-keras for multiclass classification.
  • Because no tool dominates, AutoML buyers should select based on task type and time limits rather than expecting a universal solution.
  • The 70 percent completion target means some tools fail on a substantial share of datasets within 15 minutes, so deployment should account for non-completion rates such as Darwin failing on about 26 percent of binary datasets and 56 percent of regression datasets.
  • Since most tools improve with more time up to 3 hours, time-limit comparisons are sensitive and short budgets may understate slower-converging tools like Auto-sklearn.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ranking likely does not transfer to raw, uncleaned data: because all datasets were preprocessed into numerical features, the comparison ignores data cleaning, type detection, and feature engineering, where commercial tools claim some of their biggest strengths.
  • The 15-minute cap could systematically favor fast-converging methods; a longer budget might lift Auto-sklearn and TPOT relative to H2O-Automl and Auto-keras.
  • The same evaluation protocol could be extended to natural-language or image inputs, where neural architecture search tools like Auto-keras may have a larger advantage than on tabular data.
  • The tool versions are frozen in 2019, so the ranking is a snapshot; rapid AutoML development means the current ordering may already be stale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper surveys and empirically compares seven AutoML tools (H2O-Automl, Auto-keras, Auto-sklearn, TPOT, Ludwig, Darwin, and Auto-ml) on roughly 300 OpenML datasets across binary classification, multiclass classification, and regression. After a feature/functionality comparison of the common AutoML pipeline, the authors run a large benchmark with a 15-minute time limit (Section IV-A), time-limit experiments at 5 minutes to 3 hours on a small random subset (Section IV-B), and a robustness study with ten runs on one dataset per task (Section IV-C). The central conclusion is that no tool outperforms all others, and that H2O-Automl, Auto-keras, and Auto-sklearn perform better than Ludwig, Darwin, TPOT, and Auto-ml, with H2O-Automl and Auto-keras recommended for production environments depending on the task.

Significance. If scoped correctly, this is a useful, reproducible benchmark: the paper reports tool versions, OpenML dataset IDs, hardware configuration (EC2 p2.xlarge), time limits, and completion rates, and all results are direct measurements with no fitted parameters. The disclosure of the Capital One Ventures investment in H2O.ai is appropriate. The main contribution is a data-driven snapshot of AutoML performance on standard tabular benchmarks under a time budget, which is of practical interest to practitioners. However, the headline ranking as stated in Section V is not properly scoped to the 15-minute budget, and the time-limit dependence is not established on the full evaluation set. With that limitation addressed, the paper would make a solid empirical contribution.

major comments (3)
  1. [Sections IV-A, IV-B, V] The conclusion in Section V that 'H2O-Automl, Auto-keras and Auto-sklearn performed better than Ludwig, Darwin, TPOT and Auto-ml' is not explicitly scoped to the 15-minute time limit. The 15-minute budget was selected via a 70% completion target in Section IV, and the authors state in Section IV-B that Auto-sklearn 'needs almost 2-3 hours to obtain reasonable results' while H2O-Automl, Auto-keras, and Ludwig converge in roughly 15 minutes. The time-limit curves in Figure 7 use only one randomly selected dataset per sample-size range, not the segmented datasets of Section IV-A, so they cannot establish whether the ranking is stable across time limits on the full benchmark. The central claim should be explicitly limited to the 15-minute budget, or the authors should provide evidence that the ranking is stable for longer budgets.
  2. [Section IV-A (completion-target analysis, Figure 4 captions)] Darwin is excluded from the completion-target analysis 'to be fair to the other tools' because it is slow, yet its non-completion rates (26% for binary, 51% for multiclass, 56% for regression) are subsequently used as evidence in the qualitative comparison ('Darwin cannot deliver results for 26% of all datasets' and 'Darwin can only complete work on about half of the datasets'). This is an inconsistent treatment: the completion penalty is counted against Darwin in the quality assessment but removed when choosing the budget. The authors should either include Darwin's non-completion in the budget-setting analysis or report the ranking both with and without non-completion penalties, so that the reader can see how much of the ranking is driven by the time limit.
  3. [Section IV (opening) and Section V] The experiments deliberately use OpenML datasets that are already pre-processed into numerical features, so the measured comparison covers only model selection, hyperparameter optimization, and (for Auto-keras) architecture search, not data preprocessing or feature engineering. Section III, however, identifies preprocessing and feature engineering as major differentiators among the tools. The broad conclusion in Section V that 'there is no perfect tool' and the comparative ranking are therefore limited to a subset of the AutoML pipeline. The conclusion should state this scope explicitly, or the experiments should include raw-data evaluation for at least a subset of datasets.
minor comments (4)
  1. [Figure 4 caption] The caption for Figure 4 lists non-completion percentages for Ludwig, H2O-Automl, TPOT, Darwin, and Auto-sklearn but omits Auto-keras, which is included in the figure; the caption should report the Auto-keras value for consistency with Figures 5 and 6.
  2. [Section IV-C vs. Section V] Section IV-C uses only one randomly selected dataset per task for the robustness study, and Section V states that 'Auto-keras is very stable across all tasks.' This stability claim is based on a single dataset per task and should be qualified as preliminary, or more datasets should be used in the robustness experiment.
  3. [Minor text and notation] There are several minor consistency issues: 'multiclass' is spelled as both 'multiclass' and 'muticlass' in Section IV-A; the phrase '5,400 experiments' in Section IV is correct for 6 tools but later the text mentions '6 tools' while the tool list includes seven entries including Auto-ml (which is excluded from the extensive experiments); the Appendix table contains some dataset IDs that appear in multiple lists, which could be noted.
  4. [Figure 2 and Section III footnotes] Figure 2's functionality table relies on many footnotes (12*), and the text acknowledges that some cells are unclear from documentation. This is acceptable but the authors should consider a more compact legend or inline notes to improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports direct measurements on external OpenML datasets, with no fitted parameters, equations, or load-bearing self-citations.

full rationale

The paper is an empirical benchmark study, not a derivation. Its central claim, that no tool outperforms all others and that H2O-Automl, Auto-keras, and Auto-sklearn perform better overall, is supported by direct accuracy and MSE measurements on OpenML datasets under stated time limits. These measurements are external to the authors' own prior work, and the paper contains no equations, fitted parameters, or definitions that could reduce one quantity to another. The 15-minute budget is a protocol choice, not a constructed equivalence: the paper explicitly reports the budget rationale, completion rates, and additional time-limit experiments in Section IV-B. The Capital One Ventures investment in H2O.ai is disclosed, but disclosure of a potential conflict does not make the measurement circular; the ranking is not derived from the investment. Limitations such as the use of preprocessed numerical features and the time-limit dependence of the ranking are real scoping concerns but are matters of experimental validity, not circularity. No step in the paper's argument reduces by construction to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no fitted parameters, no new mathematical axioms, and no invented entities. Its central claims rest on benchmark dataset selection and the chosen evaluation protocol, which are standard domain assumptions rather than ad hoc constructs.

assumptions (2)
  • domain assumption The OpenML datasets, already converted to numerical features, are representative of real-world AutoML use cases and provide a fair common input for all tools.
    Invoked in Section IV to justify using pre-processed OpenML data so that the same features are fed to every tool, which narrows the comparison to model selection and hyperparameter tuning.
  • domain assumption The 15-minute time limit, derived from a 70% completion target, is a fair budget that does not systematically distort tool rankings.
    Used in Section IV to set the runtime for 5,400 experiments; the paper acknowledges Darwin is slow and excludes it from the target analysis, and later runs a time-limit sensitivity experiment, indicating the assumption is important and partially relaxed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools." pith.science (2026). https://pith.science/paper/72DNKCTM

@misc{pith2026190805557,
  author       = {Pith},
  title        = {Pith review of: Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72DNKCTM}},
  note         = {Machine review of arXiv:1908.05557}
}
read the original abstract

There has been considerable growth and interest in industrial applications of machine learning (ML) in recent years. ML engineers, as a consequence, are in high demand across the industry, yet improving the efficiency of ML engineers remains a fundamental challenge. Automated machine learning (AutoML) has emerged as a way to save time and effort on repetitive tasks in ML pipelines, such as data pre-processing, feature engineering, model selection, hyperparameter optimization, and prediction result analysis. In this paper, we investigate the current state of AutoML tools aiming to automate these tasks. We conduct various evaluations of the tools on many datasets, in different data segments, to examine their performance, and compare their advantages and disadvantages on different test cases.

Figures

Figures reproduced from arXiv: 1908.05557 by the authors.

Figure 1
Figure 1. The common AutoML pipeline. Most AutoML tools follow a common three stage pipeline illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison table of functionality for AutoML tools. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data segments used for evaluation. Each cell is referred to as a [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Evaluation of AutoML tools on binary classification task across ten data segments (depicted in Figure 3). Each diagram refers to a data segment. All [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Evaluation of AutoML tools on multiclass classification task across ten data segments (depicted in Figure 3). Each diagram refers to a data segment. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of AutoML tools on regression task across ten data segments (depicted in Figure 3). Each diagram refers to a data segment. All experiments [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Evaluation of AutoML on multiple time limits. The left (middle) subgraphs show the accuracy of tools for binary (multiclass) classification. The right [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Evaluation of AutoML tools on robustness. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interpretable by Design: MH-AutoML for Transparent and Efficient Android Malware Detection without Compromising Performance

    cs.CR 2025-06 conditional novelty 5.0 of 10

    MH-AutoML is a domain-specific AutoML framework for Android malware detection that combines automated modeling with built-in interpretability, and its evaluation shows competitive recall and higher transparency scores...

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Auto-weka 2.0: Automatic model selection and hyperparameter optimization in weka,

    L. Kotthoff, C. Thornton, H. H. Hoos, F. Hutter, and K. Leyton- Brown, “Auto-weka 2.0: Automatic model selection and hyperparameter optimization in weka,” The Journal of Machine Learning Research , vol. 18, no. 1, pp. 826–830, Jan. 2017

    work page 2017

  2. [2]

    The weka data mining software: An update,

    M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: An update,” ACM SIGKDD Explorations Newsletter, vol. 11, no. 1, pp. 10–18, 2009

    work page 2009

  3. [3]

    Efficient and robust automated machine learning,

    M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter, “Efficient and robust automated machine learning,” in Advances in Neural Information Processing Systems 28 , 2015, pp. 2962– 2970

    work page 2015

  4. [4]

    Evaluation of a tree-based pipeline optimization tool for automating data science,

    R. S. Olson, N. Bartley, R. J. Urbanowicz, and J. H. Moore, “Evaluation of a tree-based pipeline optimization tool for automating data science,” in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2016 . New York, NY , USA: ACM, 2016, pp. 485–492

    work page 2016

  5. [5]

    Auto-ml: Automated machine learning for production and analytics,

    “Auto-ml: Automated machine learning for production and analytics,” https://github.com/ClimbsRocks/$auto ml$, accessed: 2019-04-10

    work page 2019

  6. [6]

    Auto-keras: An efficient neural architecture search system,

    H. Jin, Q. Song, and X. Hu, “Auto-keras: An efficient neural architecture search system,” in arXiv, 2018

    work page 2018

  7. [7]

    “Mljar,” https://github.com/mljar/mljar-api-python, accessed: 2019-04- 10

    work page 2019

  8. [8]

    Datarobot usage examples,

    “Datarobot usage examples,” https://github.com/datarobot/ datarobot-sagemaker-examples, accessed: 2019-04-10

    work page 2019

Show all 22 references
  1. [9]

    Datarobot documentation,

    “Datarobot documentation,” https://www.datarobot.com/about-us/, ac- cessed: 2019-04-10

    2019

  2. [10]

    Datarobot python client,

    “Datarobot python client,” https://datarobot-public-api-client. readthedocs-hosted.com/en/v2.11.0/setup/getting{}started.html, accessed: 2019-04-10

    2019

  3. [11]

    H2o.ai automl github,

    “H2o.ai automl github,” https://github.com/h2oai/h2o-3, accessed: 2019- 04-10

    2019

  4. [12]

    H2o.ai automl documentation,

    “H2o.ai automl documentation,” http://docs.h2o.ai/h2o/latest-stable/ h2o-docs/automl.html, accessed: 2019-04-10

    2019

  5. [13]

    H2o-driverlessai,

    “H2o-driverlessai,” http://docs.h2o.ai/driverless-ai/latest-stable/docs/ userguide/index.html, accessed: 2019-04-10

    2019

  6. [14]

    Darwin-sparkcognition,

    “Darwin-sparkcognition,” https://github.com/sparkcognition/darwin-sdk, accessed: 2019-04-10

    2019

  7. [15]

    Google cloud automl,

    “Google cloud automl,” https://cloud.google.com/automl/, accessed: 2019-04-10

    2019

  8. [16]

    Automated machine learning with azureml,

    “Automated machine learning with azureml,” https://github.com/ Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/ automated-machine-learning, accessed: 2019-04-10

    2019

  9. [17]

    Transmogrifai,

    “Transmogrifai,” https://github.com/salesforce/TransmogrifAI, accessed: 2019-04-10

    2019

  10. [18]

    “Ludwig,” https://github.com/uber/ludwig, accessed: 2019-04-10

    2019

  11. [19]

    Neural architecture search with reinforcement learning,

    B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in arXiv, Nov. 2016

    2016

  12. [20]

    Efficient neural architecture search via parameters sharing,

    H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in Proceedings of the 35th International Conference on Machine Learning , vol. 80, Stock- holmsmssan, Stockholm Sweden, 10–15 Jul 2018, pp. 4095–4104

    2018

  13. [21]

    Sequential model-based optimization for general algorithm configuration,

    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in Proceedings of the 5th International Conference on Learning and Intelligent Optimization , ser. LION’05, 2011, pp. 507–523

    2011

  14. [22]

    Openml: Networked science in machine learning,

    J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo, “Openml: Networked science in machine learning,” ACM SIGKDD Explorations Newsletter, vol. 15, pp. 49–60, Jun. 2014. APPENDIX TABLE I DATASET DESCRIPTIONS . Binary classification Multiclass classfication Regression Id Name I...

    2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.