REVIEW 5 major objections 7 minor 27 references
AutoML-Med: A Framework for Automated Machine Learning in Medical Tabular Data
T0 review · 5 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read AutoML-Med, an automated machine learning tool for medical tabular data, reports higher balanced accuracy and higher sensitivity than competing AutoML tools on two imbalanced clinical datasets.
desk verdict AutoML-Med is a genuinely useful engineering idea—LHS-PRCC for pipeline refinement—but its headline numbers rest on an unstated test-separation protocol and a non-equivalent baseline, so it deserves review rather than acceptance as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the LHS-PRCC pair: Latin Hypercube Sampling adapted to a discrete space of preprocessing choices (each of five steps is divided into intervals, a value is sampled per interval, and its integer part selects a method), followed by Partial Rank Correlation Coefficient computed on rank-transformed variables to isolate which preprocessing steps independently drive model performance. Because PRCC removes the influence of other variables by regressing each input on the rest and correlating residuals, it lets the tool focus a final grid search on the most influential preprocessing steps rather than enumerating the full Cartesian product of methods. This converts pipeline search from an exponential enumeration into a stratified sample plus a targeted refinement.
What would settle it
Run the tool on the same public diabetes dataset with the test split held in a separate file and the code modified so that PRCC ranking and grid-search fine-tuning can see only training folds, then compare balanced accuracy and sensitivity on the untouched test set with the reported values; if they fall toward the comparison baselines' levels, the claimed advantage comes from test information rather than from the preprocessing search.
Extended reading notes
Core claim
The core discovery is that in medical AutoML the performance bottleneck often lies in preprocessing configuration rather than model choice, and that a two-stage search can exploit this. AutoML-Med first uses Latin Hypercube Sampling to draw a stratified sample of complete five-step preprocessing pipelines (imputation, class balancing, feature engineering, scaling, feature selection), trains candidate models on each, and records evaluation metrics. It then locks in the best-performing model and applies a Partial Rank Correlation Coefficient (PRCC) sensitivity analysis to identify which preprocessing steps most influence the target metric; a grid search over variations of those steps, with the other steps held fixed, produces the final pipeline. On the multiple sclerosis task this yields balanced accuracy 0.8896 and sensitivity 0.8539, against 0.8466-0.8492 and 0.7158-0.7492 for the comparison AutoML tools; on the type 2 diabetes task it yields balanced accuracy 0.7436 and sensitivity 0.7968, against 0.6398-0.665 and 0.3781-0.4634 for the published models, at the cost of lower specificity in both cases.
Load-bearing premise
The load-bearing premise is that the final one-third test split is never used during pipeline selection, PRCC ranking, or grid-search fine-tuning; the paper does not explicitly state this, and any leakage from the test set into the search would make the reported balanced accuracy and sensitivity optimistically biased.
Editorial extensions
If this is right
- On the two tested datasets, the tool's higher sensitivity means more true positives among at-risk patients, at the price of more false positives; the paper argues this is the clinically preferable trade for risk screening.
- If preprocessing configuration is the main lever, then medical AutoML users can expect meaningful gains from automating imputation, balancing, and feature-selection choices even when standard predictive models are used.
- The approach transfers across medical domains: the same pipeline design produced higher balanced accuracy on both a proprietary neurological dataset and a public behavioral-risk diabetes dataset, suggesting the preprocessing search generalizes.
- Because the tool requires no advanced programming skills, clinicians could run it directly, lowering the barrier to adopting machine learning in hospital workflows.
Reading between the lines
- One implicit consequence is that the LHS-PRCC machinery is not specific to medicine; it could be applied to any tabular AutoML setting with a small, structured pipeline space, such as credit scoring or industrial fault detection, and might help there too.
- The reported sensitivity gains on the diabetes task come with specificity around 0.69, meaning roughly 31% of healthy patients would be flagged; a deployment study should weigh the cost of follow-up testing against the value of catching additional cases.
- A natural testable extension is to run AutoML-Med with the PRCC fine-tuning stage disabled; if most of the sensitivity gain disappears, the claimed advantage comes from the sensitivity-analysis mechanism rather than from the initial stratified sampling.
- The paper averages metrics over ten runs with different seeds; a sharper test would report per-run variability of the selected pipelines, to show whether the same preprocessing configurations are consistently chosen or whether different configurations achieve similar scores.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents AutoML-Med, an AutoML framework for medical tabular data that searches over five ordered preprocessing stages (imputation, class balancing, feature engineering, scaling, and feature selection) together with a predictive model. The search uses Latin Hypercube Sampling to generate pipeline candidates, trains models on each preprocessed dataset, and then uses the Partial Rank Correlation Coefficient to identify the most influential preprocessing steps for a final grid-search refinement. The experimental sections report results on a proprietary Multiple Sclerosis cohort (Table 1) and the public BRFSS 2014 diabetes dataset (Table 2), claiming higher balanced accuracy and sensitivity than Auto-sklearn, GAMA, AutoBalance, and the published models of Xie et al.
Significance. The problem addressed is real, and the general architecture is plausible: the focus on preprocessing steps, class imbalance, and missing values is appropriate for clinical tabular data. The choice of balanced accuracy and sensitivity as primary metrics is well justified for triage applications. The evaluation on a public dataset is a commendable step toward reproducibility. However, the current manuscript does not yet establish the central performance claim. The description of pipeline selection and fine-tuning never states that the final test split is excluded from selection; the diabetes comparison is against numbers extracted from an earlier paper under different data handling; no significance tests are provided; and the proposed PRCC mechanism is applied to categorical preprocessing choices without methodological justification. The withheld code further prevents independent verification. If the design is corrected and the comparisons tightened, the tool could be a useful contribution, but the evaluation as written is not conclusive.
major comments (5)
- [§3.2, §3.3, §4.1, §4.2] The most load-bearing ambiguity is which data split is used to compute the target metrics that drive pipeline selection and PRCC fine-tuning. Section 3.2 states only that 'a random cross-validation search is performed for the best hyperparameters for the predictive model being trained'; it does not say that the preprocessed pipelines themselves are evaluated by cross-validation or on the training partition. Section 3.3 then selects 'the pipeline that obtains the highest value on the target metric(s)' and presents it 'along with a complete report of its performance.' If those target metrics are computed on the same one-third split reported in Tables 1 and 2, the reported balanced accuracy and sensitivity are optimistically biased by selection-on-test, and the comparisons to Auto-sklearn, GAMA, AutoBalance, and Xie et al. would be invalid. Averaging over ten seeds does not remove selection-on-test bias. The manuscript must state explicitly that the test split is used only once at the end, and if that was not the case, the experiments must be re-run with a properly held-out test set.
- [§4.2, Table 2] The Type 2 Diabetes comparison is not a controlled benchmark. The authors state that they applied the filtering procedure of Xie et al. [22] but did not remove patients with missing data, and they compare to the published numbers of [22] rather than re-running the same models on the same train/test splits under the same preprocessing. Differences in missing-data handling, feature encoding, class imbalance treatment, and evaluation protocol can fully explain the reported gap. At minimum, the authors should re-run the models of [22] in identical conditions and report the metrics on the same test partitions; otherwise the claim of 'significantly higher sensitivity and balanced accuracy' is not supported.
- [§3.3] PRCC is applied to preprocessing steps that are categorical choices. The independent variables are the discretized integer indices returned by LHS, whose ordering is arbitrary. A rank-based partial correlation on these indices is not a well-defined sensitivity measure for unordered categorical choices; the result depends on the arbitrary ordering of methods within each step. The manuscript must clarify how the categorical steps are encoded for PRCC (e.g., one-hot dummy variables) and why PRCC is appropriate, or replace it with a categorical sensitivity analysis. This issue is central because PRCC is the mechanism that selects which steps are refined in the final grid search.
- [Tables 1 and 2] No statistical significance tests are reported, and for several metrics the claimed advantage is small. In Table 1, AutoML-Med's F1-macro (0.8745) and MCC (0.7558) are within one standard deviation of the competing tools (e.g., GAMA MCC 0.7683), and the AUC is actually lower than all three baselines. The paper should therefore restrict its claims to the metrics where the difference is meaningful, or provide paired significance tests across the ten seeds.
- [§4.1] The comparison with Auto-sklearn, GAMA, and AutoBalance does not specify the search budgets, time limits, or hyperparameter configurations given to the competing tools. AutoML results depend strongly on the number of pipeline evaluations and the allowed runtime. Without this information, the balanced-accuracy difference could be an artifact of unequal computational effort. Please report the compute budgets and ideally run the baselines under matched budgets.
minor comments (7)
- [Abstract] The abstract contains a typo: 'a heterogeneous feature types' should read 'heterogeneous feature types'.
- [Table 1] Table 1 mixes comma and period decimal separators (e.g., '0,8735' vs '0.9187'); use consistent formatting.
- [Title page and Conclusion] The manuscript claims 'compliance with double-blind review' but lists author names and affiliations in the header; this statement should be removed or the paper should be anonymized.
- [§3.2] Section 3.2 mentions a set of evaluation metrics but the selection criterion is later referred to as 'the target metric(s)'; clarify how multiple metrics are aggregated when more than one target is selected.
- [§3.1] The phrase 'Latin Hypercube Sampling - Partial Rank Correlation Coefficient (LHS-PRCC)' suggests a joint method, but LHS and PRCC are used in distinct phases; the naming could be clarified.
- [References] Reference [3] is incomplete ('S. Garcia and altri') and should be replaced with the full author list.
- [Conclusion] The conclusion states that the code will be shared only after review; this is acceptable for now, but it means the reproducibility claim cannot currently be verified.
Circularity Check
No demonstrated circularity; the search optimizes the reported target metrics, but no equation or statement shows test data are used for selection.
full rationale
AutoML-Med's derivation chain is a standard AutoML search: LHS samples preprocessing combinations, models are trained, target metrics (balanced accuracy, F1, F-beta, MCC) are computed, PRCC ranks preprocessing-step influence, grid search fine-tunes influential steps, and the pipeline with the highest target-metric value is selected. Reportedly, these target metrics are also the headline numbers in Tables 1 and 2. This is circular only if the same data are used both for pipeline selection and for the final reported evaluation; the manuscript does not explicitly define a validation partition for the selection steps, and Section 4.1 describes only a 2/3 training / 1/3 test split. That under-specification is a real reproducibility risk: if the 1/3 split leaked into selection, the balanced-accuracy and sensitivity gains would be selection-on-test artifacts. However, the paper contains no equation or passage showing that the test split is used in selection, no load-bearing self-citations, and no uniqueness claims; the comparisons are against external AutoML tools and published results. The withheld code (Section 5) prevents independent inspection but is a reproducibility limitation rather than evidence of circularity. The claimed improvements therefore are not shown to be equivalent to the search objective by construction. The finding is no significant circularity, with a request to document the validation protocol before code release.
Assumptions & free parameters
free parameters (4)
- m (number of top PRCC preprocessing steps refined by grid search) =
2 (default)
- Number of LHS pipeline samples =
not reported
- Grid search variations per refined step =
not reported
- Set of preprocessing methods per step =
not enumerated for experiments
assumptions (5)
- domain assumption LHS-PRCC can be applied to discrete pipeline choices by interval-indexing continuous samples.
- domain assumption PRCC on rank-transformed pipeline indicators captures which preprocessing steps most influence model performance.
- domain assumption The BRFSS filtering procedure from Xie et al. [22] can be reused, and the published metrics are a valid baseline even though AutoML-Med imputes rather than drops missing rows.
- domain assumption The MS dataset labels and clinical features are reliable and the EDSS-derived risk target is appropriate.
- standard math Standard scikit-learn and imbalanced-learn implementations of preprocessing and models are correct.
Cite this review
Pith. "Pith review of AutoML-Med: A Framework for Automated Machine Learning in Medical Tabular Data." pith.science (2026). https://pith.science/paper/MSOJ5VOI
@misc{pith2026250802625,
author = {Pith},
title = {Pith review of: AutoML-Med: A Framework for Automated Machine Learning in Medical Tabular Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/MSOJ5VOI}},
note = {Machine review of arXiv:2508.02625}
}
read the original abstract
Medical datasets are typically affected by issues such as missing values, class imbalance, a heterogeneous feature types, and a high number of features versus a relatively small number of samples, preventing machine learning models from obtaining proper results in classification and regression tasks. This paper introduces AutoML-Med, an Automated Machine Learning tool specifically designed to address these challenges, minimizing user intervention and identifying the optimal combination of preprocessing techniques and predictive models. AutoML-Med's architecture incorporates Latin Hypercube Sampling (LHS) for exploring preprocessing methods, trains models using selected metrics, and utilizes Partial Rank Correlation Coefficient (PRCC) for fine-tuned optimization of the most influential preprocessing steps. Experimental results demonstrate AutoML-Med's effectiveness in two different clinical settings, achieving higher balanced accuracy and sensitivity, which are crucial for identifying at-risk patients, compared to other state-of-the-art tools. AutoML-Med's ability to improve prediction results, especially in medical datasets with sparse data and class imbalance, highlights its potential to streamline Machine Learning applications in healthcare.
Figures
Reference graph
Works this paper leans on
-
[22]
Building risk prediction models for type 2 diabetes using machine learning techniques,
Z. Xie, O. Nikolayeva, J. Luo, and D. Li, “Building risk prediction models for type 2 diabetes using machine learning techniques,” Preventing chronic disease, vol. 16, p. E130, 2019. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/31538566/
-
[1]
Why do tree- based models still outperform deep learning on tabular data?
L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree- based models still outperform deep learning on tabular data?” arXiv preprint arXiv:2207.08815, July 2022. [Online]. Available: https://arxiv.org/abs/2207.08815
arXiv 2022
-
[2]
Why do tree-based models still outperform deep learning on tabular data?
——, “Why do tree-based models still outperform deep learning on tabular data?” inAdvances in Neural Information Processing Systems (NeurIPS), 2022
work page 2022
-
[3]
Deep neural networks and tabular data: A survey,
S. Garcia and altri, “Deep neural networks and tabular data: A survey,”arXiv preprint arXiv:2110.01889, October 2021. [Online]. Available: https://arxiv.org/abs/2110.01889
arXiv 2021
-
[4]
M. Kuhn and K. Johnson,Feature Engineering and Selection: A Practical Approach for Predictive Models. CRC Press, 2019
work page 2019
-
[5]
Data preprocessing in data mining,
S. Garc ´ıa, J. Luengo, and F. Herrera, “Data preprocessing in data mining,”Springer Briefs in Computer Science, 2015
work page 2015
-
[6]
Handling class imbalance in medical datasets,
M. A. Rahman and D. N. Davis, “Handling class imbalance in medical datasets,”Healthcare Informatics Research, vol. 19, no. 3, pp. 166–175, 2013
work page 2013
-
[7]
A survey on feature selection methods,
G. Chandrashekar and F. Sahin, “A survey on feature selection methods,”Computers & Electrical Engineering, vol. 40, no. 1, pp. 16–28, 2014
work page 2014
Show all 27 references
-
[8]
Feurer, A
M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter,Auto-sklearn: Efficient and Robust Automated Machine Learning. Springer International Publishing, 2019, pp. 113–134. [Online]. Available: https://doi.org/10.1007/978-3-030-05318-5 6
2019 doi
-
[9]
Scikit-learn: Machine learning in python,
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. VanderPlas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in python,” CoRR, vol. abs/1201.049...
2012 arXiv
-
[10]
H2o automl: Scalable automatic machine learning,
E. LeDell and S. Poirier, “H2o automl: Scalable automatic machine learning,” 2020. [Online]. Available: https://api.semanticscholar.org/ CorpusID:221338558
2020
-
[11]
GAMA: A general automated machine learning assistant,
P. Gijsbers and J. Vanschoren, “GAMA: A general automated machine learning assistant,” inMachine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track - European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part V...
2020 doi
-
[12]
Evaluation of a tree-based pipeline optimization tool for automating data science,
R. S. Olson, N. Bartley, R. J. Urbanowicz, and J. H. Moore, “Evaluation of a tree-based pipeline optimization tool for automating data science,” inProceedings of the 2016 on Genetic and Evolutionary Computation Conference, Denver, CO, USA, July 20 - 24, 2016, T. Friedrich, F. ...
2016
-
[13]
TPOT2: A new graph-based implementation of the tree-based pipeline optimization tool for automated machine learning,
P. H. Ribeiro, A. K. Saini, J. Moran, N. Matsumoto, H. Choi, M. E. Hernandez, and J. H. Moore, “TPOT2: A new graph-based implementation of the tree-based pipeline optimization tool for automated machine learning,” inGenetic Programming Theory and Practice XX [GPTP 2023], S. Wi...
2023 doi
-
[14]
Benchmarking automatic machine learning frameworks,
A. Balaji and A. Allen, “Benchmarking automatic machine learning frameworks,”CoRR, vol. abs/1808.06492, 2018. [Online]. Available: http://arxiv.org/abs/1808.06492
2018 arXiv
- [15]
-
[16]
Benchmarking automl frameworks for disease prediction using medical claims,
R. A. A. Romero, M. N. Y . Deypalan, S. Mehrotra, J. T. Jungao, N. E. Sheils, E. Manduchi, and J. H. Moore, “Benchmarking automl frameworks for disease prediction using medical claims,” BioData Min., vol. 15, no. 1, 2022. [Online]. Available: https: //doi.org/10.1186/s13040-02...
2022 doi
-
[17]
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare,
J. Waring, C. Lindvall, and R. Umeton, “Automated machine learning: Review of the state-of-the-art and opportunities for healthcare,”Artif. Intell. Medicine, vol. 104, p. 101822, 2020. [Online]. Available: https://doi.org/10.1016/j.artmed.2020.101822
2020
-
[18]
A method- ology for performing global uncertainty and sensitivity analysis in systems biology,
S. Marino, I. B. Hogue, C. J. Ray, and D. E. Kirschner, “A method- ology for performing global uncertainty and sensitivity analysis in systems biology,”Journal of Theoretical Biology, vol. 254, no. 1, pp. 178–196, 2008
2008
-
[19]
Comparison of 3 methods for selecting values of input variables in the analysis of output from a computer code,
M. D. McKay, R. J. Beckman, and W. J. Conover, “Comparison of 3 methods for selecting values of input variables in the analysis of output from a computer code,”Technometrics, vol. 21, no. 2, pp. 239–245, 1979
1979
-
[20]
Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques,
A. Saltelli and J. Marivoet, “Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques,” Reliability Engineering & System Safety, vol. 28, no. 2, pp. 229–253, 1990
1990
-
[21]
Examining the influence of nondimensionalization on partial rank correlation coefficient results when modeling the epithelial mesenchymal transition,
K. I. Gasior, “Examining the influence of nondimensionalization on partial rank correlation coefficient results when modeling the epithelial mesenchymal transition,” vol. 87, no. 1, p. 15. [Online]. Available: https://doi.org/10.1007/s11538-024-01393-y
-
[23]
The immunology of multiple sclerosis,
K. E. Attfield, L. T. Jensen, M. Kaufmann, M. A. Friese, and L. Fugger, “The immunology of multiple sclerosis,”Nature Reviews Immunology, vol. 22, no. 12, pp. 734–750, 2022
2022
-
[24]
Multiple scle- rosis genomic map implicates peripheral immune cells and microglia in susceptibility,
International Multiple Sclerosis Genetics Consortium, “Multiple scle- rosis genomic map implicates peripheral immune cells and microglia in susceptibility,”Science, vol. 365, no. 6460, p. eaav7188, Sep. 2019
2019
-
[25]
Lifestyle and environmental factors in multiple sclerosis,
L. Alfredsson and T. Olsson, “Lifestyle and environmental factors in multiple sclerosis,”Cold Spring Harb. Perspect. Med., vol. 9, no. 4, p. a028944, Apr. 2019
2019
-
[26]
Rising prevalence of multiple sclerosis worldwide: Insights from the atlas of ms, third edition,
C. Walton, R. King, L. Rechtman, and et al., “Rising prevalence of multiple sclerosis worldwide: Insights from the atlas of ms, third edition,”Multiple Sclerosis Journal, vol. 26, no. 14, pp. 1816–1821, 2020
2020
-
[27]
Economic costs of diabetes in the U.S. in 2007,
American Diabetes Association, “Economic costs of diabetes in the U.S. in 2007,”Diabetes Care, vol. 31, no. 3, pp. 596–615, Mar. 2008
2007
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.