Pith. sign in

REVIEW 5 major objections 5 minor 8 references

SMAC clearly outperforms other tuning methods for tree-boosting across 59 tabular datasets

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SMAC beats TPE, GP-based Bayesian optimization, random grid, Hyperband, and deterministic grid for tuning tree-boosting on 59 OpenML datasets, with no small hyperparameter subset safely left at defaults.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A well-run 59-dataset benchmark that plausibly shows SMAC best under a trial budget, with caveats worth refereeing before the practical headline is taken at face value. the 5 major comments →

arxiv 2602.05786 v3 pith:I7Y2IZJ4 submitted 2026-02-05 cs.LG stat.APstat.ML

Selecting Hyperparameters for Tree-Boosting

Floris Jan Koster , Fabio Sigrist This is my paper
classification cs.LG stat.APstat.ML
keywords hyperparameter optimizationtree-boostinggradient boostingtabular dataSMACTPEearly stoppingmodel-based search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to determine which hyperparameter-tuning method a practitioner should use for tree-boosting—a widely used machine learning technique for tabular data—when the tuning budget is fixed. It compares six popular methods (random grid search, TPE, GP-based Bayesian optimization, Hyperband, SMAC, and deterministic full grid search) on 36 regression and 23 classification data sets. The central finding is that SMAC, a sequential model-based configuration method, produces clearly more accurate models on average than all the others, with TPE second. The paper also finds that accurate tuning typically needs more than 100 trials, that default hyperparameters often give much worse results, that all eight considered hyperparameters can matter, and that for regression it is better to choose the number of boosting iterations via early stopping than to include it in the search space.

Core claim

The paper's central claim is that, under a fixed trial budget of 135, SMAC produces substantially more accurate tree-boosting models than TPE, GP-based Bayesian optimization, random grid search, deterministic full grid search, and Hyperband, on average across 59 data sets. The advantage holds for all four metrics: R², RMSE, accuracy, and log loss. The paper further claims that accurate tuning generally requires more than 100 trials, that default hyperparameters often give substantially worse results, that all eight hyperparameters considered can have a material effect on accuracy, and that for regression tasks early stopping beats including the number of iterations in the search space.

What carries the argument

The central object is the SMAC method (sequential model-based algorithm configuration), a Bayesian optimization approach that iteratively fits a model of how hyperparameters affect validation performance and uses expected improvement to choose the next configuration to evaluate. The comparison uses five-fold cross-validation with inner validation splits, ADTM (average distance to the minimum) normalization to aggregate scores across data sets, and a fixed trial budget of 135 for all methods except Hyperband, which uses a multi-fidelity successive-halving schedule.

Load-bearing premise

The comparison counts a trial as one hyperparameter configuration evaluation and gives every method the same number of trials, but it does not measure wall-clock time or compute cost, so the practical claim that SMAC is better for real users depends on SMAC's per-trial overhead not being large enough to erase its accuracy advantage.

What would settle it

Measure total wall-clock time for each method on several of the same data sets under a fixed time budget rather than a fixed trial count, and check whether SMAC still beats random grid search and TPE; or rerun the comparison with only 50 trials and see whether the ranking holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners who can afford a few hundred tuning trials should prefer SMAC or a closely related model-based method over random or grid search, and should not rely on default hyperparameters.
  • A tuning budget below roughly 100 trials is likely to give misleadingly poor models and can distort comparisons of both models and tuning methods.
  • Tuning only a small subset of hyperparameters is risky: all considered parameters (learning rate, tree size, regularization, binning, subsampling) can materially affect accuracy.
  • For regression tasks, fixing the number of boosting rounds via early stopping is generally better than searching over it; for classification tasks, including it in the search space can be competitive.
  • No tuning method is universally best across individual data sets, so the observed average ranking should be read as guidance rather than a guarantee.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The trial-budget framing likely understates SMAC's real cost, since model-based methods carry per-trial overhead; the ranking could change under strict wall-clock time limits, which the paper itself leaves untested.
  • The finding that all hyperparameters matter suggests that warm-starting or transferring knowledge from related tuning tasks could be a promising way to reduce the required number of trials, though this is not investigated here.
  • The early-stopping versus search-space finding may extend to objectives beyond accuracy, such as calibration or fairness, but that extension is speculative.
  • The strong average performance of SMAC may partly depend on the specific search-space parameterization; alternative scales or ranges for tree-size parameters could shift the ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper benchmarks six hyperparameter optimization methods (deterministic full grid, random grid, GP-BO, TPE, Hyperband, SMAC) and LightGBM defaults for tree-boosting on 59 OpenML datasets (36 regression, 23 classification). The protocol uses 5-fold cross-validation, 20 random seeds per search, four evaluation metrics, and a fixed budget of 135 trials for all methods except Hyperband (which uses 9 successive-halving rungs with R=2150 and eta=2.8). Scores are aggregated via ADTM normalization, relative differences, and ranks. The central claim is that SMAC clearly outperforms all other methods on average, with TPE second, and the paper also reports that more than roughly 100 trials are needed, that defaults perform poorly, that no small hyperparameter subset dominates, and that early stopping for the number of iterations is preferable for regression. Code is provided.

Significance. If the central claim holds, this is a practically useful benchmark: it is broad (59 datasets), carefully repeated over seeds and folds, compares several mainstream HPO tools under a common evaluation protocol, and includes per-dataset curves as well as aggregated comparisons. The code release and the explicit treatment of early stopping versus explicit iteration search are strengths. The finding that SMAC dominates under a fixed trial budget would be a valuable recommendation for practitioners. However, the validity of that recommendation depends on the fairness of the method comparison, and the current manuscript has several uncontrolled differences across methods—different search spaces, an unequal Hyperband budget, and no wall-clock measurement—that must be resolved before the headline claim can be accepted.

major comments (5)
  1. [§2.4, Tables 1–3] The methods are compared using different search spaces. SMAC, TPE, and GP-BO use the continuous/large discrete space of Table 1, while random grid search is restricted to the coarse grid of Table 2 and deterministic full grid to the even smaller Table 3 (135 points). This confounds the HPO algorithm with the search-space granularity: random grid may underperform simply because good hyperparameters are not representable in its coarse grid. To support the claim that SMAC 'clearly outperforms' random grid, the random grid should be sampled from the same Table 1 space, or the restriction must be justified as an intentional design choice with a separate ablation.
  2. [§2.4–2.5, Table 5] The Hyperband comparison is not on equal terms. Hyperband is credited with only 9 recorded rungs rather than 135 trials, the actual number of hyperparameter configurations it evaluates is not reported, and the statement that its runtime is 'approximately equal' is explicitly not tabulated. The test scores of Hyperband are linearly interpolated to common x-axis positions, which can be misleading. As a result, the final-score comparison in Table 5 between Hyperband and the other methods does not represent the same tuning budget. Please report the number of configurations evaluated and/or wall-clock time, or restrict the headline ranking to methods with a common budget definition.
  3. [§2.4, Table 1] There is an internal inconsistency about which hyperparameters are in the search space. Table 1 includes 'Num iterations' and 'Max depth', but the text says that 'unless stated otherwise' the number of iterations is determined by early stopping and is 'not explicitly contained in the search space,' and that the default experiment only includes the number of leaves without a depth limit. The reader cannot tell which search space was actually used in the main experiments. This needs to be stated unambiguously for each experiment, since it directly affects the interpretation of Figure 11 and the main results.
  4. [§5 / Abstract] The practitioner-facing recommendation to use SMAC is based on a fixed number of trials, not wall-clock time or compute cost. Section 5 concedes that wall-clock time and compute limits were not assessed, yet the abstract motivates the study by 'users who cannot afford extensive manual trial-and-error tuning,' which is naturally a time-constrained setting. SMAC's per-trial overhead (surrogate fitting, acquisition optimization) can be substantially larger than that of random grid or TPE. Either add wall-clock measurements or soften the abstract/conclusion to make clear that the recommendation applies only when the budget is measured in trials, not in time.
  5. [§4, Figure 2] The claim that 'there is no small set of hyperparameters that is more important than others' is based on SHAP values from a mixed-effects tree-boosting model fitted to the same trials used in the comparison. SHAP values are descriptive and depend on the search distributions and ranges used; they do not establish causal importance or generalizability across different design spaces. The statement in the abstract and conclusions is stronger than this evidence supports. Please either soften the claim or provide additional support, e.g., by evaluating importance on held-out configurations or with a more principled sensitivity analysis.
minor comments (5)
  1. [§2.2] Please list the OpenML dataset IDs or versions used, not only the names, to make the benchmark fully reproducible. The code repository link is welcome but should be accompanied by a dataset manifest.
  2. [§3, Figures 1 and 4] Figure 1's confidence intervals represent uncertainty across datasets, while Figure 4 represents seed randomness. The text could be clearer that the visual separation in Figure 1 is not a statistical test of pairwise method differences; a paired test or effect-size summary would strengthen the 'clearly outperforms' claim.
  3. [§2.5] The ADTM normalization uses Q0.9 and Q0.1 instead of min/max. The authors state that the results are not sensitive to this choice, but the supporting analysis is 'not shown.' Please include it in an appendix or reference a stable supplementary result.
  4. [Throughout] The package name is written inconsistently as 'GPBoost' and 'GPBoost'; similarly, 'LightGBM' appears as 'LightGBM' and 'LightGBM'. Please standardize the spelling.
  5. [§2.1] For GP-BO, the 'gp hedge' acquisition function and the default random initial points are used, but it is not stated how many initial random points are allocated before acquisition. A clear statement of the initial-design size for each Bayesian method would help interpret the trial budget.

Circularity Check

0 steps flagged

No significant circularity: the empirical ranking is self-contained and not an artifact of fitted parameters or self-citation.

full rationale

The paper's central claim is an empirical comparison of hyperparameter optimization methods on 59 external OpenML data sets (Section 2.2, Table 5). No equation or claimed prediction reduces to a fitted parameter: the SMAC-vs-others ranking is obtained by running off-the-shelf optimizers under a fixed trial budget and evaluating on held-out test splits (Sections 2.3–2.5). The only self-citations are to the authors' GPBoost software and mixed-effects tree-boosting method (Sections 2.1 and 4), and these serve as implementation or modeling tools, not as evidence for the headline result. Section 4's meta-model is explicitly descriptive: it fits a model to the already-collected trials and reports SHAP values, but it does not feed back into the comparison or generate a prediction claimed to be derived from first principles; thus it is not circular. The paper itself flags a genuine scope limitation in Section 5 — wall-clock time and strict compute limits were not assessed — but that is a validity/interpretation concern, not circularity. Overall, the derivation chain is straightforward empirical benchmarking with no self-definitional steps, no fitted inputs renamed as predictions, and no load-bearing self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The study is empirical; no quantities are derived from first principles. The central ranking rests on dataset representativeness and the equation of budget with trial count. Normalization and Hyperband settings are hand-chosen and could affect secondary comparisons.

free parameters (3)
  • ADTM normalization quantiles = 0.1 and 0.9
    Hand-chosen to avoid distortion by early-trial outliers; paper says other quantiles give similar results, so low impact on the main claim.
  • Hyperband R and eta = R=2150, eta=2.8
    Chosen so Hyperband's total boosting iterations approximate 135,000; affects the fairness of the Hyperband comparison.
  • Trial budget = 135 trials
    Chosen to equal the size of the deterministic full grid; the paper's 'at least 100 trials' finding is defined relative to this budget choice.
axioms (5)
  • domain assumption LightGBM/GPBoost is representative of tree-boosting implementations
    All experiments use one implementation (GPBoost/LightGBM); the conclusion in Section 5 acknowledges this limitation.
  • domain assumption The 59 OpenML datasets and Grinsztajn preprocessing represent typical tabular data
    No formal representativeness argument; dataset choice follows an existing benchmark.
  • ad hoc to paper ADTM-normalized averages across datasets are comparable
    Used to aggregate heterogeneous losses; depends on quantile choice and assumes comparability after normalization.
  • domain assumption Inner validation loss (RMSE/accuracy) is a reliable proxy for test performance
    Hyperparameters are chosen on an 80/20 inner split; this is standard practice but not guaranteed.
  • ad hoc to paper SHAP values from the fitted meta-model quantify hyperparameter importance
    Section 4 concludes all hyperparameters matter from a single fitted meta-model without uncertainty intervals or significance tests.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Selecting Hyperparameters for Tree-Boosting." pith.science (2026). https://pith.science/paper/I7Y2IZJ4

@misc{pith2026260205786,
  author       = {Pith},
  title        = {Pith review of: Selecting Hyperparameters for Tree-Boosting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I7Y2IZJ4}},
  note         = {Machine review of arXiv:2602.05786}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Tree-boosting is a widely used machine learning technique for tabular data. However, its out-of-sample accuracy is critically dependent on multiple hyperparameters. In this article, we empirically compare several popular methods for hyperparameter optimization for tree-boosting including random grid search, the tree-structured Parzen estimator (TPE), Gaussian-process-based Bayesian optimization (GP-BO), Hyperband, the sequential model-based algorithm configuration (SMAC) method, and deterministic full grid search using $59$ regression and binary classification data sets. We find that the SMAC method clearly outperforms all the other considered methods on average, and it gives stable performance across a diverse collection of tabular data sets under a fixed tuning budget, which is relevant for users who cannot afford extensive manual trial-and-error tuning. We further observe that (i) a relatively large number of trials larger than $100$ is typically required for accurate tuning, (ii) using default values for hyperparameters or a full search over a small grid often yields very inaccurate models, (iii) all considered hyperparameters can have a material effect on the accuracy of tree-boosting, i.e., there is no small set of hyperparameters that is more important than others, and (iv) choosing the number of boosting iterations using early stopping yields more accurate results compared to including it in the search space for regression tasks.

Figures

Figures reproduced from arXiv: 2602.05786 by Fabio Sigrist, Floris Jan Koster.

Figure 1
Figure 1. Figure 1: Normalized scores as a function of number of trials. The confidence intervals represent the [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SHAP summary plot for regression (top) and classification (bottom) tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relative differences to the best score as a function of the number of trials. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Normalized scores as a function of the number of trials. The confidence intervals represent [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: R2 as a function of the number of trials per data set. The confidence intervals represent uncertainty due to the randomness in the hyperparameter selection methods. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: R2 as a function of the number of trials per data set. The confidence intervals represent uncertainty due to the randomness in the hyperparameter selection methods. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: RMSE as a function of the number of trials per data set. The confidence intervals represent [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RMSE as a function of the number of trials per data set. The confidence intervals represent [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy as a function of the number of trials per data set. The confidence intervals represent [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Log Loss as a function of the number of trials per data set. The confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of hyperparameter search space options regarding the maximal number of [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 3 linked inside Pith

  1. [4]

    ISBN 9783642255656

    Springer-Verlag. ISBN 9783642255656. doi: 10.1007/978-3-642-25566-3

    doi:10.1007/978-3-642-25566-3

  2. [2007]

    Eggensperger, M

    K. Eggensperger, M. Lindauer, N. J. Mallik, A. Biedenkapp, P. Gijsbers, N. Sch¨ ornig, A. Tornede, J. Vanschoren, and F. Hutter. Hpobench: A collection of reproducible multi-fidelity benchmark 9 problems for hyperparameter optimization. InNeurIPS 2021 Datasets and Benchmarks Track,

    2021

  3. [2011]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2011/file/ 86e8f7ab32cfd12577bc2619bc635690-Paper.pdf. P. B¨ uhlmann and T. Hothorn. Boosting algorithms: Regularization, prediction and model fitting. Statistical Science, pages 477–505,

    2011

  4. [2015]

    doi: 10.1109/DSAA.2015.7344817. 11 Appendix A Additional results 45 60 75 90 105 120 135 0.01 0.02 0.03 0.04 R2 45 60 75 90 105 120 135 0.05 0.10 0.15 0.20 0.25 RMSE 45 60 75 90 105 120 135 0.01 0.02 0.03 0.04 Accuracy 45 60 75 90 105 120 135 0.1 0.2 0.3 Log Loss RegressionClassification Default Hyperband TPE Deterministic Grid Random Grid GP-BO SMAC Figu...

    arXiv 2015

  5. [2017]

    URLhttps://proceedings.neurips.cc/paper_files/ paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization,

    2017

  6. [2018]

    URLhttps://arxiv.org/abs/1603.06560. M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54):1–9,

    Pith/arXiv arXiv

  7. [2021]

    URLhttps://arxiv.org/abs/2109.06716. S. Falkner, A. Klein, and F. Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. InProceedings of the 35th International Conference on Machine Learning, pages 1437–1446,

    Pith/arXiv arXiv

  8. [2022]

    Putatunda and K

    10 S. Putatunda and K. M. Rama. A comparative analysis of hyperopt algorithms for optimizing machine learning algorithms.arXiv preprint arXiv:1807.02822,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.