Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

Kaito Baba; Shuhei Watanabe; Yoshihiko Ozaki

REVIEW 2 major objections 5 minor 58 references

Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

T0 review · 2 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard HPI analysis leaks the gating variable's variance into the hyperparameters it activates; the paper proves the leak, redefines local HPI as within-regime variance, and gives a closed-form estimator.

desk verdict A principled fix for conditional HPI with a real leakage theorem, but the estimator's link to its own definition is asserted rather than proved. read the letter →

arxiv 2601.20800 v3 pith:RFM2ANC7 submitted 2026-01-28 cs.LG cs.AI

Kaito Baba , Yoshihiko Ozaki , Shuhei Watanabe This is my paper · ORCID

classification cs.LGcs.AI

keywords hyperparameterimportanceconditionalsearchspacePED-ANOVAfunctionalANOVAPearsondivergencevariancedecompositionlocalautomatedmachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on hyperparameter importance (HPI) analysis in conditional search spaces — spaces where a hyperparameter exists, or changes domain, only when another 'gating' hyperparameter takes a particular value, as in model selection or neural architecture search. It claims the standard local HPI is systematically wrong there: by the law of total variance, the importance score of a conditioned hyperparameter contains the gating variable's variance as an additive term, so inactive variables appear important and the gate itself is underrated. The proposed fix, conditional PED-ANOVA, defines importance using only the within-regime part of the variance and estimates it in closed form from one-dimensional densities, retaining the speed of the original PED-ANOVA. If the paper is right, common workarounds — filtering inactive samples, imputing defaults, expanding domains — produce misleading rankings in conditional spaces, and the corrected estimator gives the first principled alternative.

What carries the argument

The load-bearing object is the regime: each hyperparameter is partitioned into K^(d) disjoint regimes, each with its own domain (a singleton {⊥} when the parameter is inactive), assembled into an extended one-dimensional domain S^(d) = ⊔_i {i} × Z_i^(d) so that regime identity and value live in one object. Inside each regime the estimator forms the Pearson (χ²) divergence between the one-dimensional densities of the top-γ' and top-γ elite sets and aggregates over regimes with weight (α_i^(d))²/β_i^(d), the squared regime frequency in the tight set divided by its frequency in the loose set. The within-regime restriction is what excludes the inter-regime term that carries the gate's leaked var

What would settle it

Construct a synthetic conditional objective with a flat plateau: inside a regime, the objective is constant over all configurations that qualify for the top-γ' set but varies sharply just below the threshold, so the elite-set indicator carries no information about the parameter while the objective's local marginal mean does. Compute the paper's closed-form estimator (Eq 11) and a brute-force Monte Carlo evaluation of the definition (Eq 9) on the same top-γ samples. If the two diverge materially — near-zero from the estimator, substantial from the direct computation — the bridge fails and the s

Watch

Extended reading notes

Core claim

Central claim: for a conditioned hyperparameter x^(d), the standard local marginal variance splits into within-regime and inter-regime terms, and the inter-regime term equals the local marginal variance of the gating variable that selects the regime (Theorem F.1: v_γ^(d) = v_γ,within^(d) + v_γ^(c)). Because an upstream variable fixes the regime, that variance belongs to the gate, not to x^(d); a parameter inactive in a regime inherits the gate's whole importance. The fix redefines conditional local HPI as within-regime variance alone (Definition 4.1); Theorem 4.2 supplies a closed-form estimator — a regime-weighted sum of Pearson divergences between top-γ' and top-γ one-dimensional densities

Load-bearing premise

The estimator measures how much a hyperparameter's value shifts the chance of landing in the top-performing set, and the paper treats that as measuring how much it changes the objective inside that set — an equivalence inherited from PED-ANOVA and not re-proven for the conditional case.

Editorial extensions

If this is right

Filtering, imputation, and domain expansion — the standard ways existing HPI tools handle conditional spaces — are shown to be systematically misleading: they flatten importance across active and inactive parameters and suppress the gating variable that actually decides which branch can win.
Inactive conditional hyperparameters get exactly zero importance under the corrected estimator, since their regimes are degenerate and contribute zero divergence; the uncorrected score, by contrast, hands them the gating variable's variance.
The corrected importance is target-aware: as the performance quantile γ' tightens, importance shifts from the gate (which decides the winning branch) to the active parameter that must be tuned within that branch, across disjoint, overlapping, nested, and domain-shifting structures.
The estimator is closed-form and one-dimensional, so it keeps PED-ANOVA's speed; runtime comparisons show it faster than surrogate-based baselines at every evaluated sample count.
The definition strictly generalizes PED-ANOVA — a single regime recovers the original local HPI exactly — so the method is an extension rather than a competing estimator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit: any existing HPI workflow that currently discards or imputes conditional parameters can re-run the same evaluation data through the within-regime estimator, and the gap between the old and new rankings is, by the paper's leakage theorem, a direct measurement of how much gating contamination was present.
Testable extension: the regime function is supplied by the search-space designer, but nothing in the math requires it; clustering the top-γ set to learn regimes automatically would turn the estimator into a general tool for local importance in arbitrary stratified subspaces, not just explicitly declared conditional hierarchies.
Portability: the within-versus-inter variance split is a generic statement about stratified input spaces, so the same correction applies to sensitivity analysis over mixtures, missing-data strata, or any setting where a covariate defines subpopulations — not only hyperparameter optimization.
Open validation the paper leaves for follow-up: because the closed form estimates the variance of the elite-set indicator rather than of the objective's local mean directly, comparing it against a brute-force evaluation of the definition on objectives with flat plateaus inside the top region would show how far the indicator proxy stretches.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real contribution. The leakage observation (Theorem F.1) — that standard local HPI gives conditioned parameters the gating variable's variance additively — is new, and the paper shows the effect empirically. The proposed fix, within-regime variance, is natural, and the closed-form estimator in Eq. (11) is correctly derived in Appendix B.3 for the indicator objective. When K=1 it reduces to PED-ANOVA. Code is public and the experiments are reproducible. That part deserves a serious referee.

The soft spot is the one the reader flagged: Definition 4.1 defines conditional local HPI through the objective's local marginal mean (Eq. 9), but Theorem 4.2 and Algorithm 1 estimate the within-regime variance of the indicator b_{γ'}. The paper says normalizing the theorem's quantity gives the conditional local HPI, but the bridge is not proved. This is inherited from PED-ANOVA's indicator approximation, but it is not re-established for the conditional decomposition. It may be fine, but the authors should either prove the approximation under stated conditions or redefine the estimator as the indicator-based quantity and soften the definitional claim. The abstract also overstates the leakage theorem slightly: Eq. (38) holds under the gating assumptions in Appendix F.1, not for every conditional space. That is a one-sentence fix.

Minor stuff: the baselines are deliberately naive (filtering, imputation, expansion), which is the point, but a dummy-coded f-ANOVA baseline would make the comparison stronger. The main-text real-world evaluation is one YAHPO instance with the rest in an appendix; acceptable but thin. The runtime comparison is fine.

The central argument holds up in my reading. The estimator is well-defined, the math is clean, and the empirical behavior matches the leakage theorem. With the definitional gap closed or honestly scoped, this is a solid contribution to AutoML and HPO interpretation. Recommend peer review with a request to address the missing bridge.

Referee Report

2 major / 5 minor

Summary. The paper proposes conditional PED-ANOVA (condPED-ANOVA), an HPI estimator for conditional hyperparameter search spaces. Standard local HPI is decomposed into within-regime and inter-regime variance; the authors argue that the inter-regime term is induced by gating hyperparameters and should not be attributed to conditionally active or domain-shifted hyperparameters. They define conditional local HPI as the normalized within-regime variance, derive a closed-form estimator (Eq. 11) that aggregates per-regime Pearson divergences, and prove the derivation in Appendix B.3. Experiments on synthetic objectives and the YAHPO Gym rbv2_super scenario show that naive filtering/imputation/expansion baselines misbehave, while condPED-ANOVA gives plausible importances and is faster than the baselines.

Significance. The contribution is timely and practically relevant: AutoML and HPO systems routinely face conditional spaces, and no principled HPI estimator has been available for them. The closed-form derivation in Appendix B.3 is clean, Theorem F.1 provides a falsifiable leakage prediction that is confirmed in Figure 3, the code is released, and the empirical comparison includes a wide range of baselines. If the gap between the objective-based Definition 4.1 and the indicator-based Theorem 4.2 is closed, the paper would be a solid and directly applicable contribution.

major comments (2)

[Definition 4.1 vs. Theorem 4.2 (§4.2–§4.3)] The definition of conditional local HPI (Definition 4.1, Eq. (9)) uses the objective-based local marginal mean g^(d)_γ(I,Z). Theorem 4.2 then states that the within-regime variance is given by Eq. (11) with D_PE of the indicator b_γ' = 1{x∈X_γ'}. The proof in Appendix B.3 settles Eq. (11) for the indicator only; no statement or argument shows that this indicator variance equals or approximates the objective variance in Eq. (9). Since all experiments use Eq. (11) as the implemented estimator of Definition 4.1, this is a load-bearing gap. If the indicator approximation of PED-ANOVA (Section 3.3) is inherited, it must be stated explicitly and, ideally, re-proved under the within-regime decomposition; otherwise, please redefine conditional local HPI as an indicator-based quantity.
[Theorem F.1 (App. F.1–F.2)] The leakage theorem is restricted to a single gating coordinate with I^(d)=φ(x^(c)) and no direct gating effect (Eq. (37)). For general conditional spaces, where a parent also affects performance within the regime or regimes depend on multiple coordinates, the equality v^(d)_γ = v^(d)_γ,within + v^(c)_γ need not hold. Section 4.2 and the conclusion nonetheless present the leakage as a general failure of standard local HPI. Please state the assumptions under which leakage applies or provide a more general result; otherwise the motivation is broader than the theory.

minor comments (5)

[Algorithm 1] Algorithm 1 does not list the regime mapping r^(d) as input or explain how D^(d)_{γ,i} is constructed from it; add this to make the procedure reproducible.
[Eq. (5) and Section 3.3] The symbol v^(d)_γ in Eq. (5) is used for the indicator-based local variance without a distinguishing superscript, while Section 3.2 defines it for the objective; add a distinguishing notation or an explicit approximation statement.
[App. B.3] The proof sets κ=|D_γ'|/|D_γ|=γ'/γ, ignoring the floor effect of the empirical quantile. State this as an approximation (or use the exact discrete factor).
[Title/Abstract] The title and abstract mention 'dynamic' search spaces, but the paper treats static conditional structure; consider clarifying the terminology.
[Section 6] The real-world results are qualitative; a brief statement that no ground-truth importance is available would set appropriate expectations.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction: Theorem 4.2/Eq (11) and Theorem F.1 are derived from variance decompositions; the only inherited element is PED-ANOVA's indicator proxy, which is external published code, not a fitted input.

full rationale

The central derivation chain is not circular in the fitting or definitional-collapse sense. Theorem 4.2/Eq (11) is derived for the within-regime variance of the level-set indicator b_{γ'} (Appendix B.3, Eq (30) and its proof), and Section 4.3 explicitly says the authors 'adopt a PED-ANOVA-style construction.' The jump from objective-based Definition 4.1 (Eq (9)) to the indicator-based estimator is the same indicator approximation used in PED-ANOVA [48], an external, published, Optuna-implemented method; sharing a co-author (Watanabe) does not make the cited result self-supporting. Theorem F.1/Eq (38) follows algebraically from the law of total variance plus stated assumptions (Eqs (36)-(37)); Figure 3 is a simulation consistent with that algebra, not a fitted constant. No parameter is tuned to the target claim: KDE uses Scott's rule (Appendix C.1). The only caveat is that Theorem 4.2's quantity is indicator-based whereas Definition 4.1 is objective-based; this is an unproved approximation bridge inherited from PED-ANOVA, a correctness/validity concern rather than a circular reduction. Score 2 reflects that minor inherited self-citation/indicator proxy, not a finding that the paper's predictions reduce to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central estimator depends on the regime specification (known from the search space), the indicator-objective bridge inherited from PED-ANOVA, and a standard variance decomposition. No fitted constants or invented entities; KDE bandwidth is a data-dependent smoothing choice.

free parameters (1)

KDE bandwidth (Scott's rule) = n^{-1/5} · σ̂ per coordinate
Used to estimate the one-dimensional densities p_{γ',i} and p_{γ,i} in Eq (11). It is a data-dependent smoothing parameter, but a standard plug-in rule, not tuned to the target result.

assumptions (4)

domain assumption Regime function r^{(d)} and regime-specific domains Z_i^{(d)} are known a priori.
Section 4.1 formalizes conditionality as regimes; the estimator requires the practitioner to specify these, and errors in the regime map would propagate.
ad hoc to paper The objective-based conditional local HPI (Eq 9) is approximated by the indicator-based within-regime variance of b_{γ'} (Eq 11).
Section 4.3 identifies Theorem 4.2's indicator-based quantity with Definition 4.1's objective-based quantity without a proof; this matches PED-ANOVA's indicator approximation (Section 3.3) but is a load-bearing modeling choice.
domain assumption Gating variable has no within-regime effect (Eq 37: E[b_{γ'} | x^{(c)}] = E[b_{γ'} | I^{(d)}]).
Theorem F.1's leakage result assumes this; real gating variables may also have within-regime effects, so the leakage bound v_γ^{(c)} ≤ v_γ^{(d)} applies only under this condition.
standard math Absolute continuity of the top-set marginals: μ_{γ'} ≪ μ_γ.
Since X_{γ'} ⊂ X_γ, absolute continuity holds; used throughout Appendix B for the Radon–Nikodým derivative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces." pith.science (2026). https://pith.science/paper/RFM2ANC7

@misc{pith2026260120800,
  author       = {Pith},
  title        = {Pith review of: Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFM2ANC7}},
  note         = {Machine review of arXiv:2601.20800}
}

read the original abstract

We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters. Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters. To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes. Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importances in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure. Our code is publicly available at https://github.com/kAIto47802/condPED-ANOVA.

Figures

Figures reproduced from arXiv: 2601.20800 by the authors.

**Figure 1.** condPED-ANOVA (𝛾 = 1.0) HPI computed for the synthetic objectives. For the objective with conditional activation (Equation (12)), the gating hyperparameter 𝑐 determines which branch is active: 𝑥 is present only when 𝑐 < 0.5, whereas 𝑦 is present only when 𝑐 ≥ 0.5. For the objective with regime-dependent domains (Equation (15)), the gating hyperparameter 𝑐 determines the domain of 𝑥 and 𝑦. The lines denote the mean, … view at source ↗

**Figure 2.** Baseline HPIs computed with naive extensions of existing methods for the synthetic objective with conditional [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Ablation study results using standard local HPI instead of our conditional local HPI. The lines denote the mean, and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Ablation study results using the naive aggregation scheme (Equation (18)) instead of our expression (Theorem 4.2). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: condPED-ANOVA HPIs on a real-world problem from YAHPO Gym [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 7.** Figure 7: condPED-ANOVA HPI computed for the synthetic [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 6.** Figure 6: Runtime comparison against baseline HPI meth [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 8.** Figure 8: condPED-ANOVA (𝛾 = 1.0) HPI computed for the synthetic objectives with varying 𝑁. The lines denote the mean, and the shaded regions denote the standard error, both computed over ten independent runs with different random seeds. zoom-in because the definition of the top…

**Figure 10.** Figure 10: Baseline HPIs computed with naive extensions [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 9.** Figure 9: Baseline HPIs computed with naive extensions of existing methods for the synthetic objective with conditional [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 11.** Figure 11: condPED-ANOVA (𝛾 = 1.0) HPI values before normalization, i.e., 𝑣 (𝑑) 𝛾,within, computed for the synthetic objectives. The lines denote the mean, and the shaded regions denote the standard error, both computed over ten independent runs with different random seeds. valu…

**Figure 12.** Figure 12: condPED-ANOVA (𝛾 = 1.0) HPI computed for the synthetic objectives on nested conditional activations (Equations (33) and (34)). The lines denote the mean, and the shaded regions denote the standard error, both computed over ten independent runs with different random s…

**Figure 13.** Figure 13: condPED-ANOVA (𝛾 = 1.0) HPI computed for the synthetic objectives on the combined conditional activation and domain shifts in a three-way branching setting (Equation (35)). The lines denote the mean, and the shaded regions denote the standard error, both computed ove…

**Figure 14.** Figure 14: condPED-ANOVA HPIs on real-world problems from the YAHPO Gym [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 4 canonical work pages

[1]

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization frame- work. InThe 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2623–2631

2019
[2]

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization.Advances in Neural Information Processing Systems24 (2011)

2011
[3]

James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of Machine Learning Research13, 10 (2012), 281–305

2012
[4]

James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. InInternational Conference on Machine Learning. PMLR, 115–123

2013
[5]

André Biedenkapp, Marius Lindauer, Katharina Eggensperger, Frank Hutter, Chris Fawcett, and Holger Hoos. 2017. Efficient parameter importance analysis via ablation with surrogates. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. doi:10.1609/aaai.v31i1.10657

work page doi:10.1609/aaai.v31i1.10657 2017
[6]

André Biedenkapp, Joshua Marben, Marius Lindauer, and Frank Hutter. 2019. CAVE: Configuration assessment, visualization and evaluation. InLearning and Intelligent Optimization: 12th International Conference, LION 12, Kalamata, Greece, June 10–15, 2018, Revised Selected Papers. Springer-Verlag, 115–130. doi:10.1007/ 978-3-030-05348-2_10

2019
[7]

Martin Binder, Florian Pfisterer, and Bernd Bischl. 2020. Collecting empirical data about hyperparameters for data driven AutoML. In7th ICML Workshop on Automated Machine Learning (AutoML 2020)

2020
[8]

Leo Breiman. 2001. Random Forests.Machine Learning45, 1 (2001), 5–32. doi:10.1023/A:1010933404324

doi:10.1023/a:1010933404324 2001

Show all 58 references

[9]

Brent Komer, James Bergstra, and Chris Eliasmith. 2014. Hyperopt-Sklearn: Automatic hyperparameter configuration for Scikit-Learn. InProceedings of the 13th Python in Science Conference. 32–37. doi:10.25080/Majora-14bd3278-006

2014 doi
[10]

Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, Kevin Leyton-Brown, et al . 2013. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. InNIPS workshop on Bayesian Optimization in Theory and Pr...

2013
[11]

Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the 35th International Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 1437–1446

2018
[12]

Chris Fawcett and Holger H. Hoos. 2016. Analysing differences between algo- rithm configurations through ablation.Journal of Heuristics22, 4 (2016), 431–458. doi:10.1007/s10732-014-9275-9

2016 doi
[13]

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. 2022. Auto-sklearn 2.0: hands-free AutoML via meta-learning. Journal of Machine Learning Research23, 1, Article 261 (2022), 61 pages

2022
[14]

Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated ma- chine learning. InAdvances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc

2015
[15]

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. InProceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, Article 392, 8 pages

2018
[16]

Giles Hooker. 2007. Generalized functional ANOVA diagnostics for high- dimensional functions of dependent variables.Journal of Computational and Graphical Statistics16, 3 (2007), 709–732

2007
[17]

Jianhua Z Huang. 1998. Projection estimation in multiple regression with ap- plication to functional ANOVA models.The Annals of Statistics26, 1 (1998), 242–272

1998
[18]

Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. 2014. An efficient approach for assessing hyperparameter importance. InInternational Conference on Machine Learning. PMLR, 754–762

2014
[19]

Hoos, and Kevin Leyton-Brown

Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2013. Identifying key algorithm parameters and instance features using forward selection. InLearning and Intelligent Optimization. Springer Berlin Heidelberg, 364–381

2013
[20]

2019.Automated machine learning: Methods, systems, challenges(1st ed.)

Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. 2019.Automated machine learning: Methods, systems, challenges(1st ed.). Springer Publishing Company, Incorporated

2019
[21]

Rodolphe Jenatton, Cedric Archambeau, Javier González, and Matthias Seeger
[22]

Hoos, Frank Hutter, and Kevin Leyton- Brown

Lars Kotthoff, Chris Thornton, Holger H. Hoos, Frank Hutter, and Kevin Leyton- Brown. 2017. Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA.Journal of Machine Learning Research18, 25 (2017), 1–5

2017
[23]

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A novel bandit-based approach to hyperparameter optimization.Journal of Machine Learning Research18, 185 (2018), 1–52

2018
[24]

Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter
[25]

Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Joshua Marben, Philipp Müller, and Frank Hutter. 2019. BOAH: A tool suite for multi-fidelity Bayesian optimization & analysis of hyperparameters.arXiv preprint arXiv:1908.06756(2019)

2019 arXiv
[26]

Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. 2013. Un- derstanding variable importances in forests of randomized trees. InAdvances in Neural Information Processing Systems, Vol. 26

2013
[27]

Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. InAdvances in Neural Information Processing Systems, Vol. 30

2017
[28]

Xingchen Ma and Matthew Blaschko. 2020. Additive tree-structured covariance function for conditional parameter spaces in Bayesian optimization. InProceed- ings of the Twenty Third International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learni...

2020
[29]

Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the State of the Art of Evaluation in Neural Language Models. InInternational Conference on Learning Representations

2018
[30]

Julia Moosbauer, Julia Herbinger, Giuseppe Casalicchio, Marius Lindauer, and Bernd Bischl. 2021. Explaining hyperparameter optimization via partial de- pendence plots. InAdvances in Neural Information Processing Systems, Vol. 34. 2280–2291

2021
[31]

Emanuel Parzen. 1962. On estimation of a probability density function and mode. The Annals of Mathematical Statistics33, 3 (1962), 1065–1076

1962
[32]

Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. 2022. YAHPO Gym - An efficient multi-objective multi-fidelity bench- mark for hyperparameter optimization. InProceedings of the First International Conference on Automated Machine Learning ...

2022
[33]

Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. 2019. Tunability: Importance of hyperparameters of Machine learning algorithms.Journal of Machine Learning Research20, 53 (2019), 1–32

2019
[34]

Walsh, and Giuseppe Casalicchio

Julian Rodemann, Federico Croppi, Philipp Arens, Yusuf Sale, Julia Herbinger, Bernd Bischl, Eyke Hüllermeier, Thomas Augustin, Conor J. Walsh, and Giuseppe Casalicchio. 2026. Explaining Bayesian optimization by Shapley values facilitates human-AI collaboration for exosuit pers...

2026
[35]

David W. Scott. 1992.Multivariate density estimation: theory, practice, and visual- ization. John Wiley & Sons. doi:10.1002/9780470316849

1992 doi
[36]

Sarah Segel, Helena Graf, Edward Bergman, Kristina Thieme, Marcel Dominik Wever, Alexander Tornede, Frank Hutter, and Marius Lindauer. 2025. DeepCAVE: A visualization and analysis tool for automated machine learning.Journal of Machine Learning Research2025, 26 (2025)

2025
[37]

Sarah Segel, Helena Graf, Alexander Tornede, Bernd Bischl, and Marius Lindauer
[38]

L. S. Shapley. 1953.A value for n-person games. Princeton University Press, Princeton, 307–318. doi:10.1515/9781400881970-018

1953 doi
[39]

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical Bayesian optimization of machine learning algorithms. InAdvances in Neural Information Processing Systems, Vol. 25

2012
[40]

Ilya M Sobol. 1993. Sensitivity estimates for nonlinear mathematical models, mathematical modeling and computational Experiment.Mathematical Modeling and Computational Experiment(1993), 407–414

1993
[41]

Ilya M Sobol. 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates.Mathematics and Computers in Simulation55, 1-3 (2001), 271–280

2001
[42]

Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. 2007. Bias in random forest variable importance measures: Illustrations, sources and a solution.BMC Bioinformatics8, 1 (2007), 25. doi:10.1186/1471-2105-8-25

2007 doi
[43]

Kevin Swersky, David Duvenaud, Jasper Snoek, Frank Hutter, and Michael A. Osborne. 2014. Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces.arXiv preprint arXiv:1409.4011(2014)

2014 arXiv
[44]

Daphne Theodorakopoulos, Frederic Stahl, and Marius Lindauer. 2024. Hyperpa- rameter importance analysis for multi-objective AutoML. InEuropean Conference on Artificial Intelligence. IOS Press. doi:10.3233/faia240602

2024 doi
[45]

Hoos, and Kevin Leyton-Brown

Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2013. Auto-WEKA: Combined selection and hyperparameter optimization of classifica- tion algorithms. InProceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 847–8...

2013
[46]

Jan N van Rijn and Frank Hutter. 2018. Hyperparameter importance across datasets. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2367–2376. Baba et al

2018
[47]

van Rijn, Bernd Bischl, and Luis Torgo

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. 2014. OpenML: Networked science in machine learning.SIGKDD Explor. Newsl.15, 2 (June 2014), 49–60. doi:10.1145/2641190.2641198

2014
[48]

Shuhei Watanabe, Archit Bansal, and Frank Hutter. 2023. PED-ANOVA: Effi- ciently quantifying hyperparameter importance in arbitrary subspaces. InPro- ceedings of the Thirty-Second International Joint Conference on Artificial Intelli- gence. 4389–4396

2023
[49]

Marcel Wever, Maximilian Muschalik, Fabian Fumagalli, and Marius Lindauer
[50]

Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. 2022. Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. InInternational Conference on Learning Representations

2022
[51]

which hyperparameters are worth tuning

Lucas Zimmer, Marius Lindauer, and Frank Hutter. 2021. Auto-PyTorch: Multi- fidelity metalearning for efficient and robust AutoDL.IEEE Transactions on Pattern Analysis and Machine Intelligence43, 9 (2021), 3079–3090. doi:10.1109/ TPAMI.2021.3067763 A Additional Related Work A....

2021
[56]

study partial dependence plots (PDPs) for explaining BO-based HPO runs and show that naive PDPs can be biased due to the non-uniform, sequential sampling of BO. They propose uncertainty- aware PDPs based on the BO surrogate, together with a partitioning of the hyperparameter s...
[57]

Here, the Pearson (𝜒 2) divergence is defined as: 𝐷PE(𝜈∥𝜇) := ∫ 𝑑𝜈 𝑑𝜇−1 2 𝑑𝜇,(24) for probability measures𝜈 and𝜇 with𝜈≪𝜇

The within-regime local marginal variance for the 𝑑-th hyperpa- rameter at level𝛾, computed using the indicator function𝑏𝛾′ := 1{𝑥∈ X𝛾′}, is given by: 𝑣(𝑑) 𝛾,within = 𝛾′ 𝛾 2 ∑︁ 𝑖:𝛽(𝑑) 𝑖 >0 𝛼(𝑑) 𝑖 2 𝛽(𝑑) 𝑖 𝐷PE 𝜇(𝑑) 𝛾′,𝑖 𝜇(𝑑) 𝛾,𝑖 ,(23) By normalizing the variance across all hype...
[58]

• iaml: Interpretable AutoML scenarios that combine predic- tive performance with resource-usage and interpretability objectives, with multi-fidelity via training-size fractions

tasks with epoch-based fidelity, exposing multiple val- idation/test performance metrics and training time. • iaml: Interpretable AutoML scenarios that combine predic- tive performance with resource-usage and interpretability objectives, with multi-fidelity via training-size f...
[2017]

InProceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol

Bayesian optimization with tree-structured dependencies. InProceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70). PMLR, 1655–1664
[2022]

SMAC3: A versatile Bayesian optimization package for hyperparameter optimization.Journal of Machine Learning Research23, 54 (2022), 1–9

2022
[2023]

InProceedings of the Second International Conference on Automated Machine Learning (Proceedings of Machine Learning Research, Vol

Symbolic explanations for hyperparameter optimization. InProceedings of the Second International Conference on Automated Machine Learning (Proceedings of Machine Learning Research, Vol. 224). PMLR, 2/1–22
[2026]

InProceedings of the AAAI Conference on Artificial Intelligence

HyperSHAP: Shapley values and interactions for explaining hyperparame- ter optimization. InProceedings of the AAAI Conference on Artificial Intelligence

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization frame- work. InThe 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2623–2631

2019

[2] [2]

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization.Advances in Neural Information Processing Systems24 (2011)

2011

[3] [3]

James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of Machine Learning Research13, 10 (2012), 281–305

2012

[4] [4]

James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. InInternational Conference on Machine Learning. PMLR, 115–123

2013

[5] [5]

André Biedenkapp, Marius Lindauer, Katharina Eggensperger, Frank Hutter, Chris Fawcett, and Holger Hoos. 2017. Efficient parameter importance analysis via ablation with surrogates. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. doi:10.1609/aaai.v31i1.10657

work page doi:10.1609/aaai.v31i1.10657 2017

[6] [6]

André Biedenkapp, Joshua Marben, Marius Lindauer, and Frank Hutter. 2019. CAVE: Configuration assessment, visualization and evaluation. InLearning and Intelligent Optimization: 12th International Conference, LION 12, Kalamata, Greece, June 10–15, 2018, Revised Selected Papers. Springer-Verlag, 115–130. doi:10.1007/ 978-3-030-05348-2_10

2019

[7] [7]

Martin Binder, Florian Pfisterer, and Bernd Bischl. 2020. Collecting empirical data about hyperparameters for data driven AutoML. In7th ICML Workshop on Automated Machine Learning (AutoML 2020)

2020

[8] [8]

Leo Breiman. 2001. Random Forests.Machine Learning45, 1 (2001), 5–32. doi:10.1023/A:1010933404324

doi:10.1023/a:1010933404324 2001