Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Hyperparameter Optimization via Interacting with Probabilistic Circuits

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read IBO-HPC replaces acquisition-function optimization with conditional sampling from a probabilistic circuit, so user priors are reflected exactly in the next candidate.

desk verdict Practical interactive HPO method with strong empirical work, but the main theory as stated doesn't hold—Prop 1 ignores the Bernoulli switch and Prop 3 is a surrogate identity, not a convergence rate. read the letter →

arxiv 2505.17804 v1 pith:G3XATSHQ submitted 2025-05-23 cs.LG cs.AI

classification cs.LGcs.AI
keywords hyperparameteroptimizationBayesianprobabilisticcircuitsinteractivemachinelearningconditionalsamplingsum-productnetworksuserpriorsacquisitionfunction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces IBO-HPC, a Bayesian optimization method for hyperparameter tuning that replaces the usual acquisition function with direct conditional sampling from a probabilistic circuit. The circuit is trained on the joint distribution of hyperparameters and evaluation scores, and new candidate configurations are drawn from the distribution conditioned on the best score observed so far, optionally with a user-provided prior over some hyperparameters. The paper argues that because candidates are generated by exact conditioning and sampling rather than by optimizing a weighted acquisition function, user beliefs are reflected in the selected configurations exactly as given. It further claims that this makes the method competitive with strong hyperparameter optimization baselines without user input and better than interactive BO baselines when user knowledge is supplied, while a decay mechanism lets it recover from misleading advice.

What carries the argument

The load-bearing object is the probabilistic circuit (PC), a tractable graphical model---here a mixed sum-product network---that encodes a joint distribution over the hybrid hyperparameter space and the evaluation score. The selection policy is built on exact conditional sampling: after fitting the PC to the evaluated configurations, it forms $s(H' \mid \hat{H}, F = f^*)$ via tractable conditioning and draws candidates from this distribution (Eq. 1). A user prior $q(\hat{H})$ is fused by sampling $N$ conditions from $q$, computing one conditional per condition, and selecting the most likely candidate from each; a Bernoulli switch with decay $\gamma^t$ controls how long the feedback is used so the method can recover from misleading advice. Fitting is done with LearnSPN, which alternates clustering and independence tests to produce a smooth, decomposable PC.

What would settle it

Run IBO-HPC on a one-dimensional objective with a known global optimum from five random initial points and measure whether samples from $s(H \mid F = f^*)$ concentrate nearer the optimum than random search; separately, provide a Gaussian prior over one hyperparameter with decay disabled and test whether the empirical density of chosen values matches the prior---if either fails, the representability premise is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that interactive hyperparameter optimization can be made both simpler and more faithful to user intent by modeling the joint distribution over hyperparameters $H$ and evaluation score $F$ with a probabilistic circuit, and selecting each new configuration by sampling from the conditional distribution $s(H' \mid \hat{H}, F = f^*)$, where $f^*$ is the best score observed so far and $\hat{H}$ is any user-supplied condition on a subset of hyperparameters. Since probabilistic circuits support exact marginalization, conditioning, and sampling, the method removes the inner-loop optimization of an acquisition function, and the selected candidates' marginal over the user-specified hyperparameters matches the user's prior $q(\hat{H})$ in the limit of many samples (Proposition 1). The paper further shows that the policy minimizes simple regret (Proposition 2) and that its per-iteration convergence is lower-bounded by an expected-improvement term expressed through error functions over the circuit's Gaussian mixture components (Proposition 3). On benchmark tasks covering neural architecture search and continuous, discrete, and mixed HPO spaces, the authors report that without user knowledge the method is competitive with strong BO baselines, while with beneficial feedback it outperforms interactive BO baselines and reaches the same quality 2--10 times faster, and with misleading feedback it recovers.

Load-bearing premise

The whole scheme rests on the learned probabilistic circuit's joint distribution over hyperparameters and scores being a faithful stand-in for the true one; with only a handful of evaluated configurations, that fidelity is not guaranteed, and a miscalibrated conditional will send sampling to the wrong regions.

Editorial extensions

If this is right

  • User priors can be injected at any iteration, and the marginal distribution of the next candidate over the specified hyperparameters converges to the user's prior, so feedback acts exactly where the user intends.
  • Removing the acquisition-function inner loop reduces suggestion overhead; the paper reports the selection policy is faster than SMAC in 4 of 5 cases, with the gap growing in larger search spaces.
  • Misleading feedback is phased out geometrically via the decay factor, and the empirical runs show the method returns to the same performance as if no feedback had been given.
  • On the benchmark suite, beneficial feedback improves convergence by a median factor of 2--10 relative to running without feedback, and the method outperforms the interactive baselines in most compared tasks.
  • Because the policy is defined as a distribution over configurations, the same sampling machinery applies whether feedback is a point value or a distribution over hyperparameter values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not pursued in the paper: the same conditional-sampling mechanism could be applied to multi-fidelity HPO by treating fidelity as an additional variable in the joint distribution, so a user could steer both the configuration and the budget.
  • Because the feedback-adhering property is defined formally, it could be adopted as an evaluation criterion for other interactive BO methods, turning 'the prior was not reflected' from a qualitative complaint into a measurable quantity.
  • A testable extension would be to set the decay $\gamma$ automatically from the data---for example, by comparing the likelihood of recently observed scores under the user-conditional and unconditional surrogates---rather than treating it as a fixed hyperparameter.
  • The paper's own limitation that low leaf variance can trap the sampler suggests a diagnostic: monitor the conditional's variance during the run and warn the user when it collapses, which would also make the method more robust for non-experts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces IBO-HPC, a Bayesian optimization method for hyperparameter optimization that replaces the usual acquisition function with conditional sampling from a probabilistic circuit (PC). A PC is fit to evaluated configurations and their scores; new candidates are sampled from the PC conditioned on the best score observed so far. User knowledge about subsets of hyperparameters can be supplied at any time as a prior distribution, and the method also includes a Bernoulli-decay mechanism intended to let the optimizer recover from misleading feedback. The authors define a formal notion of feedback-adhering interactive policies, claim that IBO-HPC satisfies it (Proposition 1), claim global optimization in the sense of simple regret (Proposition 2), and derive a per-iteration expected-improvement lower bound (Proposition 3). They then report extensive experiments on JAHS, NAS-Bench-101/201, HPO-B, PD1, and FCNet with 500 seeds in the main runs, comparing against SMAC, BO with RF/TPE, πBO, BOPrO, Priorband, random search, and local search. The empirical section is substantial and shows that IBO-HPC is competitive without user knowledge and often faster or better with beneficial user knowledge, with median speed-ups of 2–10×.

Significance. If the theoretical claims were sound, the paper would make a useful contribution to interactive HPO: it offers an acquisition-function-free selection policy based on tractable conditional sampling, allows priors to be supplied mid-run, and provides a natural recovery mechanism from harmful feedback. The empirical study is a genuine strength: the use of standard benchmarks, the large number of seeds, the inclusion of distributional and point-mass interactions, the recovery experiments, and the reporting of Wilcoxon significance tests go well beyond what is typical. Code and raw logs are also promised, which supports reproducibility. However, several of the paper's central theoretical assertions are not supported as stated, and because the 'accurate reflection of user beliefs' claim is one of the paper's headline contributions, the theory needs substantial repair before the claims can be accepted.

major comments (4)
  1. [Section 3.3 / App. D.1, Proposition 1] Proposition 1 is not supported by the proof because Algorithm 1 only uses the user prior when the Bernoulli variable b equals 1. In Algorithm 1, Line 6 draws b~Ber(ρ); the prior-conditioned branch runs only when b=1 (Line 7), while Line 16 samples from s(H|F=f*) when b=0. The actual policy therefore induces the mixture ρ·q(Ĥ)+(1−ρ)·s(Ĥ|F=f*) over the user-specified hyperparameters, not q(Ĥ) as required by Definition 2. Since Proposition 1 explicitly assumes s(Ĥ|F=f*)≠q(Ĥ), the mixture cannot equal q. The proof in App. D.1 treats the decay mechanism as a constant and argues about the prior branch as if it were the whole policy; it never models the Bernoulli switch. The additional appeal to N→∞ and L→∞ also does not repair the issue, because Algorithm 1 makes one uniform draw from a finite set of N candidates and the Bernoulli mixture remains regardless of N. The theorem and its proof, or the algorithm and the definition of feedback adherence, need to be revised so that the stated guarantee matches what the selection policy actually computes.
  2. [App. D.4] The assertion that 'as long as the ground truth distribution p ... is representable by an SPN, we can recover p with arbitrarily small error with iterations T→∞' is unsupported and load-bearing. LearnSPN is stated to locally maximize likelihood on the current dataset, but local likelihood maximization on finite data does not imply consistency or arbitrarily small approximation error of the ground-truth joint distribution. No theorem, rate, or set of identifiability conditions is provided for this recovery claim. Since the quality of every downstream statement about candidate selection and feedback adherence depends on the surrogate s being calibrated, this gap must be addressed: either prove a suitable consistency result under explicit assumptions or clearly weaken the claim to an assumption about the surrogate's quality.
  3. [Section 3.3 / App. D.2, Proposition 2] The proof of Proposition 2 is not valid for continuous hyperparameter spaces. The argument is that positivity of s(H=θ|F=f*) for all θ implies that with T→∞ the probability of sampling the global optimum θ* tends to 1. For a continuous distribution, the probability of sampling a specific point θ* is zero, even if the density is positive there. Moreover, the surrogate is refit as data accumulates, so the sampling distribution is not an i.i.d. sequence from a fixed distribution. The claimed guarantee that simple regret reaches zero should be weakened to an ε-regret statement under additional continuity and support assumptions, or the proposition should be restricted to discrete search spaces.
  4. [Section 3.3 / App. D.3, Proposition 3] Equation (3) is presented as a lower bound on the 'convergence rate' of IBO-HPC, but the derivation in App. D.3 establishes at most a lower bound on a surrogate-based integral that is labeled expected improvement; it does not establish a bound on the algorithm's actual regret or convergence rate as a function of iterations. In addition, the result assumes all PC leaves are Gaussians, whereas the method described in Section 2 uses MSPNs with piecewise polynomial leaves, and the proof drops constants and the 1/2 scaling of the error function while claiming this does not affect the result. The proposition should either be reformulated as a bound on a well-defined algorithm quantity with matching assumptions, or its status as a convergence-rate guarantee should be substantially qualified.
minor comments (5)
  1. [Section 3.2, Eq. (1)] The displayed conditional density in Eq. (1) is written as s(H'|Ĥ,F) = s(H,F)/∫_{H'} s(H,F), which is not the standard definition of conditioning; the numerator should be the joint density over the appropriate variables, and the denominator should be the marginal over the variables being conditioned upon. Please correct the notation.
  2. [Algorithm 1, Line 17] Line 17 adds (θ', f(θ')) to the dataset, but the configuration selected for evaluation is called θ* earlier in the algorithm. The notation should be made consistent so that the selected candidate is the one evaluated and added to D.
  3. [Appendix E.5, Tables 2–4] Several p-values are formatted inconsistently, e.g., '98×10−1' and '0 .9' and '0 .96'. These should be normalized to a uniform numeric format to avoid ambiguities.
  4. [Appendix D.2, Proposition numbering] The proposition proved in App. D.2 is labeled Proposition 4, but it corresponds to Proposition 2 in the main text. Please renumber so that the appendix labels match the main text.
  5. [Definition 1 and Algorithm 1] Definition 1 frames HPO as minimization, but Algorithm 1 and the experimental section condition on f* = max f(D), which is consistent with maximizing validation accuracy. Please clarify whether the score variable is a loss to be minimized or an accuracy to be maximized, and keep the notation consistent throughout.

Circularity Check

3 steps flagged · score 6.0 of 10

Prop. 3's convergence rate is a tautology about the fitted surrogate; Prop. 1's feedback guarantee holds only for the prior-only branch; Prop. 2 restates positive-density sampling.

  1. self definitional [Section 3.3, Proposition 3; Appendix D.3, Eqs. (5)-(17)]
    "Then, the lower bound of the convergence rate of IBO-HPC is given by the expected improvement (EI) in each iteration: τ_s∑ w_i·[∏_j erf((θ*_tj−μ_ij)/(Σ_ijj√2)) − ∏_j erf((θ*_j−μ_ij)/(Σ_ijj√2)) + Lε_i]."

    The proof defines the analyzed quantity in Eq. (5) as ∫ s(θ|y*_t)·I[f(θ)<y*_t]·f(θ), an integral over IBO-HPC's own surrogate conditional, and then rewrites s as the fitted Gaussian mixture and lower-bounds the integral by the mixture's weights w_i, means μ_i, and variances Σ_ijj. The resulting 'convergence rate' therefore states that the surrogate's expected improvement is lower-bounded by an expression in the surrogate's own fitted parameters; it carries no independent information about the true objective beyond a Lipschitz assumption. This is a definitional identity about the fitted model, not a derivation of convergence of the HPO procedure.

  2. self definitional [Algorithm 1 lines 6-16; Proposition 1 and Appendix D.1]
    "if prior q(Ĥ) is given and b = 1 then ... Sample N conditions θ∼q(Ĥ) ... θ∗∼ U(C) else θ∗∼s(H|f∗); Since the decay mechanism ... can be treated as a constant in each iteration, it is enough if s(H\Ĥ|Ĥ=θ̂,F=f∗)·q(Ĥ=θ̂) ≠ s(H\Ĥ|Ĥ=∅,F=f∗)·q(Ĥ=∅) holds ... we conclude that user priors are exactly reflected as specified in our selection policy."

    The proof verifies feedback adherence only for the branch in which b=1: there, the candidates' Ĥ-values are by construction i.i.d. draws from q(Ĥ), so the marginal over Ĥ is q. Algorithm 1, however, executes that branch only with probability ρ and otherwise samples θ*∼s(H|f*) with no conditioning on Ĥ. The policy's true induced marginal is ρ q(Ĥ)+(1−ρ)s(Ĥ|f*), which equals q only if s(Ĥ|f*)=q, the very condition Proposition 1 excludes. The theorem's conclusion is thus an artifact of analyzing one branch as if it were the whole policy, not a property of the full algorithm.

1 more flagged steps
  1. renaming known result [Section 3.3, Proposition 2; Appendix D.2]
    "Assume that w > 0 holds for each weight of a PC s, that each leaf node of s is a distribution p s.t. p(x) > 0 for some x and assume f is not noisy. Then, the PC fulfills the positivity assumption, i.e. s(H = θ, F = f(θ)) > 0. It follows that s(H = θ|F = f*) > 0 for any f* and any θ∈Θ. Thus, with iterations T→∞, the probability of sampling the global optimum θ* in one of the iterations gets 1."

    The proof shows only that any PC with positive weights and positive leaves induces a conditional with full support, so repeated sampling visits every configuration with probability 1 as T→∞. That is the standard convergence statement for any positive-density sampling procedure, such as random search; it does not use the conditional-on-f* mechanism, the LearnSPN surrogate, or the acquisition-free policy. Presenting this as a proposition about IBO-HPC renames a generic property of infinite positive-density search as a method-specific guarantee.

full rationale

The empirical sections are self-contained: IBO-HPC is compared against strong external baselines on JAHS, NAS-Bench, HPO-B, PD1, and FCNet, and no fitted parameter from those experiments is recycled as a theoretical prediction, so the experimental claims are not circular. The circularity is confined to the theoretical analysis. Proposition 3 defines the algorithm's expected improvement as an integral of the surrogate's own conditional and then lower-bounds it by the surrogate's mixture mean/weight/variance parameters; the result is a tautology about the fitted PC and contributes no externally checkable convergence guarantee. Proposition 1's proof treats the Bernoulli switch as a constant and verifies feedback-adherence only in the branch that samples from q, while the actual policy mixes q with s(Ĥ|f*); the guaranteed equality is true by construction in one branch, not derived for the full policy. Proposition 2 is a generic positive-density-search statement. These issues affect the theoretical 'ensures' and 'converges' claims, so a partial circularity score is warranted. There is no load-bearing self-citation: the cited MSPN/PC/SPN results are prior external work, and the central empirical comparisons stand on independent benchmarks.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. Its contributions rest on the modeling assumptions above, chiefly that a LearnSPN-learned PC accurately captures the joint distribution of hyperparameters and scores, plus the objective-specific assumptions in Prop. 3.

free parameters (8)
  • γ (decay factor) = 0.9
    Controls how quickly user prior influence decays after an interaction; hand-set in App. E.8 and shown in Fig. 11 to affect recovery speed.
  • L (surrogate refit interval) = 20
    PC is re-fit every L iterations; hand-set in App. E.8; ablation (Fig. 13) shows little sensitivity.
  • B (candidates per conditional) = 1
    Number of samples from each conditional before selecting by max likelihood; the authors report 'we found that setting B = 1 works surprisingly well' (Section 3.2).
  • N (conditions sampled from user prior) = not specified
    The feedback-adhering proof requires N→∞; the paper never states the finite N used in experiments, so the accuracy of the prior reflection in practice is unquantified.
  • J (initial random configurations) = 5
    All algorithms were initialized with 5 random samples (Section 4).
  • RDC independence threshold = 0.3
    Threshold for the RDC independence test in LearnSPN structure learning (App. E.8).
  • Minimum instances per leaf = dynamic
    Adapted dynamically based on the number of configurations; exact rule not specified (App. E.8).
  • ρ (initial Bernoulli probability of using user prior) = not stated
    Algorithm 1 sets b∼Ber(ρ). ρ is decayed as ρ←γ·ρ, but its initial value is never given; presumably 1.
assumptions (6)
  • domain assumption The true joint distribution p(H,F) over hyperparameters and scores is representable by a smooth and decomposable PC (MSPN) and LearnSPN recovers it from data.
    Invoked in App. D.4: 'as long as the ground truth distribution p ... is representable by an SPN, we can recover p with arbitrarily small error with T→∞.' If the surrogate is not a faithful estimate, conditional sampling from s(H|F=f*) gives poor candidates.
  • domain assumption LearnSPN locally maximizes the log-likelihood of the training data and yields a well-calibrated surrogate.
    App. D.4 relies on Gens and Domingos 2013 for the local MLE property; calibration on small D is not empirically verified.
  • ad hoc to paper The objective f is differentiable, L-Lipschitz, locally convex within a ball around the optimum, and all PC leaves are Gaussians.
    Introduced solely for Proposition 3 (App. D.3). It does not hold for the discrete and categorical dimensions in NAS-Bench and JAHS, so the proposition does not apply to the evaluated tasks.
  • domain assumption Sampling from the user prior q and from the PC conditional is unbiased.
    App. D.4 asserts sampling via Box-Muller and top-down PC sampling is unbiased, which is needed for Eq. (2) to approximate the intended mixture.
  • domain assumption User priors q are tractable distributions over a subset Ĥ that can be sampled i.i.d.
    App. D.4 assumes q is simple, e.g., an isotropic Gaussian or categorical; complex user knowledge is excluded by assumption.
  • domain assumption The marginal of s over Ĥ given F=f* differs from q, so the policy is efficacious.
    Assumed in Proposition 1 to ensure user knowledge changes the sampling distribution; never verified empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hyperparameter Optimization via Interacting with Probabilistic Circuits." pith.science (2026). https://pith.science/paper/G3XATSHQ

@misc{pith2026250517804,
  author       = {Pith},
  title        = {Pith review of: Hyperparameter Optimization via Interacting with Probabilistic Circuits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3XATSHQ}},
  note         = {Machine review of arXiv:2505.17804}
}
read the original abstract

Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. Existing interactive Bayesian optimization (BO) methods incorporate human beliefs by weighting the acquisition function with a user-defined prior distribution. However, in light of the non-trivial inner optimization of the acquisition function prevalent in BO, such weighting schemes do not always accurately reflect given user beliefs. We introduce a novel BO approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable exact conditional inference and sampling. Based on conditional sampling, we construct a novel selection policy that enables an acquisition function-free generation of candidate points (thereby eliminating the need for an additional inner-loop optimization) and ensures that user beliefs are reflected accurately in the selection policy. We provide a theoretical analysis and an extensive empirical evaluation, demonstrating that our method achieves state-of-the-art performance in standard HPO and outperforms interactive BO baselines in interactive HPO.

Figures

Figures reproduced from arXiv: 2505.17804 by the authors.

Figure 1
Figure 1. Interactive Bayesian Hyperparameter Optimization. (Left) We devise an interactive BO method by employing PCs as surrogates encoding a joint distribution over hyperparameters and evaluation scores (omitted for clarity). PCs allow users to directly condition the surrogate on their beliefs during tractable candidate generation, thereby reflecting user knowledge accurately. (Right) Accurately reflecting user beliefs is … view at source ↗
Figure 2
Figure 2. IBO-HPC outperforms state of the art. For 5/5 tasks across three challenging benchmarks, IBO-HPC is competitive with strong baselines when no user knowledge is provided. When beneficial user beliefs are provided ( ), after 5 ( ) or after 10 iterations ( ), it outperforms all competitors w.r.t. convergence and/or solution quality on 4/5 tasks. we employ six real-world benchmarks: NAS-Bench-101 [Ying et al., 2019], NA… view at source ↗
Figure 3
Figure 3. IBO-HPC recovers from misleading interactions. IBO-HPC automatically recovers from ( ) misleading feedback provided as point values at the 5th iteration of the search (1st ). Also, when providing harmful and beneficial beliefs alternatively ( / ), IBO-HPC ( ) catches up with or outperforms 𝜋BO ( ) and BOPrO ( ) in 4/5 cases. and App. E.4 show that the performance of IBO-HPC without user interaction is competitive to… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: IBO-HPC reflects user priors as specified. In contrast to other weighting scheme based methods like 𝜋BO and BOPrO, IBO-HPC reflects the user prior as specified in its selection policy. A.2 Related Work: Hyperparameter Transfer Learning and Benchmarks Hyperparameter Tra…
Figure 6
Figure 6. Figure 6: IBO-HPC is competitive or outperforms strong baselines on HPO-B, PD1 and FCNet. IBO￾HPC outperforms all BO baselines that allow users to provide a prior before optimization when feedback is provided at the 2nd iteration on 3/5 tasks. For the other tasks, only one basel…
Figure 7
Figure 7. Figure 7: IBO-HPC outperforms state of the art. For 4/5 tasks across three challenging benchmarks, IBO-HPC is competitive with strong baselines when no user knowledge is provided. When beneficial user beliefs ( ) are provided, either as distributions ( ) or point values ( , ), i…
Figure 8
Figure 8. Figure 8: IBO-HPC recovers from misleading user feedback. IBO-HPC successfully and consistently recovers from misleading user feedback and performs equally well as if no feedback was given. Also, IBO-HPC handles alternating, contradictory feedback well and is able to leverage be…
Figure 9
Figure 9. Figure 9: CDF of Test Accuracy. The majority of IBO-HPC’s sampled candidate configurations are high-performing configurations. Thus, IBO-HPC invests more computational resources in good configurations than other methods. We conjecture that this is because IBO-HPC selects configu…
Figure 10
Figure 10. Figure 10: CDF of Test Accuracy/Mean Squared Error (MSE). The majority of IBO-HPC’s sampled candidate configurations are high-performing configurations. Thus, IBO-HPC invests more computational resources in good configurations than other methods. We conjecture that this is becau…
Figure 11
Figure 11. Figure 11: Ablation: Effect of γ on recovery of IBO-HPC. As expected, we found that IBO-HPC recovers faster for smaller values of 𝛾. This is because smaller 𝛾 values lead to a higher decay of the probability of conditioning on the provided user knowledge. Thus, with faster decay…
Figure 12
Figure 12. Figure 12: Conditioning on sub-optimal evaluation scores slow down IBO-HPC- Conditioning on the evaluation score of high-performing configurations is crucial for the performance of IBO-HPC. To analyze the effect of conditioning on evaluation scores of sub-optimal configurations,…
Figure 13
Figure 13. Figure 13: L has no significant effect on IBO-HPC’s performance. We found that fixing the surrogate model for 𝐿 = {5, 10, 20, 30} iterations does not lead to significant differences in the per￾formance and convergence speed of IBO-HPC. Only in earlier iterations was a significan…
Figure 14
Figure 14. Figure 14: IBO-HPC is a cost-efficient HPO method. (a) Learning a surrogate and suggesting new configurations is negligible in terms of computational costs compared to training the suggested configurations. We computed the time spent on training configurations (blue) vs. time sp…
Figure 15
Figure 15. Figure 15: IBO-HPC effectively trades off exploration and exploitation. IBO-HPC’s sampling policy naturally and effectively trades off exploration (high sampling variance in early iterations) versus exploitation (low sampling variance in later iterations). We show the sampling v…
Figure 16
Figure 16. Figure 16: IBO-HPC effectively trades off exploration and exploitation. IBO-HPC’s sampling policy naturally and effectively trades off exploration (high sampling variance in early iterations) versus exploitation (low sampling variance in later iterations). We show the sampling v…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 39 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    O ptuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. O ptuna: A next-generation hyperparameter optimization framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623--2631, 2019

  3. [3]

    Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search

    Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter. Jahs-bench-201: A foundation for research on joint architecture and hyperparameter search. In Advances in Neural Information Processing Systems (NeurIPS), 2022

  4. [4]

    Random search for hyper-parameter optimization

    James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13 0 (2), 2012

  5. [5]

    Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

    Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne - Laure Boulesteix, Difan Deng, and Marius Lindauer. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13 0 (2), 2023

  6. [6]

    Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020

    Xavier Bouthillier and Ga \"e l Varoquaux. Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020 . Research report, Inria Saclay Ile de France , January 2020

  7. [7]

    Random forests

    Leo Breiman. Random forests. Machine learning, 45: 0 5--32, 2001

  8. [8]

    Probabilistic circuits: A unifying framework for tractable probabilistic models

    YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, UCLA, 2020

Show all 46 references
  1. [9]

    NAS - B ench-201: Extending the scope of reproducible neural architecture search

    Xuanyi Dong and Yi Yang. NAS - B ench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020

  2. [10]

    Learning the structure of sum-product networks

    Robert Gens and Pedro Domingos. Learning the structure of sum-product networks. In International Conference on Machine Learning (ICML), volume 28, pages 873--880. PMLR, 2013

  3. [11]

    Landwehr, Pavel Komarov, Mehdi Cherti, Kejia (KJ) Shi, Karlson Pfannschmidt, Fabian Linzberger, Christophe Cauet, Anna Gut, Andreas Mueller, and Alexander Fabisch

    Tim Head, MechCoder Gilles, Louppe Iaroslav, Shcherbatyi fcharras, Zé Vinícius, cmmalone, Christopher Schröder, nel215, Nuno Campos, Todd Young, Stefano Cereda, Thomas Fan, Justus Schwabedal, Mikhail Hvass-Labs, Pak SoManyUsernamesTaken, Fred Callaway, Loïc Estève, Lilian Bess...

  4. [12]

    Hyperparameter transfer learning with adaptive complexity

    Samuel Horv \'a th, Aaron Klein, Peter Richtarik, and Cedric Archambeau. Hyperparameter transfer learning with adaptive complexity. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1378--1386, 2021

  5. [13]

    Sequential model-based optimization for general algorithm configuration

    Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, 2011

  6. [14]

    Identifying key algorithm parameters and instance features using forward selection

    Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Identifying key algorithm parameters and instance features using forward selection. In Learning and Intelligent Optimization: 7th International Conference, 2013

  7. [15]

    Automated Machine Learning: Methods, Systems, Challenges

    Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer Nature, 2019

  8. [16]

    bo: Augmenting acquisition functions with user beliefs for bayesian optimization

    Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, and Luigi Nardi. bo: Augmenting acquisition functions with user beliefs for bayesian optimization. In International Conference on Learning Representations (ICLR), 2022

  9. [17]

    Efficient global optimization of expensive black-box functions

    Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13: 0 455--492, 1998

  10. [18]

    Tabular benchmarks for joint architecture and hyperparameter optimization, 2019

    Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization, 2019

  11. [19]

    Ron Kohavi and George H. John. Automatic parameter selection by minimizing estimated error. In International Conference on Machine Learning (ICML), 1995

  12. [20]

    Smac3: A versatile bayesian optimization package for hyperparameter optimization

    Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23 0 (...

  13. [21]

    Position: A call to action for a human-centered automl paradigm

    Marius Lindauer, Florian Karl, Anne Klier, Julia Moosbauer, Alexander Tornede, Andreas M \" u ller, Frank Hutter, Matthias Feurer, and Bernd Bischl. Position: A call to action for a human-centered automl paradigm. In International Conference on Machine Learning (ICML), 2024

  14. [22]

    Priorband: Practical hyperparameter optimization in the age of deep learning

    Neeratyoy Mallik, Eddie Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, and Frank Hutter. Priorband: Practical hyperparameter optimization in the age of deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

  15. [23]

    J. Mockus. On the bayes methods for seeking the extremal point. IFAC Proceedings Volumes, 8 0 (1, Part 1): 0 428--431, 1975

  16. [24]

    Mixed sum-product networks: A deep architecture for hybrid domains

    Alejandro Molina, Antonio Vergari, Nicola Di Mauro, Sriraam Natarajan, Floriana Esposito, and Kristian Kersting. Mixed sum-product networks: A deep architecture for hybrid domains. In AAAI Conference on Artificial Intelligence, 2018

  17. [25]

    Domingos

    Robert Peharz, Sebastian Tschiatschek, Franz Pernkopf, and Pedro M. Domingos. On theoretical properties of sum-product networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2015

  18. [26]

    Scalable hyperparameter transfer learning

    Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

  19. [27]

    Jomaa, Martin Wistuba, and Josif Grabocka

    Sebastian Pineda - Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large-scale reproducible benchmark for black-box HPO based on openml. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1,, 2021

  20. [28]

    Tunability: Importance of hyperparameters of machine learning algorithms

    Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyperparameters of machine learning algorithms. The Journal of Machine Learning Research, 20 0 (1): 0 1934--1965, 2019

  21. [29]

    Incorporating expert prior in bayesian optimisation via space warping

    Anil Ramachandran, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Incorporating expert prior in bayesian optimisation via space warping. Knowledge-Based Systems, 195: 0 105663, 2020

  22. [30]

    Gaussian Processes for Machine Learning

    Carl Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning. MIT Press, 2006

  23. [31]

    A quantile-based approach for hyperparameter transfer learning

    David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. In International Conference on Machine Learning (ICML), pages 8438--8448, 2020

  24. [32]

    Adams, and Nando de Freitas

    Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104 0 (1): 0 148--175, 2016

  25. [33]

    Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NIPS), 2012

  26. [34]

    Artur L. F. Souza, Luigi Nardi, Leonardo B. Oliveira, Kunle Olukotun, Marius Lindauer, and Frank Hutter. Bayesian optimization with a prior for the optimum. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2021

  27. [35]

    Meta-learning: A survey

    Joaquin Vanschoren. Meta-learning: A survey. arXiv preprint arXiv:1810.03548, 2018

  28. [36]

    Recent advances in bayesian optimization

    Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent advances in bayesian optimization. ACM Computing Surveys, 55 0 (13s): 0 1--36, 2022

  29. [37]

    Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani

    Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization. Journal of Machine Learning Research, 25, 2024

  30. [38]

    Exploring the loss landscape in neural architecture search

    Colin White, Sam Nolen, and Yash Savani. Exploring the loss landscape in neural architecture search. In Conference on Uncertainty in Artificial Intelligence (UAI), 2020

  31. [39]

    Maximizing acquisition functions for bayesian optimization

    James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for bayesian optimization. Advances in neural information processing systems, 31, 2018

  32. [40]

    Sequential model-free hyperparameter tuning

    Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Sequential model-free hyperparameter tuning. In 2015 IEEE International Conference on Data Mining, pages 1033--1038, 2015

  33. [41]

    Nas-bench-101: Towards reproducible neural architecture search

    Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning (ICML), 2019

  34. [42]

    Dani Yogatama and Gideon S. Mann. Efficient transfer learning method for automatic hyperparameter tuning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2014

  35. [43]

    Collapsed variational inference for sum-product networks

    Han Zhao, Tameem Adel, Geoff Gordon, and Brandon Amos. Collapsed variational inference for sum-product networks. In International Conference on Machine Learning (ICML), volume 48, pages 1310--1318, 2016

  36. [44]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  37. [45]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  38. [46]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.