{"id":"f88fbc58-0b5c-444f-a481-7a03a6717a09","arxiv_id":"2505.17804","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A Bayesian optimization method using probabilistic circuits whose sampling-based selection policy conditions on the best score and optional user priors for interactive hyperparameter tuning.","lead":"Researchers introduce IBO-HPC, a way to tune machine learning hyperparameters that lets a human give hints partway through the search, such as 'set network depth to 16.' It uses probabilistic circuits to sample new settings conditioned on the best score so far and reports faster convergence when the hints are helpful.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feedback-adherence guarantee fails as stated: Algorithm 1's Bernoulli switch means the policy samples q(Hhat) only with probability rho, so Proposition 1's equality cannot hold; the 'ensures' claim is conditional at best.","rationale":"The reader's weakest_assumption was surrogate fidelity, which is a real risk for any BO method. I agree with that concern but consider the Proposition 1 mismatch more load-bearing because it attacks the paper's stated contribution 'ensures user beliefs are reflected accurately' using only the algorithm's own pseudocode. The empirical results may still be valid, and the method may work well in practice, but the formal guarantee needs to be weakened or the definition/proof revised. This is consistent with the reader's CONDITIONAL recommendation; I would not move to REJECT because the issue is fixable by qualifying the claim and the experiments are extensive and reproducible. Hence UNCHANGED relative to the reader's verdict.","tokens_in":32906,"tokens_out":6033,"duration_ms":73679,"concrete_test":"Fix a synthetic D and f*, set rho=0.5, gamma=1, and provide a fixed two-point prior q(Hhat). Run Algorithm 1's selection loop (Lines 6-16) 10,000 times without retraining, recording the sampled Hhat. Compute the empirical frequency of Hhat and a chi-square goodness-of-fit against q. If the empirical marginal is well described by 0.5*q + 0.5*s(Hhat|F=f*) and rejects q as the distribution, the unconditional policy is not feedback-adhering. Also re-derive Prop. 1 with the Bernoulli variable included; the Def. 2 equality then contains the extra (1-rho) term.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim (Prop. 1, App. D.1) is that IBO-HPC's selection policy is feedback-adhering: the induced marginal over user-specified hyperparameters Hhat equals user prior q(Hhat). But Algorithm 1 Line 6 draws b~Ber(rho), and the prior-conditioned branch runs only when b=1 (Line 7). When b=0, Line 16 samples from s(H|F=f*) without conditioning on Hhat. The actual unconditional policy therefore induces rho*q(Hhat)+(1-rho)*s(Hhat|F=f*) (up to finite N/B), not q(Hhat). The proof in D.1 never models b; it treats the decay mechanism as a constant and argues about the prior branch as if it were the whole policy. It also uses an N->infty, L->infty approximation while Algorithm 1 makes one uniform draw from N candidates. Since Prop. 1 itself assumes s(Hhat|F=f*) != q(Hhat), the mixture differs from q whenever that assumption holds. Thus the headline 'ensures user beliefs are reflected accurately in the selection policy' is not supported by the stated theorem; it holds only conditionally on b=1 and in the limit N->infty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IBO-HPC, a Bayesian optimization method for hyperparameter optimization that replaces the usual acquisition function with conditional sampling from a probabilistic circuit (PC). A PC is fit to evaluated configurations and their scores; new candidates are sampled from the PC conditioned on the best score observed so far. User knowledge about subsets of hyperparameters can be supplied at any time as a prior distribution, and the method also includes a Bernoulli-decay mechanism intended to let the optimizer recover from misleading feedback. The authors define a formal notion of feedback-adhering interactive policies, claim that IBO-HPC satisfies it (Proposition 1), claim global optimization in the sense of simple regret (Proposition 2), and derive a per-iteration expected-improvement lower bound (Proposition 3). They then report extensive experiments on JAHS, NAS-Bench-101/201, HPO-B, PD1, and FCNet with 500 seeds in the main runs, comparing against SMAC, BO with RF/TPE, πBO, BOPrO, Priorband, random search, and local search. The empirical section is substantial and shows that IBO-HPC is competitive without user knowledge and often faster or better with beneficial user knowledge, with median speed-ups of 2–10×.","tokens_in":33162,"tokens_out":4987,"duration_ms":56641,"significance":"If the theoretical claims were sound, the paper would make a useful contribution to interactive HPO: it offers an acquisition-function-free selection policy based on tractable conditional sampling, allows priors to be supplied mid-run, and provides a natural recovery mechanism from harmful feedback. The empirical study is a genuine strength: the use of standard benchmarks, the large number of seeds, the inclusion of distributional and point-mass interactions, the recovery experiments, and the reporting of Wilcoxon significance tests go well beyond what is typical. Code and raw logs are also promised, which supports reproducibility. However, several of the paper's central theoretical assertions are not supported as stated, and because the 'accurate reflection of user beliefs' claim is one of the paper's headline contributions, the theory needs substantial repair before the claims can be accepted.","major_comments":[{"comment":"Proposition 1 is not supported by the proof because Algorithm 1 only uses the user prior when the Bernoulli variable b equals 1. In Algorithm 1, Line 6 draws b~Ber(ρ); the prior-conditioned branch runs only when b=1 (Line 7), while Line 16 samples from s(H|F=f*) when b=0. The actual policy therefore induces the mixture ρ·q(Ĥ)+(1−ρ)·s(Ĥ|F=f*) over the user-specified hyperparameters, not q(Ĥ) as required by Definition 2. Since Proposition 1 explicitly assumes s(Ĥ|F=f*)≠q(Ĥ), the mixture cannot equal q. The proof in App. D.1 treats the decay mechanism as a constant and argues about the prior branch as if it were the whole policy; it never models the Bernoulli switch. The additional appeal to N→∞ and L→∞ also does not repair the issue, because Algorithm 1 makes one uniform draw from a finite set of N candidates and the Bernoulli mixture remains regardless of N. The theorem and its proof, or the algorithm and the definition of feedback adherence, need to be revised so that the stated guarantee matches what the selection policy actually computes.","section":"Section 3.3 / App. D.1, Proposition 1"},{"comment":"The assertion that 'as long as the ground truth distribution p ... is representable by an SPN, we can recover p with arbitrarily small error with iterations T→∞' is unsupported and load-bearing. LearnSPN is stated to locally maximize likelihood on the current dataset, but local likelihood maximization on finite data does not imply consistency or arbitrarily small approximation error of the ground-truth joint distribution. No theorem, rate, or set of identifiability conditions is provided for this recovery claim. Since the quality of every downstream statement about candidate selection and feedback adherence depends on the surrogate s being calibrated, this gap must be addressed: either prove a suitable consistency result under explicit assumptions or clearly weaken the claim to an assumption about the surrogate's quality.","section":"App. D.4"},{"comment":"The proof of Proposition 2 is not valid for continuous hyperparameter spaces. The argument is that positivity of s(H=θ|F=f*) for all θ implies that with T→∞ the probability of sampling the global optimum θ* tends to 1. For a continuous distribution, the probability of sampling a specific point θ* is zero, even if the density is positive there. Moreover, the surrogate is refit as data accumulates, so the sampling distribution is not an i.i.d. sequence from a fixed distribution. The claimed guarantee that simple regret reaches zero should be weakened to an ε-regret statement under additional continuity and support assumptions, or the proposition should be restricted to discrete search spaces.","section":"Section 3.3 / App. D.2, Proposition 2"},{"comment":"Equation (3) is presented as a lower bound on the 'convergence rate' of IBO-HPC, but the derivation in App. D.3 establishes at most a lower bound on a surrogate-based integral that is labeled expected improvement; it does not establish a bound on the algorithm's actual regret or convergence rate as a function of iterations. In addition, the result assumes all PC leaves are Gaussians, whereas the method described in Section 2 uses MSPNs with piecewise polynomial leaves, and the proof drops constants and the 1/2 scaling of the error function while claiming this does not affect the result. The proposition should either be reformulated as a bound on a well-defined algorithm quantity with matching assumptions, or its status as a convergence-rate guarantee should be substantially qualified.","section":"Section 3.3 / App. D.3, Proposition 3"}],"minor_comments":[{"comment":"The displayed conditional density in Eq. (1) is written as s(H'|Ĥ,F) = s(H,F)/∫_{H'} s(H,F), which is not the standard definition of conditioning; the numerator should be the joint density over the appropriate variables, and the denominator should be the marginal over the variables being conditioned upon. Please correct the notation.","section":"Section 3.2, Eq. (1)"},{"comment":"Line 17 adds (θ', f(θ')) to the dataset, but the configuration selected for evaluation is called θ* earlier in the algorithm. The notation should be made consistent so that the selected candidate is the one evaluated and added to D.","section":"Algorithm 1, Line 17"},{"comment":"Several p-values are formatted inconsistently, e.g., '98×10−1' and '0 .9' and '0 .96'. These should be normalized to a uniform numeric format to avoid ambiguities.","section":"Appendix E.5, Tables 2–4"},{"comment":"The proposition proved in App. D.2 is labeled Proposition 4, but it corresponds to Proposition 2 in the main text. Please renumber so that the appendix labels match the main text.","section":"Appendix D.2, Proposition numbering"},{"comment":"Definition 1 frames HPO as minimization, but Algorithm 1 and the experimental section condition on f* = max f(D), which is consistent with maximizing validation accuracy. Please clarify whether the score variable is a loss to be minimized or an accuracy to be maximized, and keep the notation consistent throughout.","section":"Definition 1 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a practically useful HPO paper that reaches beyond its theoretical grasp. The main theorem, as written, is false, and the convergence result is not a convergence rate. But the empirical study is careful and the method itself is worth engaging with.\n\nWhat's new: using probabilistic circuits as a BO surrogate and selecting candidates by sampling from s(H|F=f*) with optional user priors, avoiding acquisition-function inner-loop optimization. That combination is new relative to piBO, BOPrO, and Priorband. The definition of a feedback-adhering interactive policy is a useful formalization. The experiments are extensive: 500 seeds, six benchmarks, significance tests. With helpful user hints, IBO-HPC gets 2-10x speedups and outperforms the interactive baselines in most tasks. The recovery experiment with misleading feedback is well designed.\n\nThe soft spots are in Sections 3.2-3.3. Proposition 1's proof ignores the Bernoulli switch in Algorithm 1. The policy only samples from the user prior with probability rho; otherwise it samples from s(H|F=f*). Since the proposition assumes these two distributions differ, the induced marginal over Hhat is a mixture, not q(Hhat). So the \"ensures user beliefs are reflected accurately\" claim is not supported by the stated theorem. It holds only conditional on the prior branch being active.\n\nProposition 3 is also shaky. What it calls a lower bound on the convergence rate is an expression for the surrogate's expected improvement—an identity about the fitted model, not a statement about progress toward the true optimum. There's also an algebraic error: they treat the norm g(theta)=L||theta-theta*|| as linear and move the expectation inside, which is invalid. And Proposition 2 is just the usual positivity argument, which is fine but not a real global-optimization guarantee.\n\nOne smaller issue: the abstract says \"state-of-the-art in standard HPO,\" but the paper's own Tables 4 show the no-interaction results are competitive, not dominant. That should be softened.\n\nOverall, I trust the empirical claims more than the theory. The method is sensible, the ablation study (gamma, L, quantile conditioning) is good, and the code/data links are provided. This deserves a serious referee, but the theory should be substantially revised before publication. I'd send it to review with a note to focus on Propositions 1 and 3.","headline":"Practical interactive HPO method with strong empirical work, but the main theory as stated doesn't hold—Prop 1 ignores the Bernoulli switch and Prop 3 is a surrogate identity, not a convergence rate.","tokens_in":33761,"tokens_out":3174,"would_cite":true,"duration_ms":31348,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IBO-HPC replaces acquisition-function optimization with conditional sampling from a probabilistic circuit, so user priors are reflected exactly in the next candidate.","keywords":["hyperparameter optimization","Bayesian optimization","probabilistic circuits","interactive machine learning","conditional sampling","sum-product networks","user priors","acquisition function"],"falsifier":"Run IBO-HPC on a one-dimensional objective with a known global optimum from five random initial points and measure whether samples from $s(H \\mid F = f^*)$ concentrate nearer the optimum than random search; separately, provide a Gaussian prior over one hyperparameter with decay disabled and test whether the empirical density of chosen values matches the prior---if either fails, the representability premise is falsified.","tokens_in":32627,"feed_emoji":"⚙️","tokens_out":6282,"duration_ms":48613,"temperature":0.7,"pith_summary":"This paper introduces IBO-HPC, a Bayesian optimization method for hyperparameter tuning that replaces the usual acquisition function with direct conditional sampling from a probabilistic circuit. The circuit is trained on the joint distribution of hyperparameters and evaluation scores, and new candidate configurations are drawn from the distribution conditioned on the best score observed so far, optionally with a user-provided prior over some hyperparameters. The paper argues that because candidates are generated by exact conditioning and sampling rather than by optimizing a weighted acquisition function, user beliefs are reflected in the selected configurations exactly as given. It further claims that this makes the method competitive with strong hyperparameter optimization baselines without user input and better than interactive BO baselines when user knowledge is supplied, while a decay mechanism lets it recover from misleading advice.","feed_headline":"Conditional sampling lets user priors steer HPO exactly","feed_subtitle":"A probabilistic-circuit surrogate skips the acquisition loop, matches user beliefs, and rebounds from bad advice.","key_machinery":"The load-bearing object is the probabilistic circuit (PC), a tractable graphical model---here a mixed sum-product network---that encodes a joint distribution over the hybrid hyperparameter space and the evaluation score. The selection policy is built on exact conditional sampling: after fitting the PC to the evaluated configurations, it forms $s(H' \\mid \\hat{H}, F = f^*)$ via tractable conditioning and draws candidates from this distribution (Eq. 1). A user prior $q(\\hat{H})$ is fused by sampling $N$ conditions from $q$, computing one conditional per condition, and selecting the most likely candidate from each; a Bernoulli switch with decay $\\gamma^t$ controls how long the feedback is used so the method can recover from misleading advice. Fitting is done with LearnSPN, which alternates clustering and independence tests to produce a smooth, decomposable PC.","core_discovery":"The paper's central claim is that interactive hyperparameter optimization can be made both simpler and more faithful to user intent by modeling the joint distribution over hyperparameters $H$ and evaluation score $F$ with a probabilistic circuit, and selecting each new configuration by sampling from the conditional distribution $s(H' \\mid \\hat{H}, F = f^*)$, where $f^*$ is the best score observed so far and $\\hat{H}$ is any user-supplied condition on a subset of hyperparameters. Since probabilistic circuits support exact marginalization, conditioning, and sampling, the method removes the inner-loop optimization of an acquisition function, and the selected candidates' marginal over the user-specified hyperparameters matches the user's prior $q(\\hat{H})$ in the limit of many samples (Proposition 1). The paper further shows that the policy minimizes simple regret (Proposition 2) and that its per-iteration convergence is lower-bounded by an expected-improvement term expressed through error functions over the circuit's Gaussian mixture components (Proposition 3). On benchmark tasks covering neural architecture search and continuous, discrete, and mixed HPO spaces, the authors report that without user knowledge the method is competitive with strong BO baselines, while with beneficial feedback it outperforms interactive BO baselines and reaches the same quality 2--10 times faster, and with misleading feedback it recovers.","pith_inferences":["Not pursued in the paper: the same conditional-sampling mechanism could be applied to multi-fidelity HPO by treating fidelity as an additional variable in the joint distribution, so a user could steer both the configuration and the budget.","Because the feedback-adhering property is defined formally, it could be adopted as an evaluation criterion for other interactive BO methods, turning 'the prior was not reflected' from a qualitative complaint into a measurable quantity.","A testable extension would be to set the decay $\\gamma$ automatically from the data---for example, by comparing the likelihood of recently observed scores under the user-conditional and unconditional surrogates---rather than treating it as a fixed hyperparameter.","The paper's own limitation that low leaf variance can trap the sampler suggests a diagnostic: monitor the conditional's variance during the run and warn the user when it collapses, which would also make the method more robust for non-experts."],"forward_implications":["User priors can be injected at any iteration, and the marginal distribution of the next candidate over the specified hyperparameters converges to the user's prior, so feedback acts exactly where the user intends.","Removing the acquisition-function inner loop reduces suggestion overhead; the paper reports the selection policy is faster than SMAC in 4 of 5 cases, with the gap growing in larger search spaces.","Misleading feedback is phased out geometrically via the decay factor, and the empirical runs show the method returns to the same performance as if no feedback had been given.","On the benchmark suite, beneficial feedback improves convergence by a median factor of 2--10 relative to running without feedback, and the method outperforms the interactive baselines in most compared tasks.","Because the policy is defined as a distribution over configurations, the same sampling machinery applies whether feedback is a point value or a distribution over hyperparameter values."],"supporting_citations":[{"why":"Supplies the probabilistic circuit formalism and the tractable inference, conditioning, and sampling operations that the selection policy relies on.","marker":"[Choi et al., 2020]"},{"why":"Provides mixed sum-product networks, the specific PC variant used to model hybrid continuous/discrete hyperparameter spaces.","marker":"[Molina et al., 2018]"},{"why":"Supplies LearnSPN, the structure-learning algorithm used to fit the surrogate from evaluated configurations.","marker":"[Gens and Domingos, 2013]"},{"why":"Documents the difficulty of optimizing acquisition functions, motivating the acquisition-free sampling policy.","marker":"[Wilson et al., 2018]"},{"why":"Defines $\\pi$BO, the prior-weighted acquisition baseline the paper compares against in interactive HPO.","marker":"[Hvarfner et al., 2022]"},{"why":"Defines BOPrO, the other main interactive baseline with a user prior on the optimum.","marker":"[Souza et al., 2021]"},{"why":"Provides the general BO framework and selection-policy background used to position the method.","marker":"[Shahriari et al., 2016]"},{"why":"Supplies random search with user priors as a baseline and the broader hyperparameter-search context.","marker":"[Bergstra and Bengio, 2012]"}],"fun_headline_variants":["PC-based HPO: exact conditional sampling, no acquisition loop, robust to bad advice","Probabilistic circuits make HPO interactive: conditional sampling, exact priors","Sampling from PCs reflects user beliefs exactly, no inner-loop optimization","HPO via probabilistic circuits: conditional sampling skips acquisition functions","Exact conditional sampling from PCs: interactive HPO that withstands bad advice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme rests on the learned probabilistic circuit's joint distribution over hyperparameters and scores being a faithful stand-in for the true one; with only a handful of evaluated configurations, that fidelity is not guaranteed, and a miscalibrated conditional will send sampling to the wrong regions.","fun_headline_variants_meta":{"raw":{"variants":["PC-based HPO: exact conditional sampling, no acquisition loop, robust to bad advice","Probabilistic circuits make HPO interactive: conditional sampling, exact priors","Sampling from PCs reflects user beliefs exactly, no inner-loop optimization","HPO via probabilistic circuits: conditional sampling skips acquisition functions","Exact conditional sampling from PCs: interactive HPO that withstands bad advice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2412,"prompt_tokens":995,"completion_tokens":1417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1333}},"tokens_in":611,"tokens_out":1417,"duration_ms":10062,"temperature":1.0,"reasoning_tokens":1333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:41:47.147933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run IBO-HPC on a one-dimensional objective with a known global optimum from five random initial points and measure whether samples from $s(H \\mid F = f^*)$ concentrate nearer the optimum than random search; separately, provide a Gaussian prior over one hyperparameter with decay disabled and test whether the empirical density of chosen values matches the prior---if either fails, the representability premise is falsified.","supporting_citations":[{"cited_title":"Probabilistic circuits: A unifying framework for tractable probabilistic models","cited_arxiv_id":null,"evidence_quote":"Supplies the probabilistic circuit formalism and the tractable inference, conditioning, and sampling operations that the selection policy relies on."},{"cited_title":"Mixed sum-product networks: A deep architecture for hybrid domains","cited_arxiv_id":null,"evidence_quote":"Provides mixed sum-product networks, the specific PC variant used to model hybrid continuous/discrete hyperparameter spaces."},{"cited_title":"Learning the structure of sum-product networks","cited_arxiv_id":null,"evidence_quote":"Supplies LearnSPN, the structure-learning algorithm used to fit the surrogate from evaluated configurations."},{"cited_title":"Maximizing acquisition functions for bayesian optimization","cited_arxiv_id":null,"evidence_quote":"Documents the difficulty of optimizing acquisition functions, motivating the acquisition-free sampling policy."},{"cited_title":"bo: Augmenting acquisition functions with user beliefs for bayesian optimization","cited_arxiv_id":null,"evidence_quote":"Defines $\\pi$BO, the prior-weighted acquisition baseline the paper compares against in interactive HPO."},{"cited_title":"Adams, and Nando de Freitas","cited_arxiv_id":null,"evidence_quote":"Provides the general BO framework and selection-policy background used to position the method."},{"cited_title":"Random search for hyper-parameter optimization","cited_arxiv_id":null,"evidence_quote":"Supplies random search with user priors as a baseline and the broader hyperparameter-search context."}],"review_version":1}