{"id":"a157c42b-57cb-45dd-b13d-61f093cd84ed","arxiv_id":"2601.20800","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CondPED-ANOVA estimates hyperparameter importance in conditional search spaces by keeping only the within-regime part of the variance, so inactive parameters do not inherit the gating parameter's importance.","lead":"The paper introduces condPED-ANOVA, a method for measuring which hyperparameters matter most in automated machine learning when some hyperparameters exist or change their allowed values only under certain settings of others. It shows that a standard importance measure secretly credits the switching parameter's effect to the switched parameters, and offers a corrected estimator.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"","rationale":"","tokens_in":31444,"tokens_out":9242,"duration_ms":95768,"concrete_test":"","verdict_should_be":"CONDITIONAL","load_bearing_attack":"","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes conditional PED-ANOVA (condPED-ANOVA), an HPI estimator for conditional hyperparameter search spaces. Standard local HPI is decomposed into within-regime and inter-regime variance; the authors argue that the inter-regime term is induced by gating hyperparameters and should not be attributed to conditionally active or domain-shifted hyperparameters. They define conditional local HPI as the normalized within-regime variance, derive a closed-form estimator (Eq. 11) that aggregates per-regime Pearson divergences, and prove the derivation in Appendix B.3. Experiments on synthetic objectives and the YAHPO Gym rbv2_super scenario show that naive filtering/imputation/expansion baselines misbehave, while condPED-ANOVA gives plausible importances and is faster than the baselines.","tokens_in":31362,"tokens_out":9614,"duration_ms":105329,"significance":"The contribution is timely and practically relevant: AutoML and HPO systems routinely face conditional spaces, and no principled HPI estimator has been available for them. The closed-form derivation in Appendix B.3 is clean, Theorem F.1 provides a falsifiable leakage prediction that is confirmed in Figure 3, the code is released, and the empirical comparison includes a wide range of baselines. If the gap between the objective-based Definition 4.1 and the indicator-based Theorem 4.2 is closed, the paper would be a solid and directly applicable contribution.","major_comments":[{"comment":"The definition of conditional local HPI (Definition 4.1, Eq. (9)) uses the objective-based local marginal mean g^(d)_γ(I,Z). Theorem 4.2 then states that the within-regime variance is given by Eq. (11) with D_PE of the indicator b_γ' = 1{x∈X_γ'}. The proof in Appendix B.3 settles Eq. (11) for the indicator only; no statement or argument shows that this indicator variance equals or approximates the objective variance in Eq. (9). Since all experiments use Eq. (11) as the implemented estimator of Definition 4.1, this is a load-bearing gap. If the indicator approximation of PED-ANOVA (Section 3.3) is inherited, it must be stated explicitly and, ideally, re-proved under the within-regime decomposition; otherwise, please redefine conditional local HPI as an indicator-based quantity.","section":"Definition 4.1 vs. Theorem 4.2 (§4.2–§4.3)"},{"comment":"The leakage theorem is restricted to a single gating coordinate with I^(d)=φ(x^(c)) and no direct gating effect (Eq. (37)). For general conditional spaces, where a parent also affects performance within the regime or regimes depend on multiple coordinates, the equality v^(d)_γ = v^(d)_γ,within + v^(c)_γ need not hold. Section 4.2 and the conclusion nonetheless present the leakage as a general failure of standard local HPI. Please state the assumptions under which leakage applies or provide a more general result; otherwise the motivation is broader than the theory.","section":"Theorem F.1 (App. F.1–F.2)"}],"minor_comments":[{"comment":"Algorithm 1 does not list the regime mapping r^(d) as input or explain how D^(d)_{γ,i} is constructed from it; add this to make the procedure reproducible.","section":"Algorithm 1"},{"comment":"The symbol v^(d)_γ in Eq. (5) is used for the indicator-based local variance without a distinguishing superscript, while Section 3.2 defines it for the objective; add a distinguishing notation or an explicit approximation statement.","section":"Eq. (5) and Section 3.3"},{"comment":"The proof sets κ=|D_γ'|/|D_γ|=γ'/γ, ignoring the floor effect of the empirical quantile. State this as an approximation (or use the exact discrete factor).","section":"App. B.3"},{"comment":"The title and abstract mention 'dynamic' search spaces, but the paper treats static conditional structure; consider clarifying the terminology.","section":"Title/Abstract"},{"comment":"The real-world results are qualitative; a brief statement that no ground-truth importance is available would set appropriate expectations.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's scope and the central method is well motivated. The main formal issue is the missing bridge between the objective-based definition and the indicator-based estimator; this is fixable by a proof or by reframing the definition. The leakage theorem's scope should also be tightened. No concerns about citation practice or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution. The leakage observation (Theorem F.1) — that standard local HPI gives conditioned parameters the gating variable's variance additively — is new, and the paper shows the effect empirically. The proposed fix, within-regime variance, is natural, and the closed-form estimator in Eq. (11) is correctly derived in Appendix B.3 for the indicator objective. When K=1 it reduces to PED-ANOVA. Code is public and the experiments are reproducible. That part deserves a serious referee.\n\nThe soft spot is the one the reader flagged: Definition 4.1 defines conditional local HPI through the objective's local marginal mean (Eq. 9), but Theorem 4.2 and Algorithm 1 estimate the within-regime variance of the indicator b_{γ'}. The paper says normalizing the theorem's quantity gives the conditional local HPI, but the bridge is not proved. This is inherited from PED-ANOVA's indicator approximation, but it is not re-established for the conditional decomposition. It may be fine, but the authors should either prove the approximation under stated conditions or redefine the estimator as the indicator-based quantity and soften the definitional claim. The abstract also overstates the leakage theorem slightly: Eq. (38) holds under the gating assumptions in Appendix F.1, not for every conditional space. That is a one-sentence fix.\n\nMinor stuff: the baselines are deliberately naive (filtering, imputation, expansion), which is the point, but a dummy-coded f-ANOVA baseline would make the comparison stronger. The main-text real-world evaluation is one YAHPO instance with the rest in an appendix; acceptable but thin. The runtime comparison is fine.\n\nThe central argument holds up in my reading. The estimator is well-defined, the math is clean, and the empirical behavior matches the leakage theorem. With the definitional gap closed or honestly scoped, this is a solid contribution to AutoML and HPO interpretation. Recommend peer review with a request to address the missing bridge.","headline":"A principled fix for conditional HPI with a real leakage theorem, but the estimator's link to its own definition is asserted rather than proved.","tokens_in":31929,"tokens_out":2062,"would_cite":true,"duration_ms":22634,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard HPI analysis leaks the gating variable's variance into the hyperparameters it activates; the paper proves the leak, redefines local HPI as within-regime variance, and gives a closed-form estimator.","keywords":["hyperparameter importance","conditional search space","PED-ANOVA","functional ANOVA","Pearson divergence","variance decomposition","local importance","automated machine learning"],"falsifier":"Construct a synthetic conditional objective with a flat plateau: inside a regime, the objective is constant over all configurations that qualify for the top-γ' set but varies sharply just below the threshold, so the elite-set indicator carries no information about the parameter while the objective's local marginal mean does. Compute the paper's closed-form estimator (Eq 11) and a brute-force Monte Carlo evaluation of the definition (Eq 9) on the same top-γ samples. If the two diverge materially — near-zero from the estimator, substantial from the direct computation — the bridge fails and the s","tokens_in":31220,"feed_emoji":"🎛️","tokens_out":16290,"duration_ms":158647,"temperature":0.7,"pith_summary":"This paper takes on hyperparameter importance (HPI) analysis in conditional search spaces — spaces where a hyperparameter exists, or changes domain, only when another 'gating' hyperparameter takes a particular value, as in model selection or neural architecture search. It claims the standard local HPI is systematically wrong there: by the law of total variance, the importance score of a conditioned hyperparameter contains the gating variable's variance as an additive term, so inactive variables appear important and the gate itself is underrated. The proposed fix, conditional PED-ANOVA, defines importance using only the within-regime part of the variance and estimates it in closed form from one-dimensional densities, retaining the speed of the original PED-ANOVA. If the paper is right, common workarounds — filtering inactive samples, imputing defaults, expanding domains — produce misleading rankings in conditional spaces, and the corrected estimator gives the first principled alternative.","feed_headline":"Gating effects leak into hyperparameter importance scores","feed_subtitle":"Within-regime variance, estimated in closed form, recovers true importance of conditionally active hyperparameters.","key_machinery":"The load-bearing object is the regime: each hyperparameter is partitioned into K^(d) disjoint regimes, each with its own domain (a singleton {⊥} when the parameter is inactive), assembled into an extended one-dimensional domain S^(d) = ⊔_i {i} × Z_i^(d) so that regime identity and value live in one object. Inside each regime the estimator forms the Pearson (χ²) divergence between the one-dimensional densities of the top-γ' and top-γ elite sets and aggregates over regimes with weight (α_i^(d))²/β_i^(d), the squared regime frequency in the tight set divided by its frequency in the loose set. The within-regime restriction is what excludes the inter-regime term that carries the gate's leaked var","core_discovery":"Central claim: for a conditioned hyperparameter x^(d), the standard local marginal variance splits into within-regime and inter-regime terms, and the inter-regime term equals the local marginal variance of the gating variable that selects the regime (Theorem F.1: v_γ^(d) = v_γ,within^(d) + v_γ^(c)). Because an upstream variable fixes the regime, that variance belongs to the gate, not to x^(d); a parameter inactive in a regime inherits the gate's whole importance. The fix redefines conditional local HPI as within-regime variance alone (Definition 4.1); Theorem 4.2 supplies a closed-form estimator — a regime-weighted sum of Pearson divergences between top-γ' and top-γ one-dimensional densities","pith_inferences":["An implication the paper leaves implicit: any existing HPI workflow that currently discards or imputes conditional parameters can re-run the same evaluation data through the within-regime estimator, and the gap between the old and new rankings is, by the paper's leakage theorem, a direct measurement of how much gating contamination was present.","Testable extension: the regime function is supplied by the search-space designer, but nothing in the math requires it; clustering the top-γ set to learn regimes automatically would turn the estimator into a general tool for local importance in arbitrary stratified subspaces, not just explicitly declared conditional hierarchies.","Portability: the within-versus-inter variance split is a generic statement about stratified input spaces, so the same correction applies to sensitivity analysis over mixtures, missing-data strata, or any setting where a covariate defines subpopulations — not only hyperparameter optimization.","Open validation the paper leaves for follow-up: because the closed form estimates the variance of the elite-set indicator rather than of the objective's local mean directly, comparing it against a brute-force evaluation of the definition on objectives with flat plateaus inside the top region would show how far the indicator proxy stretches."],"forward_implications":["Filtering, imputation, and domain expansion — the standard ways existing HPI tools handle conditional spaces — are shown to be systematically misleading: they flatten importance across active and inactive parameters and suppress the gating variable that actually decides which branch can win.","Inactive conditional hyperparameters get exactly zero importance under the corrected estimator, since their regimes are degenerate and contribute zero divergence; the uncorrected score, by contrast, hands them the gating variable's variance.","The corrected importance is target-aware: as the performance quantile γ' tightens, importance shifts from the gate (which decides the winning branch) to the active parameter that must be tuned within that branch, across disjoint, overlapping, nested, and domain-shifting structures.","The estimator is closed-form and one-dimensional, so it keeps PED-ANOVA's speed; runtime comparisons show it faster than surrogate-based baselines at every evaluated sample count.","The definition strictly generalizes PED-ANOVA — a single regime recovers the original local HPI exactly — so the method is an extension rather than a competing estimator."],"fun_headline_variants":["Gating bias in HPI fixed by within-regime variance","Conditional HPI: attribute variance to the gate, not the gated","New closed-form HPI for conditional hyperparameters","Separating gating from importance in HPI","Within-regime variance recovers true conditional HPI"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The estimator measures how much a hyperparameter's value shifts the chance of landing in the top-performing set, and the paper treats that as measuring how much it changes the objective inside that set — an equivalence inherited from PED-ANOVA and not re-proven for the conditional case.","fun_headline_variants_meta":{"raw":{"variants":["Gating bias in HPI fixed by within-regime variance","Conditional HPI: attribute variance to the gate, not the gated","New closed-form HPI for conditional hyperparameters","Separating gating from importance in HPI","Within-regime variance recovers true conditional HPI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1144,"prompt_tokens":711,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":455,"tokens_out":433,"duration_ms":4796,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:11:23.266089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic conditional objective with a flat plateau: inside a regime, the objective is constant over all configurations that qualify for the top-γ' set but varies sharply just below the threshold, so the elite-set indicator carries no information about the parameter while the objective's local marginal mean does. Compute the paper's closed-form estimator (Eq 11) and a brute-force Monte Carlo evaluation of the definition (Eq 9) on the same top-γ samples. If the two diverge materially — near-zero from the estimator, substantial from the direct computation — the bridge fails and the s","supporting_citations":[],"review_version":1}