{"id":"55168345-9cea-42cb-9d74-ed52d5d39679","arxiv_id":"2501.04260","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AttnBO trains a single attention-based deep kernel Gaussian process on all subspaces of a conditional search space at once, instead of fitting one GP per subspace.","lead":"This paper introduces AttnBO, a Bayesian optimization method that models every subspace of a tree-structured hyperparameter space with one Gaussian process, using an attention-based encoder to embed differently shaped configurations into a shared space. The reported experiments show faster regret reduction than separate-model baselines on simulation, NAS, OpenML, and HPO-B tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical case is not yet load-bearing: encoder hyperparameters are selected on the SVM/XGBoost evaluation tasks and the AddTree simulation baseline is self-admittedly non-reproduced, so the reported gains may be artifacts; a fair held-out evaluation is required.","rationale":"The reader's emphasis on the Eq. (3) embedding omission is a legitimate representational risk, but I do not think it is the most decisive issue. The manuscript's own details already provide two concrete threats to the empirical claim that are independent of whether the latent space is well-behaved. Test-set architecture selection is an internal-validity problem: Table 2 is used to choose the encoder, and the chosen configuration is then evaluated on the same tasks that drove the choice. Since Table 2 shows ranking differences of about one rank across configurations, this is not a negligible effect. The AddTree note is even more direct: the authors state they could not reproduce the state-of-the-art baseline's published results. A comparison against a baseline one cannot reproduce is not a comparison against that method. These are not disagreements with current consensus; they are unaddressed threats to the paper's central demonstration. The architecture concern is real but secondary: if a fair evaluation still shows AttnBO winning, the embedding question can be settled by the synthetic twin-subspace test; if the fair evaluation does not show a win, the embedding question is moot. Thus the verdict remains CONDITIONAL, contingent on a fair, code-released evaluation.","tokens_in":21230,"tokens_out":13830,"duration_ms":148915,"concrete_test":"Run a single fair-evaluation protocol: hold out two of the six OpenML datasets before the Table 2 architecture selection, select the encoder on the remaining four, fix it, and evaluate AttnBO against all baselines on the held-out datasets with per-seed repetition; in parallel, rerun the Fig. 3 simulation using the official AddTree implementation with its original initialization. If AttnBO is still best on held-out datasets and the reproduced AddTree curves match the original paper, the concern is resolved; otherwise the current evidence is an artifact of test-set tuning and a weak baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical claim: AttnBO \"improves the efficacy and efficiency of BO within conditional search spaces\" relative to per-subspace and additive-tree baselines. The two pieces of evidence that carry this claim are the OpenML ranking experiments (Fig. 5) and the simulation comparison (Fig. 3). Both are compromised by details stated in the manuscript itself. (1) In \"Implementation details\", the encoder hyperparameters \"are selected based on their performance on the SVM and XGBoost tasks\"; Table 2 selects na, nb, and the MLP by the \"lowest combined average ranking across the two tasks\". The same two tasks are then reported in Fig. 5(a,b), and the composed CASH task in Fig. 5(c) inherits those components. This is test-set tuning: the advantage over baselines can come from model selection, not from the unified-response-surface mechanism. (2) In the Fig. 3 caption, the authors state that they reimplemented AddTree and \"Probably, we did not get the same results as shown in their paper due to the different number of initial points.\" AddTree is the primary state-of-the-art conditional-space baseline; a non-reproducing implementation makes the simulation superiority claim untestable. The HPO-B comparison (Fig. 6) additionally uses baseline numbers from Pineda-Arango et al. (2021) rather than a common rerun. If the OpenML gains shrink or the simulation curve flips under a fair protocol, the central claim loses its support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AttnBO, a Bayesian optimization method for conditional (tree-structured) search spaces. It represents each hyperparameter by a structure-aware embedding (identity, index, value, and father-identity), treats a configuration as a sequence of such token embeddings, and applies a Transformer-style attention encoder followed by average pooling to map variable-length configurations from different subspaces into a common latent space, on which a single deep-kernel Gaussian Process is trained with EI acquisition. The paper reports experiments on a simulation function, a NAS space, three OpenML tasks (SVM, XGBoost, and a combined CASH space), and the HPO-B benchmark, claiming improved regret and sample efficiency over per-subspace and additive-tree baselines.","tokens_in":21468,"tokens_out":6671,"duration_ms":64852,"significance":"If the empirical claims hold, the unified-surrogate approach is a meaningful step beyond separate per-subspace GPs and additive-tree kernels, with an explicit mechanism for sharing observations across subspaces and a batch-query capability. The method is clearly described, the algorithm is self-contained, and the ablations on embedding design and encoder architecture (Fig. 8 and Table 2) are useful diagnostics. The per-search-space HPO-B results in the supplementary material are a further strength. However, the evaluation protocol contains load-bearing weaknesses (see major comments), so the significance of the empirical claims should be assessed after those are addressed.","major_comments":[{"comment":"The encoder hyperparameters (na, nb, and the presence of the MLP) are selected based on the combined average ranking on the SVM and XGBoost tasks, and these same two tasks appear as the first two rows of the main evaluation in Fig. 5(a,b) and as components of the CASH task in Fig. 5(c). This constitutes test-set tuning: the reported gains on these tasks may reflect model selection rather than the proposed unified-response-surface mechanism. Please select the architecture on a held-out set of tasks or via nested cross-validation, and re-report the main results for the configuration chosen without access to the evaluation tasks.","section":"Implementation details; Table 2"},{"comment":"The caption of Fig. 3 states, 'Probably, we did not get the same results as shown in their paper due to the different number of initial points.' AddTree is the primary state-of-the-art baseline for conditional BO, and the simulation comparison is the main controlled experiment supporting the efficacy claim. A reimplementation that does not reproduce the original AddTree results makes the reported superiority over AddTree untestable. Please use the official AddTree implementation and verify it reproduces the published results under the exact evaluation protocol, or provide a corrected comparison with a documented protocol if full reproduction is not possible.","section":"Fig. 3 caption (Simulation Function)"},{"comment":"The HPO-B comparison in Fig. 6 uses baseline numbers 'as reported in (Pineda-Arango et al. 2021)' rather than a common rerun under the same protocol. Because AttnBO uses its own initialization, fine-tuning schedule, and acquisition implementation, differences in regret may reflect evaluation settings rather than algorithmic merit. Please rerun the baselines under a shared protocol, or at minimum verify that the cited numbers are directly comparable (e.g., same number of trials, same initialization, same metric, same data splits).","section":"HPO-B Benchmark (Fig. 6)"},{"comment":"The structure-aware embedding in Eq. (3) encodes only the value, identity, index, and immediate father's identity of each hyperparameter; it does not encode the father's value or the full ancestor path. In deep trees such as the NAS space (four or more levels), two configurations from different subspaces can have identical token-level embeddings for a node while differing in higher-level ancestry. The paper does not provide evidence that the attention mechanism recovers this ancestry context (e.g., via a latent-space visualization or an ablation with full ancestor information). Without such evidence, the claim that the embedding preserves structural information is not fully supported, and the NAS results may be sensitive to this representational choice. Please add a diagnostic analysis or an ablation with complete ancestor paths.","section":"Structure-aware Embeddings (Eq. 3)"}],"minor_comments":[{"comment":"Several equations contain notation errors, e.g., 'i⊂1,...,n' should be 'i=1,...,n', and Eq. (6) has an unmatched parenthesis. Please proofread the mathematical notation.","section":"Notation throughout"},{"comment":"The legend in Fig. 6 uses 'AttnBOWS' and 'AttnBO_WS' inconsistently; these should be unified.","section":"Fig. 6 legend"},{"comment":"The sentence 'The parameters of the encoder are selected based on their performance on the SVM and XGBoost tasks' should clarify whether this selection is done once on a validation split or on the same tasks that are later reported; as written, it invites the test-set tuning concern above.","section":"Experiments; Implementation details"},{"comment":"The ablation in Fig. 8 is reported only on SVM and XGBoost tasks, the same tasks used for architecture selection; reporting the ablation on the CASH or NAS tasks would make the conclusion more robust.","section":"Ablation Study"},{"comment":"The statement 'we don't consider these methods as our competitors' for VAE-based latent-space BO methods (Kusner et al.; Lu et al.; Tripp et al.; Grosnit et al.; Maus et al.) is acceptable, but a brief discussion of why they do not handle mixed categorical/numerical conditional spaces would help readers.","section":"Related work"},{"comment":"There is a formatting error in the author block: 'figure1 wxx@sjtu.edu.cn' should be the email address 'figure1wxx@sjtu.edu.cn' or similar. Please correct this in the camera-ready version.","section":"Author affiliations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not state whether code will be released; given the AddTree reproduction issue and the use of external baseline numbers, code and data availability would substantially strengthen reproducibility. The paper is within scope for a machine learning / AutoML venue, but the evaluation protocol needs to be tightened before the empirical claims can be taken as established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read of the AttnBO paper. The core idea is worth a look: instead of per-subspace GPs or additive tree kernels, they encode each hyperparameter with a structure-aware embedding (identity, index, value, father identity), run a transformer encoder over the token sequence, average-pool, and fit one deep-kernel GP on the resulting latent space. That specific combination is new relative to AddTree, Bandits-BO, and the VAE-based structured BO work. If it works, it gives AutoML a single surrogate that shares data across all branches of a conditional space, and the HPO-B warm-starting experiment is a sensible test of that. The ablations support the embedding design. No circularity in the surrogate fitting itself; the GP is trained by marginal likelihood and evaluated on out-of-sample regret.\n\nThe soft spots are mostly in the experiments, and they are real. The encoder architecture is selected on the SVM and XGBoost tasks that later appear in the main evaluation (Table 2, Fig. 5). That is test-set tuning, and it means part of the reported gain could come from model selection rather than the unified-response-surface mechanism. The paper does say the architecture choice had minimal impact on accuracy, and the table supports that the differences are modest, but without a separate validation task or nested selection we can't rule it out. The simulation comparison with AddTree is self-admittedly not reproduced ('Probably, we did not get the same results...'), which undermines the Fig. 3 superiority claim. HPO-B uses baseline numbers from the benchmark paper rather than a common rerun, and there are no error bars or significance tests anywhere. No code released.\n\nNone of this kills the idea. The central mechanism is plausible and the failures are addressable. But the empirical case, as it stands, is not load-bearing. A fair held-out evaluation is needed: pick the architecture on one set of tasks, evaluate on others; rerun all baselines under the same protocol; report variance; release code.\n\nWho is this for? Someone working on BO for conditional or tree-structured spaces, or on deep kernel surrogates. It's a reasonable contribution to the subfield, but only after the evidence is cleaned up. I'd send it to review, but I'd expect major revision. Not something I'd cite yet.","headline":"AttnBO's unified attention-based deep GP is a plausible and somewhat novel idea, but the empirical evidence is weakened by test-set architecture selection and an unreproduced AddTree baseline.","tokens_in":22132,"tokens_out":2633,"would_cite":false,"duration_ms":25025,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single Gaussian Process with a structure-aware, attention-based deep kernel can model the response surfaces of all subspaces in a conditional search space.","keywords":["Bayesian optimization","conditional search space","Gaussian process","deep kernel learning","self-attention","hyperparameter optimization","neural architecture search","transfer learning"],"falsifier":"Construct a synthetic conditional space with two deep branches in which the same hyperparameter identity appears under two different ancestors, and make the optimal value of that hyperparameter's child opposite in the two branches. Train AttnBO with the same budget used on the simulation function and check whether its posterior distinguishes the two branches; if it cannot place higher mass near both optima while keeping uncertainty low between them, the embedding lacks the ancestry information the unified model depends on.","tokens_in":20946,"feed_emoji":"🎯","tokens_out":8313,"duration_ms":74796,"temperature":0.7,"pith_summary":"Bayesian optimization normally assumes a flat search space, but real AutoML problems such as algorithm selection and neural architecture search contain conditional dependencies that split the search space into subspaces with different structures and dimensionalities. This paper tries to establish that a single Gaussian Process can model the response surfaces of all these subspaces at once, rather than fitting one GP per subspace or adding kernels along a tree. The route is to represent each hyperparameter as a token carrying its identity, value, index, and parent's identity, and to use a self-attention encoder to project every configuration into one shared latent space where a standard GP applies. If the claim holds, the practical payoff is sample efficiency: observations from any subspace strengthen the surrogate everywhere, and the optimizer can propose batch queries across subspaces in one iteration. The paper's evidence is lower regret on a simulation function, better accuracy or ranking on NAS and OpenML tasks, and competitive warm-started performance on the HPO-B benchmark.","feed_headline":"One GP models every subspace in a conditional search space","feed_subtitle":"Structure-aware embeddings and self-attention let one surrogate share data across all subspaces, saving evaluations.","key_machinery":"The load-bearing object is the structure-aware hyperparameter embedding of Eq. (3): $\\mathrm{emb}(p) = \\mathrm{concat}(\\mathrm{id\\_emb}(p), \\mathrm{idx\\_emb}(p), \\mathrm{value\\_emb}(p), \\mathrm{id\\_emb}(p^{\\uparrow}))$, which encodes each hyperparameter's own identity, its index, its value, and its parent's identity. This tokenization lets a configuration be treated as a sequence, and the attention-based encoder models pairwise hyperparameter relationships before average pooling maps all sequences into one continuous latent space; a standard GP kernel is applied there. The machinery works because deep kernel learning lets all parameters, including the embedding and encoder, be learned end-to-end from the marginal likelihood, so the surrogate adapts the latent space to the observed response surface.","core_discovery":"AttnBO's central move is to replace the collection of per-subspace surrogates with one deep-kernel GP over a learned latent space. Each hyperparameter in a configuration is embedded by concatenating four trainable pieces: its own identity embedding, its index within vector-valued hyperparameters, a linear projection of its value, and the identity embedding of its immediate parent node. A transformer-style encoder applies self-attention over these token embeddings to capture relationships among hyperparameters, then average pooling compresses sequences of any length into a fixed-dimensional vector. This vector becomes the input to a Matérn 5/2 kernel inside the deep kernel learning framework, and the embedding, encoder, and kernel parameters are trained jointly by maximizing the log marginal likelihood. The resulting posterior can be queried for any configuration in any subspace, and expected improvement is optimized per subspace to produce either the single best query or a parallel batch.","pith_inferences":["The simulation section notes that the authors did not reproduce AddTree's published curve and attribute the gap to different initial-point counts; that makes the simulation comparison a same-setup comparison rather than an exact replication.","The paper's embedding records only the immediate parent, not the full ancestor path or the parent's value; in trees four or more levels deep, the attention encoder must reconstruct ancestry from the whole sequence, and testing an embedding augmented with ancestor-path tokens would isolate whether this reconstruction is what makes the method work.","A single smooth GP over a shared latent space assumes that subspaces are related through the learned geometry; if the true response surfaces are sharply discontinuous across subspaces, a unified model could hurt, and a direct check is to measure held-out predictive log-likelihood per subspace against separate GPs.","Because the framework only needs a parent pointer, it should extend to graphs or multi-tree combinations, and to new search spaces at meta-test time by adding a new root identity; none of these extensions is tested in the paper."],"forward_implications":["Observations from every subspace are pooled into one GP, so the surrogate improves uniformly rather than only in the subspace where a query was made.","Hyperparameters that are semantically related but live in different subspaces, such as gamma under different SVM kernels, are linked through shared attention weights and a common latent space.","Optimizing the acquisition function per subspace yields a batch of queries in one BO iteration, allowing parallel evaluation without a separate batching strategy.","Pre-training on HPO-B provides a warm start that reaches competitive regret in few trials, indicating the learned representation transfers across search spaces."],"supporting_citations":[{"why":"Provides the AddTree covariance-function baseline, the simulation-function setup, and the tree-structured formulation this work compares against.","marker":"Ma and Blaschko (2020b)"},{"why":"Defines the decomposition of a conditional search space into flat subspaces and the semi-parametric GP approach that AttnBO generalizes.","marker":"Jenatton et al. (2017)"},{"why":"Supplies the deep kernel learning framework used to train the GP with the attention-based feature extractor.","marker":"Wilson et al. (2016)"},{"why":"Introduces the self-attention mechanism used as the deep feature extractor to model hyperparameter relationships.","marker":"Vaswani et al. (2017)"},{"why":"Provides the Bandits-BO baseline with per-subspace GPs and Thompson sampling, the primary GP-based comparison.","marker":"Nguyen et al. (2020)"},{"why":"Supplies the HPO-B-v3 benchmark and the meta-learning comparison used to test warm-started transfer.","marker":"Pineda-Arango et al. (2021)"}],"fun_headline_variants":["One GP for every subspace via self-attention","Attention unifies conditional search space surrogates","Single surrogate models all response surfaces","Hyperparameter embeddings plus attention: one BO","Unified GP with self-attention for conditional spaces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a hyperparameter's identity, value, index, and immediate parent's identity give the attention model enough information to tell apart different branches of the search-space tree; if two deep branches produce indistinguishable token sequences, the unified latent space would merge distinct subspaces and the single-GP claim would collapse.","fun_headline_variants_meta":{"raw":{"variants":["One GP for every subspace via self-attention","Attention unifies conditional search space surrogates","Single surrogate models all response surfaces","Hyperparameter embeddings plus attention: one BO","Unified GP with self-attention for conditional spaces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1360,"prompt_tokens":955,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":571,"tokens_out":405,"duration_ms":4330,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:38:08.965189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic conditional space with two deep branches in which the same hyperparameter identity appears under two different ancestors, and make the optimal value of that hyperparameter's child opposite in the two branches. Train AttnBO with the same budget used on the simulation function and check whether its posterior distinguishes the two branches; if it cannot place higher mass near both optima while keeping uncertainty low between them, the embedding lacks the ancestry information the unified model depends on.","supporting_citations":[],"review_version":1}