REVIEW 4 major objections 6 minor 20 references
Modeling All Response Surfaces in One for Conditional Search Spaces
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that a single Gaussian Process with a structure-aware, attention-based deep kernel can model the response surfaces of all subspaces in a conditional search space.
desk verdict AttnBO's unified attention-based deep GP is a plausible and somewhat novel idea, but the empirical evidence is weakened by test-set architecture selection and an unreproduced AddTree baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the structure-aware hyperparameter embedding of Eq. (3): $\mathrm{emb}(p) = \mathrm{concat}(\mathrm{id\_emb}(p), \mathrm{idx\_emb}(p), \mathrm{value\_emb}(p), \mathrm{id\_emb}(p^{\uparrow}))$, which encodes each hyperparameter's own identity, its index, its value, and its parent's identity. This tokenization lets a configuration be treated as a sequence, and the attention-based encoder models pairwise hyperparameter relationships before average pooling maps all sequences into one continuous latent space; a standard GP kernel is applied there. The machinery works because deep kernel learning lets all parameters, including the embedding and encoder, be learned end-to-end from the marginal likelihood, so the surrogate adapts the latent space to the observed response surface.
What would settle it
Construct a synthetic conditional space with two deep branches in which the same hyperparameter identity appears under two different ancestors, and make the optimal value of that hyperparameter's child opposite in the two branches. Train AttnBO with the same budget used on the simulation function and check whether its posterior distinguishes the two branches; if it cannot place higher mass near both optima while keeping uncertainty low between them, the embedding lacks the ancestry information the unified model depends on.
Extended reading notes
Core claim
AttnBO's central move is to replace the collection of per-subspace surrogates with one deep-kernel GP over a learned latent space. Each hyperparameter in a configuration is embedded by concatenating four trainable pieces: its own identity embedding, its index within vector-valued hyperparameters, a linear projection of its value, and the identity embedding of its immediate parent node. A transformer-style encoder applies self-attention over these token embeddings to capture relationships among hyperparameters, then average pooling compresses sequences of any length into a fixed-dimensional vector. This vector becomes the input to a Matérn 5/2 kernel inside the deep kernel learning framework, and the embedding, encoder, and kernel parameters are trained jointly by maximizing the log marginal likelihood. The resulting posterior can be queried for any configuration in any subspace, and expected improvement is optimized per subspace to produce either the single best query or a parallel batch.
Load-bearing premise
The load-bearing premise is that a hyperparameter's identity, value, index, and immediate parent's identity give the attention model enough information to tell apart different branches of the search-space tree; if two deep branches produce indistinguishable token sequences, the unified latent space would merge distinct subspaces and the single-GP claim would collapse.
Editorial extensions
If this is right
- Observations from every subspace are pooled into one GP, so the surrogate improves uniformly rather than only in the subspace where a query was made.
- Hyperparameters that are semantically related but live in different subspaces, such as gamma under different SVM kernels, are linked through shared attention weights and a common latent space.
- Optimizing the acquisition function per subspace yields a batch of queries in one BO iteration, allowing parallel evaluation without a separate batching strategy.
- Pre-training on HPO-B provides a warm start that reaches competitive regret in few trials, indicating the learned representation transfers across search spaces.
Reading between the lines
- The simulation section notes that the authors did not reproduce AddTree's published curve and attribute the gap to different initial-point counts; that makes the simulation comparison a same-setup comparison rather than an exact replication.
- The paper's embedding records only the immediate parent, not the full ancestor path or the parent's value; in trees four or more levels deep, the attention encoder must reconstruct ancestry from the whole sequence, and testing an embedding augmented with ancestor-path tokens would isolate whether this reconstruction is what makes the method work.
- A single smooth GP over a shared latent space assumes that subspaces are related through the learned geometry; if the true response surfaces are sharply discontinuous across subspaces, a unified model could hurt, and a direct check is to measure held-out predictive log-likelihood per subspace against separate GPs.
- Because the framework only needs a parent pointer, it should extend to graphs or multi-tree combinations, and to new search spaces at meta-test time by adding a new root identity; none of these extensions is tested in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AttnBO, a Bayesian optimization method for conditional (tree-structured) search spaces. It represents each hyperparameter by a structure-aware embedding (identity, index, value, and father-identity), treats a configuration as a sequence of such token embeddings, and applies a Transformer-style attention encoder followed by average pooling to map variable-length configurations from different subspaces into a common latent space, on which a single deep-kernel Gaussian Process is trained with EI acquisition. The paper reports experiments on a simulation function, a NAS space, three OpenML tasks (SVM, XGBoost, and a combined CASH space), and the HPO-B benchmark, claiming improved regret and sample efficiency over per-subspace and additive-tree baselines.
Significance. If the empirical claims hold, the unified-surrogate approach is a meaningful step beyond separate per-subspace GPs and additive-tree kernels, with an explicit mechanism for sharing observations across subspaces and a batch-query capability. The method is clearly described, the algorithm is self-contained, and the ablations on embedding design and encoder architecture (Fig. 8 and Table 2) are useful diagnostics. The per-search-space HPO-B results in the supplementary material are a further strength. However, the evaluation protocol contains load-bearing weaknesses (see major comments), so the significance of the empirical claims should be assessed after those are addressed.
major comments (4)
- [Implementation details; Table 2] The encoder hyperparameters (na, nb, and the presence of the MLP) are selected based on the combined average ranking on the SVM and XGBoost tasks, and these same two tasks appear as the first two rows of the main evaluation in Fig. 5(a,b) and as components of the CASH task in Fig. 5(c). This constitutes test-set tuning: the reported gains on these tasks may reflect model selection rather than the proposed unified-response-surface mechanism. Please select the architecture on a held-out set of tasks or via nested cross-validation, and re-report the main results for the configuration chosen without access to the evaluation tasks.
- [Fig. 3 caption (Simulation Function)] The caption of Fig. 3 states, 'Probably, we did not get the same results as shown in their paper due to the different number of initial points.' AddTree is the primary state-of-the-art baseline for conditional BO, and the simulation comparison is the main controlled experiment supporting the efficacy claim. A reimplementation that does not reproduce the original AddTree results makes the reported superiority over AddTree untestable. Please use the official AddTree implementation and verify it reproduces the published results under the exact evaluation protocol, or provide a corrected comparison with a documented protocol if full reproduction is not possible.
- [HPO-B Benchmark (Fig. 6)] The HPO-B comparison in Fig. 6 uses baseline numbers 'as reported in (Pineda-Arango et al. 2021)' rather than a common rerun under the same protocol. Because AttnBO uses its own initialization, fine-tuning schedule, and acquisition implementation, differences in regret may reflect evaluation settings rather than algorithmic merit. Please rerun the baselines under a shared protocol, or at minimum verify that the cited numbers are directly comparable (e.g., same number of trials, same initialization, same metric, same data splits).
- [Structure-aware Embeddings (Eq. 3)] The structure-aware embedding in Eq. (3) encodes only the value, identity, index, and immediate father's identity of each hyperparameter; it does not encode the father's value or the full ancestor path. In deep trees such as the NAS space (four or more levels), two configurations from different subspaces can have identical token-level embeddings for a node while differing in higher-level ancestry. The paper does not provide evidence that the attention mechanism recovers this ancestry context (e.g., via a latent-space visualization or an ablation with full ancestor information). Without such evidence, the claim that the embedding preserves structural information is not fully supported, and the NAS results may be sensitive to this representational choice. Please add a diagnostic analysis or an ablation with complete ancestor paths.
minor comments (6)
- [Notation throughout] Several equations contain notation errors, e.g., 'i⊂1,...,n' should be 'i=1,...,n', and Eq. (6) has an unmatched parenthesis. Please proofread the mathematical notation.
- [Fig. 6 legend] The legend in Fig. 6 uses 'AttnBOWS' and 'AttnBO_WS' inconsistently; these should be unified.
- [Experiments; Implementation details] The sentence 'The parameters of the encoder are selected based on their performance on the SVM and XGBoost tasks' should clarify whether this selection is done once on a validation split or on the same tasks that are later reported; as written, it invites the test-set tuning concern above.
- [Ablation Study] The ablation in Fig. 8 is reported only on SVM and XGBoost tasks, the same tasks used for architecture selection; reporting the ablation on the CASH or NAS tasks would make the conclusion more robust.
- [Related work] The statement 'we don't consider these methods as our competitors' for VAE-based latent-space BO methods (Kusner et al.; Lu et al.; Tripp et al.; Grosnit et al.; Maus et al.) is acceptable, but a brief discussion of why they do not handle mixed categorical/numerical conditional spaces would help readers.
- [Author affiliations] There is a formatting error in the author block: 'figure1 wxx@sjtu.edu.cn' should be the email address 'figure1wxx@sjtu.edu.cn' or similar. Please correct this in the camera-ready version.
Circularity Check
No circular derivation; the unified GP is fit by marginal likelihood and tested on out-of-sample regret, with only non-load-bearing self-citation.
full rationale
The paper's central claim is empirical: AttnBO models all conditional subspaces with one deep-kernel GP, where the structure-aware embedding (Eq. 3), attention encoder, and kernel (Eq. 5) are jointly fit by maximizing the marginal likelihood (Eq. 2). The reported results are out-of-sample BO regret on a simulation, OpenML tasks, and HPO-B, so no prediction reduces algebraically to a fitted parameter or to a self-citation. The only self-citation is the supplementary reference to Xue et al. (2023) for a YAML search-space format, which is not load-bearing for the method's validity. The manuscript's own caveats — encoder hyperparameters selected on the SVM/XGBoost tasks (Implementation details, Table 2) and the AddTree reimplementation not matching the original paper (Fig. 3 caption) — are empirical-validity and reproducibility concerns, not circular reasoning within the derivation chain. No step in the paper defines X in terms of Y, renames a known result, or imports a uniqueness theorem from the authors' prior work. The derivation is self-contained: the model is fit by likelihood maximization and evaluated by independent optimization performance.
Assumptions & free parameters
free parameters (3)
- Encoder architecture (na, nb, MLP layers) =
na=6 attention blocks, nb=2 heads, MLP [128,128,128,32]
- Latent space dimension =
32
- Training hyperparameters (learning rate, epochs, noise variance) =
lr=0.001, decay 0.5/30 epochs, 100 epochs; GP noise fitted by MLE
assumptions (4)
- domain assumption The conditional search space can be represented as a single tree (or combination of trees) with a virtual root; each subspace is a root-to-leaf path.
- domain assumption A stationary GP kernel on the pooled attention-encoded latent space can adequately model the response surface across all subspaces.
- domain assumption Joint maximum-likelihood training of the deep kernel (Eq. 2/5) yields a surrogate whose uncertainty estimates are reliable enough for EI acquisition.
- domain assumption Ordinal encoding of hyperparameter identities provides a suitable input for learning semantic embeddings.
invented entities (1)
-
Virtual father vertex for root hyperparameters
Cite this review
Pith. "Pith review of Modeling All Response Surfaces in One for Conditional Search Spaces." pith.science (2026). https://pith.science/paper/NATZKPF3
@misc{pith2026250104260,
author = {Pith},
title = {Pith review of: Modeling All Response Surfaces in One for Conditional Search Spaces},
year = {2026},
howpublished = {\url{https://pith.science/paper/NATZKPF3}},
note = {Machine review of arXiv:2501.04260}
}
read the original abstract
Bayesian Optimization (BO) is a sample-efficient black-box optimizer commonly used in search spaces where hyperparameters are independent. However, in many practical AutoML scenarios, there will be dependencies among hyperparameters, forming a conditional search space, which can be partitioned into structurally distinct subspaces. The structure and dimensionality of hyperparameter configurations vary across these subspaces, challenging the application of BO. Some previous BO works have proposed solutions to develop multiple Gaussian Process models in these subspaces. However, these approaches tend to be inefficient as they require a substantial number of observations to guarantee each GP's performance and cannot capture relationships between hyperparameters across different subspaces. To address these issues, this paper proposes a novel approach to model the response surfaces of all subspaces in one, which can model the relationships between hyperparameters elegantly via a self-attention mechanism. Concretely, we design a structure-aware hyperparameter embedding to preserve the structural information. Then, we introduce an attention-based deep feature extractor, capable of projecting configurations with different structures from various subspaces into a unified feature space, where the response surfaces can be formulated using a single standard Gaussian Process. The empirical results on a simulation function, various real-world tasks, and HPO-B benchmark demonstrate that our proposed approach improves the efficacy and efficiency of BO within conditional search spaces.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
For every search space, we will give both the Y AML-style and figure-style representation. Simulation Benchmark The tree-structure search space of the simulation function that was originally presented in Jenatton et al. (2017) con- sists of 9 hyperparameters as Listing 1 and Fig. 9 shows. This space has three binary decision variables x1, x2, x3, two shar...
work page 2017
-
[2]
conv op. The convolution operation type for a single layer of each block. In our settings, following Tan et al. Listing 2: Y AML of the SVM search space. C: type : f l o a t range : [ 0 . 0 0 1 . . . 1 0 0 0 ] k e r n e l: type : c h o i c e range : {"linear" , "poly" , "sigmoid" , " rbf"} submodule : poly : degree : type : i n t range : [ 2 . . . 6 ] gam...
work page 2019
-
[3]
Bayesian Optimization with Robust Bayesian Neu- ral Networks. In NeurIPS. Swersky, K.; Duvenaud, D.; Snoek, J.; Hutter, F.; and Os- borne, M. A. 2014. Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces. arXiv:1409.4011. Tan, M.; Chen, B.; Pang, R.; Vasudevan, V .; Sandler, M.; Howard, A.; and Le, Q. V . 201...
arXiv 2014
- [4]
-
[5]
The ratio for expending, if using the in- verted residue block (Sandler et al
expend ration. The ratio for expending, if using the in- verted residue block (Sandler et al. 2018)
work page 2018
- [6]
-
[7]
Deep Kernel Learning. In AISTATS. Wistuba, M.; and Grabocka, J. 2021. Few-Shot Bayesian Optimization with Deep Kernel Surrogates. In ICLR. Xue, C.; Liu, W.; Xie, S.; Wang, Z.; Li, J.; Peng, X.; Ding, L.; Zhao, S.; Cao, Q.; Yang, Y .; He, F.; Cai, B.; Bian, R.; Zhao, Y .; Zheng, H.; Liu, X.; Liu, D.; Liu, D.; Shen, L.; Li, C.; Zhang, S.; Zhang, Y .; Chen, ...
work page 2021
-
[8]
The number of strides for each block is represented in binary
stride layer. The number of strides for each block is represented in binary. The optimization hyperparameters The details of each hy- perparameter are as follows: Listing 3: Y AML of the XGBoost search space. b o o s t e r: type : c h o i c e range : { g b t r e e , g b l i n e a r} submodule : g b t r e e: n e s t i m a t o r s: type : i n t range : [ 5 ...
Show all 20 references
-
[9]
The number of blocks
nums block. The number of blocks
-
[11]
The second type is the depthwise separable convolu- tion layer (Howard et al. 2017). It has the same function as the standard convolution layer but is more efficient, which is a form of factorized convolutions with a stan- dard convolution into a depthwise convolution and a 1×...
2017
-
[12]
The size of the convolution kernel in one convolution block
kernel size. The size of the convolution kernel in one convolution block
-
[15]
The ratio of squeezing and expending if con- taining such structure
seratio. The ratio of squeezing and expending if con- taining such structure
-
[16]
The number of channels for each block
nums channel. The number of channels for each block
-
[18]
The learning rate determines the speed of the network’s training and convergence
learning rate. The learning rate determines the speed of the network’s training and convergence
-
[19]
Size of the change in the parameter when the optimizer updates the parameter
step size. Size of the change in the parameter when the optimizer updates the parameter
-
[20]
Batch size determines how many data points will be used for training in each iteration
batch size. Batch size determines how many data points will be used for training in each iteration. The structure of the search space of the NAS task is shown in Listing 4 and Fig. 13. Meta-Learning on HPO-B-v3 Benchmark HPO-B-v3 (Pineda-Arango et al. 2021) is a large-scale hy...
2021
-
[2013]
In SIGKDD
Auto-WEKA: combined selection and hyperparame- ter optimization of classification algorithms. In SIGKDD. Tripp, A.; Daxberger, E. A.; and Hern ´andez-Lobato, J. M
-
[2016]
Ann Math Artif Intell
Bayesian optimization for learning gaits under uncer- tainty - An experimental comparison on a dynamic bipedal walker. Ann Math Artif Intell . Cowen-Rivers, A. I.; Lyu, W.; Tutunov, R.; Wang, Z.; Gros- nit, A.; Griffiths, R.; Maraval, A. M.; Hao, J.; Wang, J.; Pe- ters, J.; an...
2022 arXiv
-
[2017]
In IJCNN
Bayesian optimization for conditional hyperparam- eter spaces. In IJCNN. Lin, T.; Wang, Y .; Liu, X.; and Qiu, X. 2022. A survey of transformers. AI Open. Lin, X.; Ding, C.; Zeng, J.; and Tao, D. 2020. Gps-net: Graph property sensing network for scene graph generation. In CVPR...
2022
-
[2020]
In NeurIPS
Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining. In NeurIPS. Vanschoren, J.; van Rijn, J. N.; Bischl, B.; and Torgo, L
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.