REVIEW 4 major objections 5 minor 16 references
Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that a product kernel over an embedding index lets Bayesian optimization switch linear projections region by region.
desk verdict Unfinished draft whose one new idea is explicitly acknowledged as known mixed-variable BO, with no quantitative experiments or new theory to back the central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the index-conditioned product kernel K((x_i,z_i),(x_j,z_j)) = K_x(x_i,x_j) * K_z(z_i,z_j) on the augmented space of low-dimensional points x and embedding indices z. Here K_x is an ARD kernel (Matern or squared exponential) over the latent optimization variable, and K_z is a smooth exponential kernel exp(-$lambda^{2}$ (z_i-z_j)^2) over the discrete embedding index. The product form is what lets one GP share data across K random Gaussian embeddings: it treats z as an extra input, so the surrogate can learn which embeddings correlate with which regions and weight them accordingly at acquisition time. The paper also motivates an orthogonalized 'cross embedding' variant obtained by orthogonalizing the random projection matrix, but the adaptive kernel is the mechanism that carries the central claim.
What would settle it
Build a synthetic objective whose active directions change by region (e.g., in a 20-dimensional box, only coordinates 1-5 matter when x_1<0 and only coordinates 6-10 matter when x_1>0), run SA-cREMBO with K random embeddings small enough that some regions are almost surely uncovered, and record the selected index at each evaluation. If the surrogate still reduces uncertainty and finds the optimum where no embedding contains the local active coordinates, the local-coverage assumption is not the bottleneck; if performance collapses or the selected embedding is uncorrelated with the region's active subspace, the paper's central mechanism fails.
Extended reading notes
Core claim
The central discovery is that the embedding itself can be treated as a model choice rather than a fixed preprocessing step. SA-cREMBO defines the surrogate as f(x,z):=f(A_z x) over the augmented space R^d times {1,...,K} and factorizes the GP kernel as K((x_i,z_i),(x_j,z_j)) = K_x(x_i,x_j) * K_z(z_i,z_j), with K_z(z_i,z_j)=exp(-$lambda^{2}$ (z_i-z_j)^2). Because all observations feed a single GP, information from every embedding contributes to every prediction, and the learned correlations in the kernel let the model up-weight embeddings that locally capture the objective's active subspace. The paper states this formulation enables adaptive selection among multiple subspaces and mitigates the limitation of a single projection in standard REMBO, and presents it as a way to capture locally varying effective dimensionality, nonstationarity, and heteroscedasticity.
Load-bearing premise
The method's central assumption is that in every region the optimizer visits, at least one of the K random linear projections happens to align with the directions that actually matter there, and that the discrete embedding index can be treated by a smooth exponential kernel even though the index is unordered; the paper does not analyze the probability of either condition.
Editorial extensions
If this is right
- Under one GP, every objective evaluation updates the model for all K embeddings, so SA-cREMBO should reach a given regret with fewer total evaluations than K independent REMBO runs.
- The learned index kernel provides a location-dependent posterior over embedding relevance, so the acquisition step can switch projections without a separate model-selection loop.
- Standard REMBO is recovered as a special case (K=1 or a delta kernel over z), making the framework a strict generalization rather than a competing algorithm.
- On objectives whose active directions differ across regions—the nonstationary and heteroscedastic cases the paper targets—the method should outperform both single-embedding REMBO and ensemble approaches that train separate surrogates.
- Because bad embeddings are down-weighted rather than discarded, the optimizer gains robustness against the failure of any single random projection.
Reading between the lines
- A testable extension, not stated in the paper: replace the smooth scalar kernel on z with learned latent vectors for each embedding, which would remove the arbitrary ordering that the exponential kernel exp(-lambda^2 (z_i-z_j)^2) imposes on indices that are actually unordered.
- If local coverage is the true bottleneck, the method's gain should scale with how the K embeddings are drawn: a greedy scheme that adds a new embedding when the GP's predictive variance stays high inside a region could outperform fixed random draws, though that is an inference beyond the paper.
- The framework suggests a bridge to heteroscedastic GP modeling: the index dimension can act as a discrete 'regime' label, so the same kernel machinery could model nonstationary variance, though the paper does not demonstrate this directly.
- A natural stress test would be a benchmark where each region's active subspace is known; then one can measure whether the learned K_z actually up-weights the embedding whose projection contains that subspace, which the paper does not report.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes SA-REMBO/cREMBO, an extension of REMBO in which multiple random Gaussian embeddings are used and an embedding index z is treated as an additional discrete input to a single Gaussian process surrogate with a product kernel. The paper claims that this construction yields adaptive embedding selection, improved sample efficiency, and robustness on nonstationary high-dimensional objectives, and it promises both theoretical analysis and empirical demonstrations. In the submitted version, the theoretical analysis consists of restatements of known REMBO results, and the experimental content is a single figure with no quantitative results, so the central claims are not supported by the evidence provided.
Significance. The underlying idea of sharing information across multiple random embeddings through an index-conditioned kernel is potentially interesting, since the failure of a single random projection is a recognized limitation of REMBO. If the method were precisely specified, accompanied by a coverage condition ensuring that at least one embedding captures local structure, and validated with controlled benchmark comparisons, it could be a useful contribution. However, as submitted, the significance remains purely conceptual: no new theorem about the proposed kernel is proved, no algorithm is fully specified, and the experimental section contains no numbers, error bars, or protocol details. The positive feature is that the paper coherently restates the standard REMBO theorems in the supplementary material, but those results are not novel and do not concern the proposed product kernel.
major comments (4)
- [Section 5, Figure 3] The experimental section contains a single subsection and one figure with no quantitative results. There is no table of final objective values, no number of initial points, no optimization budget, no number of independent runs, no error bars or confidence intervals, and no specification of the baseline methods or their configurations. The abstract's claim that the method 'empirically demonstrate[s] the advantage' over REMBO and other low-rank BO methods is therefore unsupported by anything reported in the manuscript. Because the central claim is an empirical advantage, this omission is load-bearing and cannot be fixed by a local edit.
- [Section 4.3 and Supplementary Material] The section titled 'Theoretical Implications' contains only qualitative statements about sample efficiency and posterior relevance; no theorem, proposition, or derivation is stated for the index-conditioned product kernel. The proofs in the Supplementary Material, equations (4) through (17), are restatements of the original REMBO guarantees for a single random embedding and a fixed effective subspace. They do not analyze the product kernel kx·kz, the adaptive selection mechanism, or the stability of the model under varying embeddings. Thus the theoretical contribution promised in the abstract is absent.
- [Section 4.1, kernel definition for z] The kernel kz(zi,zj) = exp(-lambda^2(zi-zj)^2) assigns an Euclidean metric to the discrete, unordered embedding index. Relabeling the embeddings permutes the GP posterior and the acquisition function arbitrarily, so the claimed 'adaptive selection' behavior could be an artifact of the chosen label order rather than of the objective's structure. The paper neither justifies this ordering nor demonstrates invariance under permutation. In addition, no condition is stated to guarantee that, in every region the optimizer visits, at least one of the K embeddings covers the local effective subspace; without such a coverage condition, the surrogate cannot represent the objective in regions where all projections fail.
- [Section 4 and Algorithm 1] The method description is internally inconsistent. Algorithm 1 and the surrounding text describe cREMBO, which augments a single random projection with an orthogonal projection obtained via Gram-Schmidt, while Section 4.1 introduces K arbitrary random embedding matrices and a product-kernel GP over (x,z). No algorithm is provided for the latter formulation: there is no statement of how the K embeddings are generated, which index values are used during training, how the acquisition function is optimized over z, or how the final recommendation is produced. As a result, the proposed method is underspecified and not reproducible from the manuscript.
minor comments (5)
- [Title and Abstract] The title uses 'Adaptive Linear Embedding,' the abstract introduces 'Self-Adaptive cross embedding REMBO (SA-cREMBO),' and Section 4 calls the method 'cross Random Embedding Bayesian Optimization (cREMBO)'; these names should be unified throughout the paper.
- [Section 5 and Section 6] Section 5 ends after a single experimental subsection, and Section 6 is a second heading titled 'Method Description' with no content, followed directly by Section 7. This structural error should be corrected.
- [Related Work, references [8] and [9]] The text states that Eriksson et al. [8] and Binois et al. [9] 'proposed using multiple random embedding matrices in parallel,' but reference [8] is TuRBO and reference [9] is a heteroscedastic GP method; these citations do not support the claimed related work.
- [Figure 3] The figure is difficult to interpret: axis labels and legends are unclear, the name 'Harzmann6' appears to be a typo for 'Hartmann6,' and no uncertainty information is displayed.
- [Supplementary proofs] In the proof of Theorem 2, the text refers to 'Proposition 1' and 'Proposition 8,' but no propositions are numbered in the main text; these references should be corrected.
Circularity Check
No circular derivation found; the main gaps are missing experiments and unproven assertions, not self-referential reasoning.
full rationale
The paper's core construction is explicit and non-circular. It defines f(x,z) := f(A_z x) and a product kernel K((xi,zi),(xj,zj)) = Kx(xi,xj)*Kz(zi,zj), then claims that this joint model can adaptively select embeddings. This is a modeling proposal rather than a derivation that reduces to its own inputs. The REMBO theorems quoted in Section 2 are attributed to Wang et al. [7], an external source, and concern a fixed single embedding; they do not presuppose the index-conditioned GP, so no self-citation chain is load-bearing. The paper does not fit a parameter to a subset of data and then rename it as a prediction, nor does it import a uniqueness theorem from the authors' own prior work. Section 4.3, titled 'Theoretical Implications,' contains only qualitative assertions and no theorem, and Section 5 contains no quantitative experimental results despite the abstract's claim of empirical demonstration. Section 4.4 itself lists kernel design for the discrete index and mixed-variable acquisition optimization as unresolved limitations. These are genuine evidential and rigor gaps, but they are not circularity. The arbitrary ordering imposed by kz(zi,zj)=exp(-lambda^2(zi-zj)^2) is an explicit modeling choice, not a hidden reuse of the conclusion. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (2)
- index kernel smoothing parameter lambda =
not specified
- number of embeddings K =
not specified
assumptions (4)
- domain assumption The objective f has a global low effective dimensionality d_e that a random embedding can capture, per REMBO's Theorem 1 (quoted from [7]).
- domain assumption There exists at least one embedding among the K random matrices whose projection aligns with the local effective subspace in each region of the search space.
- ad hoc to paper The discrete embedding index can be modeled as a continuous smooth variable via the exponential kernel kz.
- domain assumption The GP surrogate with a product kernel over (x,z) can accurately represent f(x,z)=f(A_z x).
invented entities (1)
-
Discrete embedding index z
Cite this review
Pith. "Pith review of Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization." pith.science (2026). https://pith.science/paper/O5O5VRVO
@misc{pith2026250511281,
author = {Pith},
title = {Pith review of: Adaptive Linear Embedding for Nonstationary High-Dimensional Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/O5O5VRVO}},
note = {Machine review of arXiv:2505.11281}
}
read the original abstract
Bayesian Optimization (BO) in high-dimensional spaces remains fundamentally limited by the curse of dimensionality and the rigidity of global low-dimensional assumptions. While Random EMbedding Bayesian Optimization (REMBO) mitigates this via linear projections into low-dimensional subspaces, it typically assumes a single global embedding and a stationary objective. In this work, we introduce Self-Adaptive embedding REMBO (SA-REMBO), a novel framework that generalizes REMBO to support multiple random Gaussian embeddings, each capturing a different local subspace structure of the high-dimensional objective. An index variable governs the embedding choice and is jointly modeled with the latent optimization variable via a product kernel in a Gaussian Process surrogate. This enables the optimizer to adaptively select embeddings conditioned on location, effectively capturing locally varying effective dimensionality, nonstationarity, and heteroscedasticity in the objective landscape. We theoretically analyze the expressiveness and stability of the index-conditioned product kernel and empirically demonstrate the advantage of our method across synthetic and real-world high-dimensional benchmarks, where traditional REMBO and other low-rank BO methods fail. Our results establish SA-REMBO as a powerful and flexible extension for scalable BO in complex, structured design spaces.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[12]
Mixed-variable bayesian optimiza- tion,
E. Daxberger, A. Makarova, M. Turchetta, and A. Krause, “Mixed-variable bayesian optimiza- tion,” arXiv preprint arXiv:1907.01329, 2019
arXiv 1907
-
[7]
Bayesian optimization in high dimensions via random embeddings,
Z. Wang, M. Zoghi, F. Hutter, D. Matheson, and N. De Freitas, “Bayesian optimization in high dimensions via random embeddings,” in Twenty-Third international joint conference on artificial intelligence, 2013
work page 2013
-
[1]
Multi-fidelity surrogate-based optimization for electromagnetic simulation acceleration,
Y . Wang, P. D. Franzon, D. Smart, and B. Swahn, “Multi-fidelity surrogate-based optimization for electromagnetic simulation acceleration,” ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 25, no. 5, pp. 1–21, 2020
work page 2020
-
[2]
A global bayesian optimization algorithm and its application to integrated system design,
H. M. Torun, M. Swaminathan, A. K. Davis, and M. L. F. Bellaredj, “A global bayesian optimization algorithm and its application to integrated system design,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 26, no. 4, pp. 792–802, 2018. 10
work page 2018
-
[3]
W. Lyu, F. Yang, C. Yan, D. Zhou, and X. Zeng, “Batch bayesian optimization via multi- objective acquisition ensemble for automated analog circuit design,” inInternational conference on machine learning. PMLR, 2018, pp. 3306–3314
work page 2018
-
[4]
Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms,
J. Bergstra, D. Yamins, D. D. Cox et al., “Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms,” in Proceedings of the 12th Python in science conference, vol. 13. Citeseer, 2013, p. 20
work page 2013
-
[5]
Practical bayesian optimization of machine learning algorithms,
J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” Advances in neural information processing systems, vol. 25, 2012
2012
-
[6]
A bayesian interactive optimization approach to procedural animation design,
E. Brochu, T. Brochu, and N. De Freitas, “A bayesian interactive optimization approach to procedural animation design,” in Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2010, pp. 103–112
work page 2010
Show all 16 references
-
[8]
Scalable global optimization via local bayesian optimization,
D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek, “Scalable global optimization via local bayesian optimization,” Advances in neural information processing systems, vol. 32, 2019
2019
-
[9]
Practical heteroscedastic gaussian process modeling for large simulation experiments,
M. Binois, R. B. Gramacy, and M. Ludkovski, “Practical heteroscedastic gaussian process modeling for large simulation experiments,” Journal of Computational and Graphical Statistics, vol. 27, no. 4, pp. 808–821, 2018
2018
-
[10]
High dimensional bayesian optimization via condensing-expansion projection,
J. Lu and R. J. Zhu, “High dimensional bayesian optimization via condensing-expansion projection,” arXiv preprint arXiv:2408.04860, 2024
2024 arXiv
-
[11]
Bayesian optimization in high-dimensional spaces: A brief survey,
M. Malu, G. Dasarathy, and A. Spanias, “Bayesian optimization in high-dimensional spaces: A brief survey,” in2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA). IEEE, 2021, pp. 1–8
2021
-
[13]
Multi-task bayesian optimization,
K. Swersky, J. Snoek, and R. P. Adams, “Multi-task bayesian optimization,”Advances in neural information processing systems, vol. 26, 2013
2013
-
[14]
Bayesian optimization in a billion dimensions via random embeddings,
Z. Wang, F. Hutter, M. Zoghi, D. Matheson, and N. De Feitas, “Bayesian optimization in a billion dimensions via random embeddings,” Journal of Artificial Intelligence Research, vol. 55, pp. 361–387, 2016
2016
-
[15]
Re-examining linear embeddings for high- dimensional bayesian optimization,
B. Letham, R. Calandra, A. Rai, and E. Bakshy, “Re-examining linear embeddings for high- dimensional bayesian optimization,” arXiv preprint arXiv:2001.11659, 2020
2001 arXiv
-
[16]
Smoothed analysis of the condition numbers and growth factors of matrices,
A. Sankar, D. A. Spielman, and S.-H. Teng, “Smoothed analysis of the condition numbers and growth factors of matrices,” SIAM Journal on Matrix Analysis and Applications, vol. 28, no. 2, pp. 446–476, 2006. 11
2006
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.