REVIEW 3 major objections 4 minor 42 references
ALAS: Additive Learnable Alpha-Stable Kernels for Flexible Bayesian Optimization
T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read ALAS claims that a single learned stability parameter α in an α-stable spectral kernel lets Gaussian-process surrogates self-adapt from smooth to rough objectives, with regret bounds that depend explicitly on α.
desk verdict Solid learnable α-stable kernel for BO; theory rests on an unverified Widom step, and the experiments omit the most direct baseline and error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The base object is the symmetric α-stable spectral density p(ω;α,δ), whose characteristic function exp(-|δt|^α) gives the powered-exponential envelope exp(-|τ/ℓ|^α); cosine modulation shifts the spectrum into a pair of stable densities centered at ±γ. The argument's load-bearing step is a classical theorem relating the eigenvalue counting function of the kernel integral operator on [0,L] to the Lebesgue measure of the spectral super-level set {ω:S(ω)>ε}; applying it to the α-stable tail converts α directly into the polynomial decay exponent 1+α of the operator eigenvalues, which then yields the information-gain and regret rates.
What would settle it
Compute the kernel operator eigenvalues of k(τ)=exp(-|τ/ℓ|^α) (and its cosine-modulated version) numerically on [0,L] for α=0.5, 1.0, 1.5. If h^{1+α}λ_h is not eventually bounded above and below, or if the counting function N(ε) does not scale as ε^{-1/(1+α)}, the polynomial eigen-decay — and with it the information-gain and regret bounds — has no support.
Extended reading notes
Core claim
On its own terms, this paper establishes that the smoothness of a GP surrogate can be made a learnable parameter without losing the standard Bayesian-optimization guarantees. For the 1D kernel k(τ)=w exp(-(2πδ|τ|)^α) cos(2πγτ), the spectral density is a pair of symmetric α-stable densities, so its tail decays polynomially with exponent 1+α. The paper proves that this polynomial spectral tail forces the kernel operator eigenvalues to decay as h^{-(1+α)} and hence forces information gain γ_T=Õ(T^{1/(1+α)}) and regret R_T=Õ(T^{1/2+1/(2(1+α))}). It also proves an additive variant whose spectrum is the union of 1D spectra, giving γ_T ≤ Õ(d·T^{1/(1+α_min)}). The claim is that α is therefore a genu
Load-bearing premise
The entire rate analysis rests on a quoted theorem equating the kernel operator's eigenvalue counting function with the Lebesgue measure of the spectral super-level set {ω:S(ω)>ε}, a theorem whose hypotheses are nowhere verified for the powered-exponential kernel exp(-|τ/ℓ|^α), which is non-differentiable at τ=0 for α<2.
Editorial extensions
If this is right
- For smooth objectives, marginal-likelihood training drives α toward 2 and ALAS behaves like a standard smooth spectral kernel; no manual kernel selection is needed.
- For rough objectives, smaller learned α produces slower theoretical rates but a better-fitting surrogate; the regret bound remains sublinear for every α>0.
- ALAS-Sep's additive construction gives high-dimensional BO a bound that depends on dimension only through the linear factor d, and the overall rate is controlled by the least smooth coordinate.
- Because the bounds plug into the standard GP-UCB analysis, acquisition functions such as UCB, EI, and PI inherit the same sublinear regret guarantee.
- The learned α values can be read as an empirical regularity diagnostic; controlled experiments show they track the Fourier-coefficient decay of the target.
Reading between the lines
- The theoretical hinge is the quoted eigenvalue-counting theorem; a direct numerical check of whether h^{1+α}λ_h stays bounded for the powered-exponential kernel on [0,L] — a kernel that is not differentiable at τ=0 when α<2 — would either confirm or undermine the regret bound.
- The paper implies a trade-off that is likely universal: roughness itself costs sample complexity, since smaller α yields larger information gain, so a learned α is not a free lunch but an honest accounting of how hard the objective is.
- The construction suggests natural extensions — per-dimension α learned by a small network, multiple α-stable components in a mixture, or asymmetric stable spectra — whose eigen-decay proofs should carry through as long as the spectral tail is regularly varying.
- For practitioners, the benchmark pattern suggests a decision rule: use joint ALAS when cross-dimension interactions dominate and ALAS-Sep when the objective is near-additive or coordinates have unequal roughness, accepting a warm-up cost while α is being learned.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes ALAS, a stationary GP kernel family whose spectral density is composed of symmetric α-stable components with a learnable stability parameter α∈(0,2], and ALAS-Sep, an additive variant with per-dimension α_j. The central claims are (i) the 1D ALAS kernel has Mercer eigenvalues λ_h=Θ(h^{-(1+α)}), leading to information gain γ_T=Õ(T^{1/(1+α)}) and cumulative regret R_T=Õ(T^{1/2+1/(2(1+α))}); (ii) ALAS-Sep inherits the rate governed by the smallest α_j, up to a factor d; and (iii) empirically ALAS/ALAS-Sep perform competitively on benchmarks from d=3 to 30, with learned α tracking objective roughness. The theoretical derivation follows the standard route via Bochner, Mercer, Widom, and GP-UCB, but the Widom step is used as an unverified black box and the additive eigenvalue proof has a centering/union gap.
Significance. If the eigenvalue-decay claim is established, the paper gives a clean quantitative link between a learnable spectral-tail parameter and BO complexity: α acts as a complexity knob spanning Gaussian (α=2) to heavy-tailed (α<2) regimes, and the additive variant yields a dimension-dependence that is linear rather than exponential. The kernel family is simple, retains exact GP inference, and is accompanied by released code and a broad benchmark comparison. The paper does not ship machine-checked proofs; its theoretical contribution is a standard analysis plus a new kernel family, and the empirical evaluation covers multiple acquisitions. However, the two proof gaps identified below must be fixed before the central claims are fully supported.
major comments (3)
- [Appendix A.1, Eq. (34); Prop. 4.1] The entire rate chain rests on Widom's theorem quoted as N(ε)≍|{ω:S(ω)>ε}|. The manuscript neither states the hypotheses of the theorem nor verifies them for S(ω)=(w/2)[p(ω−γ;α,δ)+p(ω+γ;α,δ)]. Widom's result is not unconditional; it requires regularity, monotonicity, and/or regular-variation conditions on the symbol. The SαS density is smooth and regularly varying, so I expect the conclusion is true, but as written the proposition is a black box. Please state the theorem precisely and verify its conditions, including the effect of the shift γ and the finite interval [0,L]. Since Prop. 4.1 and Theorem 4.2 inherit this step, this is a load-bearing gap.
- [Appendix A.3, Prop. A.4 and Cor. A.5] The proof replaces each ALAS-Sep component by its 'centered version' satisfying (51), but the kernel actually defined in (19)–(20) is not centered. For a sum of uncentered kernel operators, the nonzero spectrum is not the multiset union of the component spectra; that union property follows only if the components act on orthogonal subspaces or if centering makes the cross terms vanish. The assertion that centering affects only finitely many eigenvalues is not proved and is generally false for an infinite-rank operator perturbed by a finite-rank operator. As a result, the proof of Cor. A.5 is invalid. The final information-gain bound can be recovered directly from Lemma 4.3 without the union; please revise the appendix to either prove the union for the actual kernel or derive the additive bound via Lemma 4.3 and remove the false spectral-union claim.
- [Sec. 4.3, Eq. (18)] The d-dimensional ALAS kernel in (18) has Fourier transform equal to a product of univariate SαS densities, since exp(−Σ_j(2πδ_j|τ_j|)^α) factors across coordinates; it is not a d-dimensional symmetric α-stable density. Thus 'built from symmetric α-stable spectral components' and 'single d-dimensional stationary component' overstate the spectral connection. More importantly, all theoretical results are for 1D components and the additive ALAS-Sep; no eigenvalue, information-gain, or regret statement is made for the joint ALAS kernel in (18). Please clarify the actual spectral density of (18) and state precisely which results apply to which construction.
minor comments (4)
- [Fig. 3] The main optimization figure reports only means over 10 seeds with no error bars or confidence bands. This makes it difficult to judge whether the apparent gains of ALAS/ALAS-Sep are significant, especially in panels with narrow y-ranges such as Rastrigin-30d. Please add error bars or shaded bands, or provide a statistical significance analysis.
- [Table 3] Several entries have very large standard deviations (e.g., RBF PLL −2.86±6.07 on Weierstrass, Matérn-1/2 RMSE 5.25±2.29 on Branin). Please discuss seed-level variability and consider reporting medians or trimmed statistics, since these numbers weaken the 'robust performance' claim.
- [Appendix A.1] The reference to Widom (1963) should include the exact theorem statement and a pointer to the specific result being used. As written, Eq. (34) is a one-line invocation of a nontrivial asymptotic result, which is insufficient for a self-contained proof.
- [Sec. 4.2 / Algorithm 1] The theoretical results are stated for a kernel with fixed α, while Algorithm 1 re-estimates α by marginal likelihood at each iteration. The paper should explicitly state that the GP-UCB analysis applies to the kernel family with a fixed α and does not cover the MLL-based estimation procedure; otherwise the phrase 'learning α' overstates the theoretical support.
Circularity Check
No circularity: ALAS theory is derived from external standard theorems; the learned α is a kernel hyperparameter, not a fitted prediction.
full rationale
The paper's derivation chain is self-contained in the relevant sense. The ALAS kernel is defined as a powered-exponential envelope times a cosine (eq. 12); Bochner's theorem gives its spectral density as a shifted symmetric α-stable density (eq. 13); the standard tail property of α-stable laws (Prop. A.1, citing Samorodnitsky & Taqqu) gives the polynomial spectral tail |ω|^{-(1+α)}; Widom's theorem (external, 1963) transfers spectral superlevel counts to Mercer eigenvalue counting (App. A.1, eq. 34); and standard Mercer/GP-UCB inequalities (Srinivas et al.; Vakili et al.) convert eigenvalue decay to information-gain and regret bounds. Each step uses an externally established theorem; no step assumes the conclusion. The parameter α is a hyperparameter of the kernel definition, not a constant fitted to the eigenvalue rate or regret curve; the eigenvalue and regret statements are deterministic claims about the kernel for any fixed α. The adaptivity experiments fit α by marginal likelihood and report it, but that is standard hyperparameter learning, not a fitted value relabeled as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz justified only by the authors' prior work. The main correctness concern — that App. A.1 invokes Widom's theorem without stating or verifying its hypotheses for this spectral density — is a proof-rigor risk, not circularity: the cited theorem is independent external support, and the paper does not redefine the theorem to match its target result.
Assumptions & free parameters
free parameters (5)
- alpha (stability index) =
learned via MLL; e.g., 1.24 on Weierstrass, 2.0 on smooth tasks
- gamma (modulation frequency) =
learned, initialized from empirical spectrum
- delta_j (spectral scale/lengthscale) =
learned (ARD per dim for ALAS; per dim for ALAS-Sep)
- w (component weight) =
learned
- sigma^2 (noise variance) =
learned with weak priors
assumptions (7)
- standard math Bochner's theorem: stationary kernels are Fourier transforms of nonnegative spectral measures.
- standard math Mercer's theorem: continuous kernel on compact domain has discrete eigenexpansion.
- standard math Widom's theorem: N(ε) ≍ |{ω: S(ω)>ε}| for stationary kernels on an interval.
- standard math Symmetric α-stable densities have polynomial tails |ω|^{-(1+α)} for α<2.
- standard math GP-UCB regret satisfies R_T = Õ(√(T γ_T)).
- domain assumption The objective function lies in the RKHS of the ALAS kernel with the learned α.
- ad hoc to paper ALAS-Sep components are centered so constants are zero eigenfunctions.
Cite this review
Pith. "Pith review of ALAS: Additive Learnable Alpha-Stable Kernels for Flexible Bayesian Optimization." pith.science (2026). https://pith.science/paper/FULIUDN7
@misc{pith2026260718282,
author = {Pith},
title = {Pith review of: ALAS: Additive Learnable Alpha-Stable Kernels for Flexible Bayesian Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/FULIUDN7}},
note = {Machine review of arXiv:2607.18282}
}
abstract
Bayesian Optimization is widely used for expensive black-box optimization, yet its success often depends on choosing a kernel that matches the objective's unknown structure. In this work, we propose ALAS, a flexible Gaussian Process kernel family built from symmetric $\alpha$-stable spectral components. By learning the stability parameter $\alpha$, ALAS adapts its effective smoothness from data, capturing both smooth trends and sharp irregularities. We present two parameterizations: ALAS, a single stationary component with joint spectral modulation, and ALAS-Sep, a separable variant that learns dimension-wise tail behavior to improve robustness on approximately decomposable objectives. Experiments on standard benchmarks and real-world surrogates demonstrate strong and robust performance across diverse settings.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:1807.02811 , year =
A Tutorial on Bayesian Optimization , author =. arXiv preprint arXiv:1807.02811 , year =
-
[2]
Artificial intelligence and statistics , pages=
Deep kernel learning , author=. Artificial intelligence and statistics , pages=. 2016 , organization=
2016
-
[3]
2006 , publisher =
Gaussian Processes for Machine Learning , author =. 2006 , publisher =
2006
-
[4]
International conference on machine learning , pages=
Gaussian process kernels for pattern discovery and extrapolation , author=. International conference on machine learning , pages=. 2013 , organization=
2013
-
[5]
arXiv preprint arXiv:0912.3995 , year=
Gaussian process optimization in the bandit setting: No regret and experimental design , author=. arXiv preprint arXiv:0912.3995 , year=
-
[6]
IEEE transactions on information theory , volume=
Information-theoretic regret bounds for gaussian process optimization in the bandit setting , author=. IEEE transactions on information theory , volume=. 2012 , publisher=
2012
-
[7]
Advances in neural information processing systems , volume=
Practical bayesian optimization of machine learning algorithms , author=. Advances in neural information processing systems , volume=
-
[8]
Artificial intelligence and statistics , pages=
Deep gaussian processes , author=. Artificial intelligence and statistics , pages=. 2013 , organization=
2013
Show all 42 references
-
[9]
Proceedings of the IEEE , volume=
Taking the human out of the loop: A review of Bayesian optimization , author=. Proceedings of the IEEE , volume=. 2015 , publisher=
2015
-
[10]
Advances in neural information processing systems , volume=
Non-stationary spectral kernels , author=. Advances in neural information processing systems , volume=
-
[11]
Advances in Neural Information Processing Systems 30 (NeurIPS) , year =
Spectral Mixture Kernels for Multi-Output Gaussian Processes , author =. Advances in Neural Information Processing Systems 30 (NeurIPS) , year =
-
[12]
Proceedings of the 32nd International Conference on Machine Learning , series =
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) , author =. Proceedings of the 32nd International Conference on Machine Learning , series =. 2015 , month =
2015
-
[13]
, title =
Stein, Michael L. , title =. 1999 , doi =
1999
-
[14]
Artificial intelligence and statistics , pages=
Student-t processes as alternatives to Gaussian processes , author=. Artificial intelligence and statistics , pages=. 2014 , organization=
2014
-
[15]
Advances in Neural Information Processing Systems , volume=
Beyond Gaussian processes: On the distributions of infinite networks , author=. Advances in Neural Information Processing Systems , volume=
-
[16]
Transactions of the American Mathematical Society , volume=
Asymptotic behavior of the eigenvalues of certain integral equations , author=. Transactions of the American Mathematical Society , volume=. 1963 , publisher=
1963
-
[17]
Advances in Neural Information Processing Systems , volume=
Band-limited Gaussian processes: The sinc kernel , author=. Advances in Neural Information Processing Systems , volume=
-
[18]
Advances in Neural Information Processing Systems , volume=
Bayesian optimization of risk measures , author=. Advances in Neural Information Processing Systems , volume=
-
[19]
arXiv preprint arXiv:1506.02236 , year=
Generalized spectral kernels , author=. arXiv preprint arXiv:1506.02236 , year=
-
[20]
The Journal of Machine Learning Research , volume=
Sparse spectrum Gaussian process regression , author=. The Journal of Machine Learning Research , volume=. 2010 , publisher=
2010
-
[21]
Advances in neural information processing systems , volume=
Scalable Levy process priors for spectral kernel learning , author=. Advances in neural information processing systems , volume=
-
[22]
1994 , publisher=
Stable non-Gaussian random processes: stochastic models with infinite variance , author=. 1994 , publisher=
1994
-
[23]
International conference on artificial intelligence and statistics , pages=
High-dimensional Bayesian optimization via additive models with overlapping groups , author=. International conference on artificial intelligence and statistics , pages=. 2018 , organization=
2018
-
[24]
International conference on machine learning , pages=
High dimensional Bayesian optimisation and bandits via additive models , author=. International conference on machine learning , pages=. 2015 , organization=
2015
-
[25]
Artificial Intelligence and Statistics , pages=
Discovering and exploiting additive structure for Bayesian optimization , author=. Artificial Intelligence and Statistics , pages=. 2017 , organization=
2017
-
[26]
International Conference on Artificial Intelligence and Statistics , pages=
On information gain and regret bounds in gaussian process bandits , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=
2021
-
[27]
International Conference on Machine Learning , pages=
On kernelized multi-armed bandits , author=. International Conference on Machine Learning , pages=. 2017 , organization=
2017
-
[28]
arXiv preprint arXiv:2109.14074 , year=
Gaussian processes with spectral delta kernel for higher accurate potential energy surfaces for large molecules , author=. arXiv preprint arXiv:2109.14074 , year=
-
[29]
2017 IEEE International Conference on Robotics and Automation (ICRA) , pages=
Learning composable models of parameterized skills , author=. 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2017 , organization=
2017
-
[30]
Journal of complexity , volume=
Scrambling Sobol'and Niederreiter--Xing Points , author=. Journal of complexity , volume=. 1998 , publisher=
1998
-
[31]
2020 , publisher=
Univariate stable distributions , author=. 2020 , publisher=
2020
-
[32]
ACM Transactions on mathematical software (TOMS) , volume=
Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , author=. ACM Transactions on mathematical software (TOMS) , volume=. 1997 , publisher=
1997
-
[33]
arXiv preprint arXiv:1412.6980 , year=
Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=
-
[34]
and Bingham, D
Surjanovic, S. and Bingham, D. , title =. 2013 , note =
2013
-
[35]
Advances in neural information processing systems , volume=
Random features for large-scale kernel machines , author=. Advances in neural information processing systems , volume=
-
[36]
Computers & Geosciences , volume=
Efficient computation of maximum likelihood estimators in a spatial linear model with power exponential covariogram , author=. Computers & Geosciences , volume=. 2001 , publisher=
2001
-
[37]
Journal of the Royal Statistical Society Series C: Applied Statistics , volume=
Model-based geostatistics , author=. Journal of the Royal Statistical Society Series C: Applied Statistics , volume=. 1998 , publisher=
1998
-
[38]
International conference on machine learning , pages=
Input warping for Bayesian optimization of non-stationary functions , author=. International conference on machine learning , pages=. 2014 , organization=
2014
-
[39]
Advances in neural information processing systems , volume=
Nonstationary covariance functions for Gaussian process regression , author=. Advances in neural information processing systems , volume=
-
[40]
arXiv preprint arXiv:2102.06571 , year=
Bayesian neural network priors revisited , author=. arXiv preprint arXiv:2102.06571 , year=
-
[41]
The Annals of Statistics , volume=
Heavy-tailed Bayesian nonparametric adaptation , author=. The Annals of Statistics , volume=. 2024 , publisher=
2024
-
[42]
arXiv preprint arXiv:2305.10664 , year=
Posterior inference on shallow infinitely wide Bayesian neural networks under weights with unbounded variance , author=. arXiv preprint arXiv:2305.10664 , year=
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.