REVIEW 3 major objections 6 minor 1 cited by
A Bayesian Lasso based Sparse Learning Model
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper proposes BLS, a sparse Bayesian learning model whose pruning threshold is set by the noise variance, so it prunes exactly zero weights and estimates the noise level accurately.
desk verdict A legitimate but incremental variant of Fast Laplace—noise-scaled prior is a real difference, the threshold math checks out, but missing hyperparameter values and code make the empirical claims unverifiable as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the hierarchical scale-mixture representation of the Laplace prior: each weight $w_i$ gets a Gaussian prior $w_i \mid \tau_i, \sigma^2 \sim N(0, \tau_i \sigma^2)$ with $\tau_i \sim \mathrm{Exponential}(\lambda/2)$, so that integrating out $\tau_i$ recovers the Laplace density. The optimization machinery is the fast marginal likelihood maximization that updates one $\tau_i$ at a time; the derivative of the log marginal likelihood with respect to $\tau_i$ is a quadratic, yielding the closed-form update and the pruning threshold in Equation (17). The decisive feature is that $\sigma^2$ enters that threshold, which is what differentiates BLS from the Fast Laplace method. The estimate of $\sigma^2$ in Equation (19) is derived from an inverse-Gamma prior with hyperparameters $c$ and $d$.
What would settle it
Run the one-dimensional Sinc experiment at $\sigma = 0.5$ while varying $c$ and $d$ in Equation (19) over a wide range; if the estimated $\sigma$ and the mean squared error move substantially away from the paper's reported values, or if holding $\sigma^2$ fixed at the true value changes which basis functions are pruned, the noise-robustness claim does not hold as stated.
Extended reading notes
Core claim
On its own terms, the paper establishes that the Bayesian Lasso's Laplace prior, written as a scale mixture of Gaussians with per-weight variance hyperparameters $\tau_i$, can be optimized by fast marginal likelihood maximization instead of Gibbs sampling. The optimization yields a closed-form pruning rule: $\tau_i = 0$ whenever $q_i^2 - s_i < \lambda \sigma^2$, which after rescaling reads $\tilde{q}_i^2 - \tilde{s}_i < \lambda$; because $\sigma^2$ appears explicitly, the threshold grows with the noise level and the model prunes more aggressively in noisy data. Simulations with one- and two-dimensional Sinc functions at several noise levels show BLS achieving the lowest average mean squared error and the most accurate estimates of the noise standard deviation, while the comparison methods give heavily biased $\sigma$ estimates. Applied to variable selection, the same algorithm selects a small, stable set of variables on the diabetes and mammalian eye gene-expression data, with prediction error competitive with the Lasso and the Bayesian Lasso.
Load-bearing premise
The claimed noise robustness and accurate noise-variance estimation rest on the inverse-Gamma prior for $\sigma^2$ in Equation (19), whose hyperparameters $c$ and $d$ the paper never specifies; if those values are chosen poorly, the pruning threshold inherits the error.
Editorial extensions
If this is right
- If BLS works as claimed, a user can obtain a sparse kernel model without running a Gibbs sampler, which removes the main computational bottleneck of the Bayesian Lasso.
- Because the pruning threshold scales with $\sigma^2$, the number of retained relevance vectors should adapt to noise automatically, giving denser fits for clean data and sparser fits for noisy data.
- The predictive distribution in Equations (24)-(25) supplies both a point prediction and a prediction interval in closed form, so the method is usable for online or repeated prediction tasks.
- The same algorithm, with the kernel matrix replaced by the design matrix, yields a variable-selection procedure that achieves exact zero coefficients, something the Gibbs-sampling Bayesian Lasso does not do.
Reading between the lines
- The authors leave implicit that BLS is a natural candidate for compressive sensing, since Fast Laplace is already used for signal reconstruction and BLS adds a noise-dependent pruning rule to that setting.
- An untested implication is that the reported advantage in estimating $\sigma^2$ may depend on the unspecified inverse-Gamma hyperparameters $c$ and $d$; varying them would show how much of the advantage is structural rather than tuned.
- Because the pruning criterion reduces to $\tilde{q}_i^2 - \tilde{s}_i < \lambda$ when the noise scaling is factored out, BLS can be viewed as a bridge between Lasso-type $\lambda$ shrinkage and RVM-type marginal likelihood pruning, suggesting a unified way to relate regularization strength to noise level.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a sparse Bayesian learning method, BLS, that combines the hierarchical Laplace-prior formulation of the Bayesian Lasso (Park and Casella 2008) with the fast marginal-likelihood maximization of Tipping and collaborators. The weight prior is scaled by the noise variance, and the authors derive an analytical pruning condition (Eq. 17) under which an individual hyperparameter is set to zero. They also give updates for λ (Eq. 18) and σ² (Eq. 19), and they compare BLS with the RVM and the Fast Laplace method on one- and two-dimensional Sinc simulations, as well as with the Bayesian Lasso and Lasso on two real datasets. The central claims are that BLS is sparse, fast, robust to noise, and provides more stable estimates of the noise variance than the compared methods.
Significance. If the empirical claims hold, the paper offers a conceptually simple and computationally fast sparse kernel method whose pruning threshold automatically adapts to the noise level, and it gives a useful alternative for estimating σ² in sparse Bayesian regression. The analytical derivation in Section 2.1 is internally consistent; I checked the derivative of the marginal log-likelihood, the quadratic form, and the threshold condition, and they are correct apart from minor typos. The paper does not ship code, and the missing hyperparameter values are a serious reproducibility gap. The noise robustness is a direct design consequence of the σ²-scaled prior rather than an independent prediction, so circularity is not a concern; the main risk is empirical reproducibility.
major comments (3)
- [Section 3, Eq. (19)] The update for σ² uses inverse-Gamma hyperparameters c and d, but the manuscript never states the values used in the simulations, and the corresponding hyperparameters a and b in Eq. (18) are only described as being "set to low values." Because the pruning rule in Eq. (17) depends directly on σ², the retained basis functions and therefore every row of Tables 1, 2, and 3 depend on these unspecified choices. The reported advantage of BLS in estimating σ² could be an artifact of an unreported prior setting. Please report c, d, a, and b, and provide either code or a complete algorithmic protocol; without this, the central empirical claims are not reproducible.
- [Tables 1 and 3] The BLS estimates of σ at the lowest noise level are several times the true value: Table 1 reports 0.029 for true σ = 0.01 and Table 3 reports 0.053 for true σ = 0.01. This weakens the blanket statements in Section 1.3 and Section 5 that BLS gives "more precise" or "closest" estimates of the noise variance. The paper should qualify these claims and discuss the upward bias at low noise, including how it depends on the unspecified hyperparameters c and d.
- [Section 3, Tables 1–3] The simulation MSE is reported as ||y* − f(x)||²/N, but the text does not state whether y* is evaluated on the training grid or on an independent test grid. Since the basis functions are centered at training inputs, an in-sample reconstruction error is not the same as predictive performance. Please define the evaluation points explicitly and, ideally, report out-of-sample error as well.
minor comments (6)
- [Section 2.1, after Eq. (16)] The expression "L(τ)/dτ_i" should read "dL(τ)/dτ_i" in the sentence describing the derivative at τ_i = 0.
- [Section 2.1, threshold discussion] The sentence "Θ2 < s_i² + 2s_iλσ⁻²" should state "√Θ < s_i² + 2s_iλσ⁻²".
- [Algorithm 1] The equality case q_i² − s_i = λσ⁻² is not handled in the if-else structure; although it has measure zero, the implementation needs a convention for this boundary.
- [Section 4.1.2] The sentence "20 genes for 120 samples" appears inconsistent with the description of 200 predictors; this is likely a typo and should be corrected.
- [Figure 1] The shaded region is labeled a "95% confidence interval" but it is constructed from the predictive variance in Eq. (25); it should be called a predictive interval.
- [Eq. (23)] The factorization leading to "σ⁻² q̃_i² − s̃_i ≤ λ" involves dividing both sides of the previous inequality by σ⁻²; this step should be stated explicitly for clarity.
Circularity Check
No circularity: the BLS derivation is self-contained and the empirical comparisons use external benchmarks.
full rationale
The derivation chain is not circular. The BLS model is constructed from the hierarchical Bayesian Lasso prior of Park and Casella (2008) and the fast type-II maximum-likelihood estimation of Tipping et al. (2003), both external to the present authors. The central analytical result, the pruning condition q_i^2 - s_i <= lambda*sigma^{-2} in Equation (17), follows from differentiating the log marginal likelihood and solving for tau_i, an internally consistent derivation. The dependence of the pruning threshold on the noise variance is a direct analytical consequence of the sigma^2-scaled Laplace prior, not a fitted quantity later renamed as a prediction. The simulation and real-data comparisons in Sections 3 and 4 are evaluated against independently implemented benchmarks (FRVM, FLAP, Bayesian Lasso, Lasso) on generated and public data, so the empirical claims are externally falsifiable rather than forced by construction. The unspecified inverse-Gamma hyperparameters c and d in Equation (19) impair reproducibility of the reported sigma^2 estimates, but an unspecified prior hyperparameter is a reporting defect, not a circular step, because no claimed result equals an input by definition. No self-referential citation chain, imported uniqueness theorem, or ansatz-by-citation is present, so the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Gamma prior shape a for lambda =
Not specified
- Gamma prior rate b for lambda =
Not specified
- Inverse-gamma shape c for sigma squared =
Not specified
- Inverse-gamma rate d for sigma squared =
Not specified
- Initial noise variance sigma squared =
var(y) x 0.1
assumptions (4)
- domain assumption Regression model y = Phi w + epsilon with epsilon ~ N(0, sigma squared I)
- standard math Laplace prior equals scale mixture of Gaussians with exponential mixing density
- domain assumption Hyperparameters tau, sigma squared, lambda are point-estimated by type-II maximum likelihood
- domain assumption Coordinate-ascent algorithm converges to a sparse local optimum
Cite this review
Pith. "Pith review of A Bayesian Lasso based Sparse Learning Model." pith.science (2026). https://pith.science/paper/NQBLPQOL
@misc{pith2026190807220,
author = {Pith},
title = {Pith review of: A Bayesian Lasso based Sparse Learning Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/NQBLPQOL}},
note = {Machine review of arXiv:1908.07220}
}
read the original abstract
The Bayesian Lasso is constructed in the linear regression framework and applies the Gibbs sampling to estimate the regression parameters. This paper develops a new sparse learning model, named the Bayesian Lasso Sparse (BLS) model, that takes the hierarchical model formulation of the Bayesian Lasso. The main difference from the original Bayesian Lasso lies in the estimation procedure; the BLS method uses a learning algorithm based on the type-II maximum likelihood procedure. Opposed to the Bayesian Lasso, the BLS provides sparse estimates of the regression parameters. The BLS method is also derived for nonlinear supervised learning problems by introducing kernel functions. We compare the BLS model to the well known Relevance Vector Machine, the Fast Laplace method, the Byesian Lasso, and the Lasso, on both simulated and real data. The numerical results show that the BLS is sparse and precise, especially when dealing with noisy and irregular dataset.
Figures
Forward citations
Cited by 1 Pith paper
-
Proximal Iteration for Nonlinear Adaptive Lasso
A closed-form proximal operator for jointly updating coefficients and their adaptive Lasso penalties enables debiased variable selection with arbitrary sparsity structure in nonlinear models.
Reference graph
Works this paper leans on
-
[1]
Agarwal, A. & Triggs, B. (2005), ‘Recovering 3d human pose from monocular images’,IEEE transactions on pattern analysis and machine intelligence 28(1), 44–58. Andrews, D. F. & Mallows, C. L. (1974), ‘Scale mixtures of normal distributions’,Journal of the Royal Statistical Society: Series B (Methodological) 36(1), 99–102. Ashburner, J. (2007), ‘A fast diff...
arXiv 2005
-
[307]
Schölkopf, B., Burges, C. J., Smola, A. J. et al. (1999), Advances in kernel methods: support vector learning , MIT press. Seeger, M. (2000), ‘Relationships between gaussian processes, support vector machines and smoothing splines’, Machine Learning . Smola, A. J., Schölkopf, B. & Müller, K.-R. (1998), ‘The connection between regularization operators and ...
work page 1999
-
[968]
Park, T. & Casella, G. (2008), ‘The bayesian lasso’,Journal of the American Statistical Association 103(482), 681–686. Rasmussen, C. E. & Quinonero-Candela, J. (2005), Healing the relevance vector machine through augmentation, in ‘Proceedings of the 22nd international conference on Machine learning’, ACM, pp. 689–696. Scheetz, T. E., Kim, K.-Y . A., Swide...
work page 2008
-
[2346]
Krishnapuram, B., Carin, L., Figueiredo, M. A. & Hartemink, A. J. (2005), ‘Sparse multinomial logistic regression: Fast algorithms and generalization bounds’,IEEE transactions on pattern analysis and machine intelligence 27(6), 957–
work page 2005
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.