Linear regression in the Bayesian framework

GdR MASCOT-NUM); Thierry A. Mara (PIMENT

REVIEW 3 major objections 3 minor 9 references

Linear regression in the Bayesian framework

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that ordinary least squares, weighted least squares, ridge regression, and LASSO are all the same Bayesian calculation run with different priors on the coefficients.

desk verdict A clear but non-novel tutorial whose OLS and ridge sections are reliable, whose LASSO derivation is mathematically wrong, and whose KIC application to LASSO does not survive scrutiny. read the letter →

arxiv 1908.03329 v1 pith:ECDDW3GS submitted 2019-08-09 stat.ME

Thierry A. Mara (PIMENT , GdR MASCOT-NUM) This is my paper · ORCID

classification stat.ME MSC 62J0562F1562J07

keywords linearregressionBayesianinferencegeneralizedleastsquaresridgeLASSOKashyapinformationcriterionmodelselectionmaximumaposterioriestimation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

These notes aim to show that least-squares, ridge, and LASSO regression are not separate algorithms but the same Bayesian calculation performed with different priors on the coefficients. Under a Gaussian error model, each choice of prior—flat, Gaussian, or Laplace—turns the log-posterior into a quadratic form, so completing the square yields the estimator as the posterior mean and its covariance in closed form. The paper also derives the Kashyap information criterion from the same setup, giving a rule for choosing which coefficient subset to keep. If the derivation is right, a practitioner gets one framework that reproduces several textbook methods and reveals what each implicitly assumes before seeing data. The LASSO part rests on an extra sign-constancy assumption that the author flags and says must be checked afterward.

What carries the argument

The load-bearing object is completing the square in the exponent of a Gaussian likelihood times a prior: rewriting $(y-\Psi a)^T C_M^{-1}(y-\Psi a)+(a-a_0)^T C_{aa}^{-1}(a-a_0)$ as $(a-\hat{a})^T \hat{C}_{aa}^{-1}(a-\hat{a})+\text{const}$ turns the posterior into $N(\hat{a},\hat{C}_{aa})$. This one algebraic manipulation produces the MLE/MAP formulas for every method in the paper. For the Laplace prior the same manipulation is applied to $b=a-a_0$ under the condition that $\operatorname{sgn}(b)$ stays fixed, and the posterior covariance is taken from the least-squares term alone.

What would settle it

For a small dataset, set $a_0=0$ and $\Lambda_{aa}=\lambda I$, then compare the closed-form $\hat{a}$ from Eq. (16) with the mode obtained by directly maximizing the posterior density in Eq. (14) on a fine grid for one or two coefficients; if the sign of $\hat{a}-a_0$ does not match the sign of the posterior mode, the sign-constancy assumption fails and Eq. (16) is not the LASSO MAP.

Watch

Extended reading notes

Core claim

The paper's claim is that the usual linear-regression toolbox is a single Bayesian inference: choose a Gaussian likelihood for the error and a prior for the coefficients, and the posterior is Gaussian because the log-posterior is a quadratic form in $a$. Completing the square gives the posterior mean and covariance in closed form. A uniform prior on a rectangular domain yields the maximum-likelihood (weighted or ordinary least-squares) estimate; a Gaussian prior $N(a_0,C_{aa})$ yields the ridge-regression MAP; a Laplace prior yields LASSO, provided the sign of $a-a_0$ is constant across the posterior. The same quadratic view supports a Laplace-approximation derivation of the Kashyap information criterion for comparing candidate coefficient subsets, with the best model being the one with lowest KIC.

Load-bearing premise

The LASSO derivation assumes that the sign of each shifted coefficient $a-a_0$ stays fixed across the posterior and that the posterior is then Gaussian, an assumption the author says must be checked after computing the estimate; if it fails, the closed-form LASSO solution and the KIC built on it are not the true Bayesian MAP.

Editorial extensions

If this is right

Ordinary and weighted least squares emerge as maximum-likelihood or MAP estimates under a flat prior, so no separate theory is needed to justify them.
Ridge regression with penalty $\lambda$ is the posterior mean of a Gaussian prior with covariance $\lambda I$, making the regularization strength a prior variance.
Under the paper's sign-constancy assumption, LASSO is the MAP estimate under a Laplace prior, given as a shifted least-squares solution with a sign-dependent correction.
The Kashyap information criterion, derived by Laplace approximation around the MAP, gives a consistent rule for selecting among linear Gaussian candidate models: choose the lowest KIC.
Because all estimators come from the same Bayesian recipe, a user can move between methods by changing only the prior, not the inference machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the unification holds, the practical choice between OLS, ridge, and LASSO becomes a choice of prior belief about coefficients, which gives a principled way to set penalties through prior widths rather than by cross-validation alone.
The same completing-the-square machinery extends to any prior whose log-density is quadratic in the coefficients, suggesting a general recipe for building new shrinkage estimators; the paper does not pursue this generalization.
For the Laplace prior, a fully Bayesian treatment would keep the non-Gaussian posterior, whose mode is the usual LASSO estimator; a corrected version would replace the Gaussian approximation with direct optimization and would need the true posterior curvature for KIC.
A testable extension is to compare KIC computed from the Gaussian approximation against numerically integrated model evidence for small models; the paper does not include such a check.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this if you need a clean derivation of Bayesian OLS and ridge. The paper is a tutorial, not a research contribution: it re-derives known formulas. The first two sections and the KIC derivation for the Gaussian case are fine. The algebra is clear and the notation is consistent. For someone who wants to see how a uniform prior leads to the MLE, and a Gaussian prior to ridge, these notes work.

The soft spot is Section 2.4. The Laplace-prior posterior is not a single Gaussian. The author fixes sgn(a - a0), completes the square, and declares p(a|y,A) Gaussian with mean a0 + H^{-1}(...) and covariance H^{-1}. That is the posterior only if the sign pattern is known in advance, and even then it is an orthant-truncated Gaussian, not a global Gaussian. In the sparse cases LASSO is supposed to produce, some coefficients sit at the prior mean, where sgn is undefined; setting it to zero silently drops the l1 penalty for that coordinate. So Eq. (16) is not the LASSO MAP. The N.B. admits the sign is unknown, which is exactly the problem: the Gaussian conclusion is doing unacknowledged work.

That error propagates. Section 3.1 derives KIC from a Laplace approximation around the MAP. For a Gaussian posterior this is standard and correct. For the LASSO case, at a kink the Hessian does not exist and the posterior is not Gaussian, so the KIC formula does not apply. The author's own conclusion section says BSPCE uses KIC with a Gaussian prior, so this flaw does not affect that application.

One minor point: in Section 2.1 the uniform prior is on a bounded rectangle; the posterior is a truncated Gaussian, not a full Gaussian. The N.B. acknowledges the support restriction, but Eq. (7) drops the indicator.

The citation pattern is fine. The self-citation to Shao et al. is only in the concluding remark and is not load-bearing.

Who is this for? Someone who wants a quick, self-contained derivation of OLS and ridge from a Bayesian viewpoint. Do not use it as a reference for the Bayesian interpretation of LASSO; Park and Casella (2008) is the right source. I would not cite it, but I would consider it for a reading group as a cautionary example of what goes wrong when you linearize a nonsmooth prior. If a teaching-oriented or arXiv-focused venue asks me, yes, referee it: it is short, readable, and the LASSO mistake is worth catching. But it should not be published as research, and as written the LASSO section needs major revision before the tutorial is trustworthy.

Referee Report

3 major / 3 minor

Summary. These notes are a pedagogical derivation of Bayesian linear regression. Starting from the linear model y = Ψa + ε with a Gaussian likelihood, the paper derives the posterior distribution of the coefficients under a uniform prior (interpreted as ordinary/generalized least squares), a Gaussian prior (ridge regression), and a Laplace prior (LASSO). It then derives the Kashyap information criterion (KIC) by Laplace approximation of the Bayesian model evidence, proposing KIC for model selection among different subsets A of basis functions. The abstract states that the article presents the author's understanding of how these methods are "better unified in the Bayesian framework," with explicit formulas derived for each estimator.

Significance. If correct, the paper would be a useful tutorial for practitioners who want to see common linear-regression estimators derived from a single Bayesian perspective. The OLS and ridge derivations are standard, the algebra is shown in detail, and the text cites the relevant historical literature. The treatment of the LASSO, however, is not the standard LASSO solution and is mathematically incorrect as written; the KIC derivation is also only valid for smooth posterior densities and cannot be applied to the LASSO case. Since the central claim of the paper is that OLS, ridge, LASSO, and KIC all fit within one framework, this is not a local blemish: the LASSO and KIC legs of the unification need substantial correction. The paper does not report numerical experiments or new statistical results, so its value is essentially tutorial, and tutorial value is only realized if the mathematics is sound.

major comments (3)

[Section 2.4, Eqs. (14)-(16)] The derivation leading to Eq. (16) is not a derivation of the LASSO posterior. The sign of b = a - a0 is a function of b, and fixing s = sgn(b) before completing the square and then declaring the posterior to be Gaussian is not valid. The exact posterior under a Laplace prior is a mixture of orthant-truncated Gaussians, one for each sign pattern, with different normalizing constants on each orthant; it is not a single Gaussian. Consequently the mean and covariance in Eq. (16) are not the LASSO MAP and its posterior covariance. A concrete failure occurs in the common sparse situation where the MAP has a coefficient exactly equal to a0: there sgn(b) is undefined, and if one treats it as zero, the l1 penalty for that coordinate is silently removed. The paper's own N.B. after Eq. (16) states that one cannot guess sgn(a - a0) before computing a-hat, which indicates that the Gaussian conclusion in Eq. (16) rests on an unverified assumption, not on a theorem.
[Section 3.1, Eqs. (18)-(23)] The KIC derivation assumes twice differentiability of log p(y, a | A) at the MAP, since Eq. (18) uses a Taylor expansion and Eq. (19) defines Σ^{-1} as the negative Hessian at a-hat. For the Laplace prior used in Section 2.4, the log-joint density is not differentiable at any coefficient value equal to the prior location a0, which is precisely the situation that makes LASSO solutions sparse. Therefore Eq. (22) is not a valid approximation of the Bayesian model evidence for LASSO models. The claim in Section 3.1 that the KIC at the MAP applies generally to the models considered in the paper is too broad; it should be restricted to likelihood/prior combinations for which the posterior density is smooth at the mode, such as the Gaussian-prior ridge case. The closing remark in Section 4 that the KIC in the BSPCE application is implemented with a Gaussian prior is consistent with this limitation, but the main text does not flag it.
[Section 2.1, Eq. (7) and N.B.] The treatment of the uniform prior is incorrect as written. If the prior is uniform on a bounded rectangular domain Ω, the posterior is proportional to the Gaussian likelihood restricted to Ω, which is a truncated Gaussian. It is not the Gaussian density N(a-tilde, C-tilde_aa) even when the unconstrained mean a-tilde lies inside Ω, because truncation changes the normalizing constant and the covariance; and when a-tilde lies outside Ω, the posterior is not zero. The N.B. after Eq. (7) asserts that the posterior is N(a-tilde, C-tilde_aa) if a-tilde ∈ Ω and zero otherwise, which is false. The standard OLS result can be recovered by taking an improper uniform prior over the whole coefficient space, but the manuscript as written bounds Ω first and then ignores the truncation. This needs to be corrected because the OLS section depends on it.

minor comments (3)

[Section 2.3, Eq. (13)] There is a typographical error: "m¯C^{-1}_{aa}" should read "C^{-1}_{aa}" in the expression for the MAP estimate.
[References] The citation "G. Schwartz, Estimating the dimensions of a model" should be to Gideon Schwarz (1978); the name is spelled Schwarz.
[Section 2.2, Eqs. (9)-(10)] The notation in the transition from the likelihood to the Gamma distribution is confusing: the text writes p(σ^2 | ·) and then p(σ^{-2} | ·), and states that the mode is σ-tilde^2. Since the Gamma distribution is stated for σ^{-2}, the mode of that Gamma distribution is 1/σ-tilde^2, not σ-tilde^2. The parameterization should be made explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a self-contained Bayesian derivation of known regression estimators and KIC.

full rationale

The paper's claims are analytic derivations: under stated likelihood and prior assumptions, it completes squares to obtain Gaussian posteriors (Eqs. 7, 13, and 16) and applies a Laplace approximation to derive the Kashyap information criterion (Eqs. 20-23). None of these steps fits a parameter to a target quantity and then re-presents that fitted value as a prediction; the OLS, ridge, and KIC results follow by algebra from the stated Gaussian model. The LASSO section introduces an explicit assumption that the sign of b remains unchanged in the posterior, labels the resulting object a Gaussian approximation, and includes an N.B. requiring an a posteriori consistency check; this is an acknowledged approximation and not a tautological reduction. The only self-citation, Shao et al. [2017], appears in the Concluding Remarks to describe where the derived formulas were subsequently used; it is not load-bearing for any derivation in this paper. Concerns about the validity of the fixed-sign LASSO approximation and about applying KIC at a non-smooth MAP are correctness or robustness concerns, not circularity. Therefore the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters fitted to data; all hyperparameters (Caa, Lambda, a0) are part of the framework and not estimated. The central axioms are the likelihood and prior choices, which are standard but determine the form of each estimator. No new entities are postulated.

assumptions (5)

domain assumption The error vector epsilon follows a zero-mean Gaussian with known covariance C_M
This is the likelihood assumption stated at the beginning of Section 2 (Eq. 5).
domain assumption The prior on coefficients is uniform over a rectangular domain Omega
Used in Section 2.1 to derive the MLE/OLS estimator.
domain assumption The prior on coefficients is Gaussian with mean a0 and covariance Caa
Used in Section 2.3 to derive ridge regression.
domain assumption The prior on coefficients is Laplace with diagonal scale Lambda
Used in Section 2.4 to derive a LASSO-type estimator.
domain assumption The Laplace approximation to the posterior is valid
Used in Section 3.1 to derive KIC, but this is not valid for non-Gaussian posteriors such as the Laplace-prior case.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Linear regression in the Bayesian framework." pith.science (2026). https://pith.science/paper/ECDDW3GS

@misc{pith2026190803329,
  author       = {Pith},
  title        = {Pith review of: Linear regression in the Bayesian framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ECDDW3GS}},
  note         = {Machine review of arXiv:1908.03329}
}

read the original abstract

These notes aim at clarifying different strategies to perform linear regression from given dataset. Methods like the weighted and ordinary least squares, ridge regression or LASSO are proposed in the literature. The present article is my understanding of these methods which are, according to me, better unified in the Bayesian framework. The formulas to address linear regression with these methods are derived. The KIC for model selection is also derived in the end of the document.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 8 canonical work pages

[1]

Efron and R

B. Efron and R. Tibshirani. An introduction to bootstrap. Chapman & Hall, 1 edition, 1993. ISBN 9780412042317

work page 1993
[2]

A. E. Hoerl and R. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12: 0 55--67, 1970

work page 1970
[3]

Tibshirani

R. Tibshirani. Regression shrinkage and selection via the LASSO . Journal of the Royal Statistical Society, Serie B, 58 0 (1): 0 267--288, 1996

work page 1996
[4]

Schwartz

G. Schwartz. Estimating the dimensions of a model. Annal of Statistics, 6: 0 461--464, 1978

work page 1978
[5]

H. Akaike. Information theory as an extension of the maximum likelihood principle. In Second International Symposium on Information Theory, pages 267--281. Akademiai Kiado, 1973

work page 1973
[6]

D. J. Spiegelhalter, N. G. Best, B. P. Carlin, and A. van der Linde . Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society, 64 0 (4): 0 583--639, October 2002. doi:0.1111/1467-9868.00353

arXiv 2002
[7]

R. L. Kashyap. Optimal choice of AR and MA parts in autoregressive moving average models. IEEE Trans. Pattern Anal. Machine Intell., 4 0 (2): 0 99--104, 1982

work page 1982
[8]

o niger, T. W \

A. Sch \"o niger, T. W \"o hling, L. Samaniego, and W. Novak. Model selection on solid ground: Rigourous comparison of nine ways to evaluate B ayesian evidence. Water Resources Research, 50: 0 9484--9513, 2014. doi:10.1002/2014WR016062

work page doi:10.1002/2014wr016062 2014

Show all 9 references

[9]

Q. Shao, A. Younes, M. Fahs, and T. A. Mara. Bayesian sparse polynomial chaos expansion for global sensitivity analysis. Computer Methods in Applied Mechanics & Engineering, 318: 0 474--496, 2017

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Efron and R

B. Efron and R. Tibshirani. An introduction to bootstrap. Chapman & Hall, 1 edition, 1993. ISBN 9780412042317

work page 1993

[2] [2]

A. E. Hoerl and R. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12: 0 55--67, 1970

work page 1970

[3] [3]

Tibshirani

R. Tibshirani. Regression shrinkage and selection via the LASSO . Journal of the Royal Statistical Society, Serie B, 58 0 (1): 0 267--288, 1996

work page 1996

[4] [4]

Schwartz

G. Schwartz. Estimating the dimensions of a model. Annal of Statistics, 6: 0 461--464, 1978

work page 1978

[5] [5]

H. Akaike. Information theory as an extension of the maximum likelihood principle. In Second International Symposium on Information Theory, pages 267--281. Akademiai Kiado, 1973

work page 1973

[6] [6]

D. J. Spiegelhalter, N. G. Best, B. P. Carlin, and A. van der Linde . Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society, 64 0 (4): 0 583--639, October 2002. doi:0.1111/1467-9868.00353

arXiv 2002

[7] [7]

R. L. Kashyap. Optimal choice of AR and MA parts in autoregressive moving average models. IEEE Trans. Pattern Anal. Machine Intell., 4 0 (2): 0 99--104, 1982

work page 1982

[8] [8]

o niger, T. W \

A. Sch \"o niger, T. W \"o hling, L. Samaniego, and W. Novak. Model selection on solid ground: Rigourous comparison of nine ways to evaluate B ayesian evidence. Water Resources Research, 50: 0 9484--9513, 2014. doi:10.1002/2014WR016062

work page doi:10.1002/2014wr016062 2014