Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Variational Bayes on Manifolds

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Variational Bayes can be run directly on curved parameter spaces, with proven convergence to a local mode of the evidence lower bound.

desk verdict Solid framework for natural-gradient VB on manifolds, but the advertised O(1/T^{2−2ε}) rate for strongly retraction-convex ELBOs doesn't follow from the proof. read the letter →

arxiv 1908.03097 v3 pith:WOLHUZGI submitted 2019-08-08 cs.LG stat.MEstat.ML

classification cs.LGstat.MEstat.ML
keywords variationalBayesnaturalgradientRiemannianmanifoldstochasticdescentFisherinformationevidencelowerboundGaussianapproximationWishart
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most variational Bayes methods optimize the evidence lower bound (ELBO) in Euclidean parameter spaces, which forces constraints like positive-definite covariance matrices into awkward reparameterizations. This paper tries to remove that restriction by treating the variational parameter space as a Riemannian manifold and using the natural gradient, preconditioned by the Fisher information matrix, as the search direction. The authors prove that the resulting manifold stochastic-gradient-with-momentum algorithm converges to a local mode of the ELBO, with rate $O(1/\sqrt{T})$ in general and $O(1/T^{2-2\epsilon})$ when the ELBO is strongly retraction-convex. They instantiate the method for Gaussian and inverse-Wishart variational families and show in experiments that it is stable and less sensitive to initialization than existing VB algorithms. If the proof holds, a broad class of constrained variational inference problems becomes unconstrained geometric optimization.

What carries the argument

The central object is the variational family $\mathcal{N}=\{q_\lambda\}$ viewed as a Riemannian submanifold or quotient manifold of the parameter space, equipped with the Fisher-Rao metric $\langle\zeta,\xi\rangle=\zeta^\top I_F(\lambda)\xi$. Lemma 3.1 identifies the natural gradient $I_F(\lambda)^{-1}\nabla L(\lambda)$ with the Riemannian gradient of the lower bound in the ambient manifold; for submanifolds it is projected onto the tangent space, and for quotient manifolds its horizontal lift is the Riemannian gradient. The update uses a retraction $R_\lambda$, a first-order approximation of the exponential map that replaces the Euclidean move $\lambda+\xi$, and a vector transport that pushes momentum vectors between tangent spaces. Theorem 1 provides the convergence machinery: bounded gradients with martingale-difference noise, retraction-Lipschitz gradients, and, for the faster rate, strong retraction-convexity.

What would settle it

Run the Manifold Gaussian VB algorithm on a low-dimensional problem where the exact Fisher information $I_F(\Sigma)$ is computable, compare the exact natural gradient with the approximation $(\Sigma\otimes\Sigma)\nabla_\Sigma L$, and test whether the approximate update still drives $\mathbb{E}|\nabla L|^2$ to zero at the claimed rate; a measured bias that does not shrink with step size, or divergence for some initialization, would falsify the claim that the practical algorithm inherits Theorem 1's guarantees.

Watch

Extended reading notes

Core claim

The central claim is that constrained variational inference is not a special case to be patched with factorizations, but a genuine Riemannian optimization problem. Lemma 3.1 shows that the natural gradient $I_F(\lambda)^{-1}\nabla L(\lambda)$ is the Riemannian gradient of the lower bound when the manifold is equipped with the Fisher-Rao metric, so the steepest-ascent direction automatically respects the information geometry of the variational family. Algorithm 1 iterates $\lambda_{t+1}=R_{\lambda_t}(\epsilon Y_t)$, with $Y_t$ a momentum-averaged estimate of the projected natural gradient; for submanifolds the projection is orthogonal, and for quotient manifolds it is the horizontal lift. Theorem 1 states that with step size $1/\sqrt{T}$ the squared Riemannian gradient norm is driven below $C/\sqrt{T}$, and that strong retraction-convexity improves the expected squared distance to the optimum to $O(T^{2\epsilon-2})$. The Gaussian instantiation uses the full covariance matrix on the manifold of symmetric positive-definite matrices with the approximation $I_F(\Sigma)\approx\Sigma^{-1}\otimes\Sigma^{-1}$, and the Wishart instantiation uses analogous natural-gradient preconditioning; both are tested on data where they converge faster and more stably than previous VB methods.

Load-bearing premise

The practical algorithms inherit the paper's convergence rates only if the approximate natural-gradient formulas are close enough to the true natural gradient; the paper gives no error bound for these approximations.

Editorial extensions

If this is right

  • For a non-convex evidence lower bound, running the manifold VB algorithm for $T$ steps with step size $1/\sqrt{T}$ gives $\min_{t\le T}\mathbb{E}|\nabla L(\lambda_t)|^2\le C/\sqrt{T}$.
  • For a strongly retraction-convex ELBO, choosing step size $1/T^\epsilon$ yields $\mathbb{E}(\lambda_T-\lambda^*)^2\le C_\epsilon T^{2\epsilon-2}$, approaching $O(1/T^2)$ as $\epsilon$ tends to zero.
  • Because the natural gradient is parameterization-invariant, the manifold update is the same geometric object regardless of how the variational family is parameterized.
  • Gaussian variational Bayes can use full covariance matrices on the positive-definite manifold rather than factorized covariance structures, with faster convergence and lower variance across random initializations than a factorized Gaussian VB.
  • The Wishart instantiation matches a gold-standard sequential Monte Carlo posterior estimate for a financial GARCH model while using a small fraction of the compute time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the convergence rates are proved for the exact Riemannian gradient, so an error analysis for the approximate natural gradients, such as $\Sigma^{-1}\otimes\Sigma^{-1}$ in the Gaussian case, would turn the practical algorithms into theorems; until then, the rates are safest read as applying to the idealized update.
  • Editorial inference: the same template, projecting a natural-gradient direction onto the tangent space and retracting, should transfer to other constrained variational families, such as low-rank factor covariances or positive-definite matrices in Gaussian processes, wherever a retraction and vector transport exist.
  • Editorial inference: a direct comparison of updates using the exact Fisher information matrix against the $\Sigma^{-1}\otimes\Sigma^{-1}$ approximation on low-dimensional Gaussian VB would give a testable measure of how much of the observed speedup comes from the approximation rather than from the manifold geometry.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a manifold-based variational Bayes framework. It treats the variational parameter space as a Riemannian manifold, equips it with the Fisher-Rao metric, identifies the natural gradient with the Riemannian gradient (Lemma 3.1), handles submanifold and quotient cases, and runs Riemannian SGD with momentum. It states convergence rates of order O(1/sqrt(T)) for non-convex ELBOs and O(1/T^(2-2ε)) for strongly retraction-convex ELBOs. It then instantiates Manifold Gaussian VB and Manifold Wishart VB, with numerical experiments on logistic regression, GARCH volatility, and covariance estimation, plus comparisons to NAGVAC and SMC.

Significance. If the convergence theorem were correct, this would be a valuable generalization of variational Bayes to constrained parameter spaces, combining information-geometric preconditioning with Riemannian optimization. The geometric derivations in Lemmas 3.1 and 3.2 are clean and self-contained, the algorithmic descriptions are concrete, and the authors provide code and an SMC comparison. However, the advertised fast strongly-convex rate is not supported by the proof as written, and the implemented algorithms rely on unquantified approximations that the theorem does not cover. These issues affect the paper's central guarantees, so the significance is currently conditional on a repair of the proof and a clarification of the scope of the theoretical claims.

major comments (3)
  1. [Section 4, proof of Theorem 1(ii), around Eq. (6.27)] The proof sets rho = 1 - gamma/(4*mu_tilde*(1-zeta)) immediately before Eq. (6.22), so 1-rho = O(gamma). The unrolled recursion then gives EV_T <= rho^T C0 + C*gamma^2/(1-rho), but the next line replaces C*gamma^2/(1-rho) by C*gamma^2. This is algebraically false: C*gamma^2/(1-rho) = 4*C*mu_tilde*(1-zeta)*gamma = O(gamma). With gamma = T^{-epsilon}, the correct noise floor is O(T^{-epsilon}), not O(T^(2*epsilon-2)). A second inconsistency appears in the same paragraph: gamma^2 = T^(-2*epsilon) equals T^(2*epsilon-2) only when epsilon = 1/2. Thus the advertised rate O(1/T^(2-2*epsilon)) in the abstract and Theorem 1 is unsupported by the proof as written.
  2. [Section 5.1, Eqs. (5.2)-(5.4), and Section 5.2, Eqs. (5.7)-(5.8)] The convergence theorem is proved for exact Riemannian gradients with unbiased estimators. The implemented Manifold Gaussian VB uses the approximation IF(Sigma) approx Sigma^{-1} tensor Sigma^{-1}, and the Manifold Wishart VB uses the natural-gradient-like replacements in (5.7) and (5.8). No error bound or perturbation analysis relates these approximate updates to the exact Riemannian gradient required by Theorem 1. Consequently, the stated convergence guarantees do not apply to the algorithms whose numerical behavior is reported. The paper should either prove that these approximations satisfy the theorem's conditions up to controlled errors, or explicitly state that the numerical results are heuristic demonstrations.
  3. [Section 3.2, Remark 3.1] The quotient-manifold construction defines the horizontal space as the orthogonal complement under the Fisher-Rao metric, but the metric is not invariant along equivalence classes, as Remark 3.1 concedes. The claim that equation (3.10) is independent of the representative lambda in [lambda] is asserted rather than proved. Without metric invariance, the horizontal lift and the induced inner product on the quotient manifold are generally representative-dependent, and the usual Riemannian-submersion condition is exactly what is needed to rule this out. This is a gap in the claimed general framework for quotient manifolds, although the paper's numerical experiments do not depend on this part.
minor comments (4)
  1. [Abstract] The abstract names Manifold Neural Net VB as one of the two developed algorithms, but the paper actually develops Manifold Gaussian VB and Manifold Wishart VB; the abstract should be corrected to match the body.
  2. [Figure 3] The legend in Figure 3 reads NAGVAG instead of NAGVAC.
  3. [Section 4, end of proof of Theorem 1(ii)] The theorem statement bounds E(lambda_T - lambda_*)^2, but the proof bounds E(L(lambda_T) - L(lambda_*)) and E|grad L|^2; the strong-convexity inequality linking squared retraction distance to the function suboptimality is not stated, so the final line of the proof does not directly deliver the theorem's claim.
  4. [Section 5.1, Eq. (5.2)] The sentence 'One can derive that IF(Sigma) approx Sigma^{-1} tensor Sigma^{-1}' gives no derivation or citation for the approximation; since this approximation is used in the implemented algorithm, a reference or explicit formula would clarify its accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the manifold VB derivation follows from the Fisher-Rao metric definition and the convergence analysis is an independent extension of prior Riemannian SGD theory; self-citations are not load-bearing.

full rationale

The derivation chain is self-contained. The natural-gradient identity (3.2) and Lemma 3.1 are proved from the definition of the Fisher-Rao metric (3.1), and the projection formulas for submanifolds and quotient manifolds follow from the defining property of a Riemannian gradient; the target conclusions are not used as inputs. Theorem 1 is an extension of the Riemannian SGD analysis in Huang et al. (2015a,b) and Absil et al. (2009), and the proof is carried out in the appendix on the explicit recursion (4.2); the advertised rates are consequences of the proof, not restatements of an assumed bound. The practical approximations in Section 5, such as IF(Σ)≈Σ^{-1}⊗Σ^{-1} in Eq. (5.2), are clearly labeled approximations used to instantiate the algorithm and are not claimed to be derived from Theorem 1; whether they satisfy the theorem's assumptions is a coverage question, not circularity. Self-citations (Tran et al. 2017, 2019, 2021) appear for standard score-function gradient identities and as experimental baselines, and are not load-bearing; the numerical benchmarks also include an analytic inverse-Wishart posterior and an SMC gold standard. The algebraic concern raised about the 1/(1−ρ) term in the strongly retraction-convex rate is a proof-correctness issue, not an equivalence between inputs and outputs, so it does not contribute to the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities or ad hoc latent variables are introduced. The listed free parameters are algorithmic hyperparameters; the most consequential uncharged item is the approximation gap between the practical natural-gradient updates and the exact gradients required by the convergence theorem, along with the relaxed quotient-manifold metric assumption.

free parameters (4)
  • Learning rate epsilon
    Step size in Algorithms 1-2; convergence proof requires specific schedules (gamma = 1/sqrt(T) or 1/T^epsilon), but in experiments epsilon is set by hand and its value is not reported.
  • Momentum weight omega
    Momentum coefficient in Algorithm 1-2; must lie in (0,1) for the proof, but the exact value in experiments is not stated.
  • Number of Monte Carlo samples S for score-function gradient = 100, 1000, or 10000 in experiments
    S=100 for Manifold GVB in the logistic regression experiment, S=1000 for Wishart; a larger S (10,000) is needed without the natural gradient. The gradient estimator's variance, and hence the effective convergence, depends on S.
  • Adaptive method hyperparameters for Wishart nu update
    Section 5.2 recommends ADAM or AdaDelta for the scalar degree-of-freedom update, whose hyperparameters are not specified; convergence theory does not cover this adaptive step.
assumptions (5)
  • domain assumption The variational family {q_lambda} is a smooth Riemannian manifold with the Fisher information matrix IF(lambda) as metric (Section 2, Eq. 2.3).
    The whole natural-gradient construction rests on the Fisher-Rao metric being well-defined and positive definite on the tangent space.
  • domain assumption The score-function gradient estimator is unbiased and has finite variance (Section 4, Eq. 4.2).
    The convergence proof writes the estimator as true gradient plus a martingale difference; this requires unbiasedness and boundedness, which is assumed but not verified.
  • ad hoc to paper The objective satisfies: totally retractive neighborhood of the optimum, L-Lipschitz Riemannian gradient w.r.t. the retraction, and a.s. bounded gradient and noise (Theorem 1).
    These conditions are stated as assumptions in Theorem 1 and are not checked for the ELBO in the numerical examples; the convergence rate depends on them.
  • ad hoc to paper For quotient manifolds, the Riemannian gradient is well-defined even though the Fisher-Rao metric is not invariant along the equivalence class (Remark 3.1).
    Remark 3.1 concedes the standard metric-invariance condition is not satisfied and asserts the gradient is still well-defined via Eq. (3.10); this is a non-standard relaxation and a load-bearing assumption for the quotient construction.
  • ad hoc to paper The approximate natural gradients used in practice are close enough to the exact Riemannian gradient for the convergence theorem to apply (Eqs. 5.2, 5.7, 5.8).
    This is the weakest link: the practical algorithms use approximations of the Fisher information and a natural-gradient-like update for nu, but the theorem is proved only for exact Riemannian gradients; no error analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Variational Bayes on Manifolds." pith.science (2026). https://pith.science/paper/WOLHUZGI

@misc{pith2026190803097,
  author       = {Pith},
  title        = {Pith review of: Variational Bayes on Manifolds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WOLHUZGI}},
  note         = {Machine review of arXiv:1908.03097}
}
abstract

Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold of VB approximating probability distributions. Our algorithm is provably convergent and achieves a convergence rate of order $\mathcal O(1/\sqrt{T})$ and $\mathcal O(1/T^{2-2\epsilon})$ for a non-convex evidence lower bound function and a strongly retraction-convex evidence lower bound function, respectively. We develop in particular two manifold VB algorithms, Manifold Gaussian VB and Manifold Neural Net VB, and demonstrate through numerical experiments that the proposed algorithms are stable, less sensitive to initialization and compares favourably to existing VB methods.

Figures

Figures reproduced from arXiv: 1908.03097 by the authors.

Figure 1
Figure 1. Left: Tangent space at λ and the retraction map. Right: Vector transport 1 In general, an exponential map at λ is defined locally near λ. In order to define exponential map on the entire tangent space, the manifold needs to be complete, see, e.g., Tu (2011). 10 [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Plots of lower bound over iteration for the Manifold GVB algorithms with (cross [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Lower bound plots for the Manifold GVB algorithm (cross red) and the NAGVAC [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The posterior estimates for the GARCH model parameters by SMC and MGVB. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Left: lower bound. Right: true posterior mean of [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 29 canonical work pages

  1. [1]

    Absil, P.-A., Mahony, R., and Sepulchre, R. (2009). Optimization algorithms on matrix manifolds . Princeton University Press

  2. [2]

    Amari, S. (1998). Natural gradient works efficiently in learning. Neural computation , 10(2):251--276

  3. [3]

    and Ganea, O.-E

    B \'e cigneul, G. and Ganea, O.-E. (2018). Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00760

  4. [4]

    Berg, R. v. d., Hasenclever, L., Tomczak, J. M., and Welling, M. (2018). Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05649

  5. [5]

    Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics , 31(3):307 -- 327

  6. [6]

    Bonnabel, S. (2013). Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control , 58(9):2217--2229

  7. [7]

    Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R. (2014). M anopt, a M atlab toolbox for optimization on manifolds. Journal of Machine Learning Research , 15:1455--1459

  8. [8]

    Edelman, A., Arias, T., and Smith, A. (1998). The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. , 20(2):303–353

Show all 34 references
  1. [9]

    D., Blei, D

    Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J. (2013). Stochastic variational inference. Journal of Machine Learning Research , 14:1303--1347

  2. [10]

    Huang, W., Absil, P.-A., and Gallivan, K. A. (2015a). A Riemannian symmetric rank-one trust-region method. Mathematical Programming , 150(2):179--216

  3. [11]

    A., and Absil, P.-A

    Huang, W., Gallivan, K. A., and Absil, P.-A. (2015b). A Broyden class of quasi-Newton methods for Riemannian optimization. SIAM Journal on Optimization , 25(3):1660--1685

  4. [12]

    I., Ghahramani, Z., Jaakkola, T

    Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine learning , 37(2):183--233

  5. [13]

    Kasai, H., Jawanpuria, P., and Mishra, B. (2019). Adaptive stochastic gradient algorithms on riemannian manifolds. arXiv preprint arXiv:1902.01144

  6. [14]

    Khan, M. E. and Lin, W. (2017). Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)

  7. [15]

    Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  8. [16]

    and Nomizu, K

    Kobayashi, S. and Nomizu, K. (1969). Foundations of Differential Geometry, Vols. 1 and 2 . Wiley, New York

  9. [17]

    Longford, N. T. (1987). A fast scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects. Biometrika , 74(4):817--827

  10. [18]

    Manton , J. H. (2002). Optimization algorithms exploiting unitary constraints. IEEE Transactions on Signal Processing , 50(3):635--650

  11. [19]

    Mardia, K. V. and Marshall, R. J. (1984). Maximum likelihood estimation of models for residual covariance in spatial regression. Biometrika , 71(1):135--146

  12. [20]

    Martens, J. (2020). New insights and perspectives on the natural gradient method. Journal of Machine Learning Research , 21(146):1--76

  13. [21]

    M., Nott, D

    Ong, V. M., Nott, D. J., Tran, M.-N., Sisson, S. A., and Drovandi, C. C. (2018a). Variational Bayes with synthetic likelihood. Statistics and Computing , 28(4):971--988

  14. [22]

    M.-H., Nott, D

    Ong, V. M.-H., Nott, D. J., and Smith, M. S. (2018b). Gaussian variational approximation with factor covariance structure . Journal of Computational and Graphical Statistics , 27(3):465--478

  15. [23]

    Rao, C. R. (1945). Information and accuracy attainable in the estimation of statistical parameters. Bull. Calcutta. Math. Soc , 37:81--91

  16. [24]

    Roy, S. K. and Harandi, M. (2017). Constrained stochastic gradient descent: The good practice. In In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA)

  17. [25]

    and Aihara, K

    Sato, H. and Aihara, K. (2019). Cholesky QR-based retraction on the generalized Stiefel manifold . Computational Optimization and Applications , 72(2):293--308

  18. [26]

    Sato , M. (2001). Online model selection based on the variational bayes. Neural Computation , 13(7):1649--1681

  19. [27]

    Tagare, H. D. (2011). Notes on optimization on Stiefel manifolds. http://noodle.med.yale.edu/ hdtag/notes/steifel\_notes.pdf

  20. [28]

    Tran, M.-N., Nguyen, N., Nott, D., and Kohn, R. (2019). Bayesian deep net GLM and GLMM . Journal of Computational and Graphical Statistics

  21. [29]

    Tran, M.-N., Nguyen, T.-N., and Dao, V.-H. (2021). A practical tutorial on Variational Bayes . arXiv:2103.01327

  22. [30]

    J., and Kohn, R

    Tran, M.-N., Nott, D. J., and Kohn, R. (2017). Variational Bayes with intractable likelihood. Journal of Computational and Graphical Statistics , 26(4):873--882

  23. [31]

    K., and Kohn, R

    Tran, M.-N., Pitt, M. K., and Kohn, R. (2014). Annealed important sampling for models with latent variables. http://arxiv.org/abs/1402.6035

  24. [32]

    Tu, L. W. (2011). An introduction to manifolds . Springer New York

  25. [33]

    Waterhouse, S., MacKay, D., and Robinson, T. (1996). Bayesian methods for mixtures of experts. In Touretzky, M. C. M. D. S. and Hasselmo, M. E., editors, Advances in Neural Information Processing Systems , pages 351--357. MIT Press

  26. [34]

    Zhou, B., Gao, J., Tran, M.-N., and Gerlach, R. (2020). Manifold optimisation assisted Gaussian variational approximation. Journal of Computational and Graphical Statistics (to appear)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.