Pith. sign in

REVIEW 3 major objections 4 minor 78 references

Optimization Guarantees for Square-Root Natural-Gradient Variational Inference

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Square-root covariances unlock natural-gradient VI convergence proofs

desk verdict The paper has the right setup and a promising continuous-time result, but the proof of the main discrete-time theorem has a sign error in the C-update bound, and the unproven iterate bounds make the global claim conditional. read the letter →

arxiv 2507.07853 v1 pith:TKN2ZHXV submitted 2025-07-10 cs.LG cs.AIstat.ML

Navish Kumar , Thomas Möllenhoff , Mohammad Emtiyaz Khan , Aurelien Lucchi This is my paper
classification cs.LGcs.AIstat.ML MSC 62F1590C25
keywords natural-gradientvariationalinferencesquare-rootparameterizationCholeskyfactorGaussianapproximationPolyak-Łojasiewiczinequalityexponentialconvergencenaturalgradientflowinverse-freeupdates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to close the gap between the empirical speed of natural-gradient variational inference and the near absence of theory. It argues that the obstacle is the parameterization: the variational objective is not convex in the covariance or precision matrix, but it is convex in the Cholesky square root of the covariance. Working in that square-root parameterization, the paper proves exponential convergence for both the continuous-time natural-gradient flow and a discrete update it calls SR-VN, under strong convexity and smoothness of the log-likelihood and a bounded-iterate condition. If the guarantees hold, natural-gradient VI can be viewed as a provably contracting algorithm rather than a heuristic.

What carries the argument

The load-bearing object is the square-root parameterization of the Gaussian covariance: $V = CC^\top$ with $C$ lower-triangular and positive diagonal. Alongside it, the $\mathrm{tril}$ operator isolates the lower-triangular part and halves the diagonal; it appears in the SR-VN updates $C_{t+1} = C_t - \rho C_t \,\mathrm{tril}[C_t^\top H_t C_t - \gamma I]$ and $m_{t+1} = m_t - \rho C_t C_t^\top g_t$. The proof runs through a Riemannian Polyak-Łojasiewicz inequality: under strong convexity, the natural-gradient norm dominates the suboptimality gap of the ELBO, and that inequality turns into exponential decay for both the continuous flow and the discrete descent method.

What would settle it

Run SR-VN on a $\delta$-strongly convex, $M$-smooth logistic-regression problem from an Assumption 1 initialization, with a step size allowed by Equation (52), and inspect the iterates: if some $C_t$ has $\|C_t\|_F > \xi_u$ or some $V_t$ has smallest eigenvalue below the assumed $\lambda_{\min}$, then Assumption 4 is violated and the theorem's hypothesis does not describe the actual trajectory.

Watch

Extended reading notes

Core claim

The central claim is that replacing the covariance $V$ with its Cholesky factor $C$, so that $V = CC^\top$, preserves the strong convexity of the underlying loss in the variational objective and makes the KL functional satisfy a local Riemannian Polyak-Łojasiewicz inequality. From that inequality the paper derives an exponential contraction rate for the natural-gradient flow, and for the discretized square-root variational Newton update it proves $L(m_{t+1}, C_{t+1}) - L(m_*, C_*) \leq (1 - 2\eta\delta)^{t+1}\,(L(m_0, C_0) - L(m_*, C_*))$, with $\eta$ defined explicitly from the step size, smoothness, and eigenvalue bounds. The argument handles the non-smooth entropy term without proximal or projection operators, and it extends to any square-root parameterization of the Gaussian covariance.

Load-bearing premise

The proof assumes that every iterate stays inside a fixed region: the Frobenius norm of the Cholesky factor lies between $\xi_l$ and $\xi_u$, and the covariance matrix remains uniformly positive definite at every step; for the logistic-regression case the paper verifies this empirically rather than proving it. The whole contraction argument collapses if an iterate leaves that region.

Editorial extensions

If this is right

  • Under Assumptions 1–4, the KL gap along the square-root natural-gradient flow contracts at an exponential rate $e^{-2\mu t}$.
  • Algorithm 1 attains a global exponential rate $(1-2\eta\delta)^{t+1}$ on strongly convex smooth losses, with $\eta$ given in closed form from the step size, smoothness constant, and covariance eigenvalue bounds.
  • The Cholesky updates need no matrix inversion and no projection, so the proven guarantees cover a practical inverse-free implementation of natural-gradient variational inference.
  • The convergence analysis extends to any square-root parameterization of the Gaussian covariance, not only the Cholesky factor.
  • On Bayesian logistic-regression benchmarks, SR-VN tracks variational Newton and is consistently faster than Euclidean and Wasserstein-gradient baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Assumption 4 is granted, the same square-root machinery should transfer to other inverse-free natural-gradient algorithms, as long as their iterates stay bounded; the paper's proof does not itself require the specific Cholesky structure beyond the bound.
  • The contraction factor depends on the ratio $\xi_l^2/\xi_u^2$ and on $\lambda_{\min}/\xi_u^2$, so a practical but unstated consequence is that keeping the variational covariance well-conditioned, via warm starts or preconditioning, should directly improve the rate.
  • A testable resolution to the unproven bounded-iterate assumption would be a step-size rule that guarantees $V_t \succcurlyeq \lambda_{\min} I$ from the update itself; proving such a rule would replace empirical verification with a theorem.
  • The biased-oracle analysis suggests that increasing the number of pieces in the piecewise expectation bounds should shrink the limiting suboptimality gap arbitrarily, making the practical deterministic implementation match the clean theory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies natural-gradient variational inference with a Gaussian family, using a square-root (Cholesky-type) parameterization of the covariance. Its main theoretical contributions are Theorem 1, an exponential convergence rate for the continuous-time natural-gradient flow, and Theorem 2, an exponential convergence rate for the discrete-time square-root variational Newton (SR-VN) algorithm. Both results are stated under assumptions of strong convexity and smoothness of the log-likelihood, plus an additional bounded-iterates assumption. The paper also presents experiments comparing SR-VN with variational Newton and Wasserstein-gradient baselines on Bayesian logistic regression.

Significance. If the main theorems were correct, the paper would provide rare global convergence guarantees for discrete-time natural-gradient variational inference under concave log-likelihoods, and the square-root parameterization is a natural and potentially useful idea. The flow analysis via a Riemannian Polyak-Łojasiewicz inequality is a reasonable structural contribution, and the paper does not fit constants to data: the claimed rates follow from stated assumptions. However, the key discrete-time result is not established by the submitted proof, because of a sign error in the central descent estimate, and the bounded-iterates assumption is not proved. These issues affect the central claims, so the significance of the paper as it stands is substantially reduced.

major comments (3)
  1. [Appendix C.4, Eq. (45)-(47)] The proof of Theorem 2 contains a sign error in the treatment of the C-update term. With A = ∇_C L(m_t,C_t) and B = C_t^⊤ A, the first-order contribution to the descent is -ρ tr(A^⊤ C_t tril(B)) = -ρ tr(B^⊤ tril(B)), which is non-positive because tr(B^⊤ tril(B)) = ∑_{i>j} B_ij^2 + (1/2)∑_i B_ii^2 ≥ 0. The proof instead derives an upper bound tr(A^⊤ C_t \bar{\bar H}_t) ≤ √(5/2) ∥C_t∥_F^2 ∥∇_C L∥_F^2 and then treats -ρ tr(...) as if it were bounded above by -ρ times that upper bound. Since an upper bound on a non-negative quantity yields a lower bound on its negative, the claimed step only gives B ≥ -√(5/2)ρ ξ_u^2 ∥∇_C L∥_F^2 + (5Mρ^2 ξ_u^4/4)∥∇_C L∥_F^2. No negative upper bound on B follows. Consequently ω_C in Eq. (47), the value of η in Eq. (22), and the contraction factor in Eq. (21) are unjustified. This is a load-bearing defect in the proof of the paper's main discrete-time convergence theorem.
  2. [Section 4, Assumption 4] Assumption 4 (bounded iterates and uniform positive definiteness of V_t) is never proved. The text states that the assumption holds for logistic regression because the Hessian is bounded, but a bounded Hessian does not imply that the iterates C_t generated by Algorithm 1 remain in a bounded region or that C_t C_t^⊤ stays uniformly positive definite. The sentence "As confirmed empirically, the same conclusion extends to the SR-VN's C update" is an empirical observation, not a proof. Since Lemma 1, Lemma 2, Theorem 1, and Theorem 2 all rely on Assumption 4, this is a central gap. The theorems should either include a proof that the assumption is satisfied under the stated conditions, or explicitly present the results as conditional on an unverified invariant-region assumption.
  3. [Appendix C.1, Lemma 1] The lower bound claimed for the inverse Fisher information in the C-block is not valid in general. The proof derives ∥F_C^{-1}∥_2 ≥ 1/(2∥C^{-1}∥_2^2) and then asserts this is ≥ λ_min^2/2 because ∥C^{-1}∥_2^2 ≥ ∥V^{-1}∥_2 = 1/λ_min. But ∥C^{-1}∥_2^2 = 1/λ_min, so the bound is actually λ_min/2, not λ_min^2/2. When λ_min > 1, λ_min^2/2 exceeds λ_min/2 and the stated inequality is false. Since λ_min^g feeds into the PL constant μ in Lemma 2 and hence into the rate in Theorem 1, the exact rate claimed in Theorem 1 is not supported as written, even though a weaker rate with μ = δ λ_min/2 may be available.
minor comments (4)
  1. [Theorem 2 statement] The label "Global Convergence" is misleading because Assumption 4 only allows iterates in a bounded, uniformly positive-definite region; the theorem is conditional on an invariant region rather than a genuinely global statement. I suggest renaming it "Convergence under Assumption 4" or proving the invariant region.
  2. [Section 4 and Appendix C.5] Equation (52) proposes a step-size rule that depends on ξ_l and ξ_u, but these constants are not known a priori in practice. The experimental section uses tuned step sizes (Table 2) without connecting them to Eq. (52), so it is unclear how the theoretical step-size condition would be implemented.
  3. [Appendix C.4] The notation \bar{\bar H}_t, \bar H, and diag(H) is introduced in the proof without definitions in the main text; a short explanation or a pointer to Table 1 would improve readability.
  4. [Section 2] There are several typographical and grammatical issues, e.g., "Given a Bayesian model withn likelihoods" and "withn" fragments; a careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the rates follow from explicit assumptions and external smoothness/convexity results; the Theorem 2 proof has a non-circular inequality-direction defect.

full rationale

The derivation chain is not circular. Theorem 1's rate follows from Assumptions 2 and 4 via Lemma 1 (bounded FIM, proved from Assumption 4 and Tan 2025's FIM formula) and Lemma 2, which rewrites Assumption 2's delta-strong convexity into a Riemannian PL inequality with constant mu = delta * lambda_g_min; this is a standard implication, not an identity with the conclusion. Theorem 2's rate is derived from Assumptions 1-4 using the Taylor expansion (Assumption 3), the descent bounds (46)-(49), and strong convexity; no constant is fitted to data, and no 'prediction' is defined in terms of a fitted quantity. The self-citations (Khan & Rue 2023, Khan et al. 2017/2018) supply the VN update and stationary-point characterization (19), which are context rather than load-bearing uniqueness claims. Two weaknesses should be recorded as non-circular: (i) Assumption 4 (bounded/positive-definite iterates) is asserted for logistic regression and only 'confirmed empirically' (Section 4), so the theorem is conditional; and (ii) in Appendix C.4 the bound on the C-update descent term has a direction error—the trace tr((nabla_C L)^T C_t \bar{\bar H}_t) is nonnegative, so the Cauchy-Schwarz upper bound cannot be substituted into -rho tr(...) to obtain a negative quadratic coefficient. This invalidates the stated omega_C and rate, but it is a proof defect, not a circular reduction of the conclusion to the inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central results rest on three problem-dependent assumptions (strong convexity, smoothness, bounded iterates) plus standard convex-analysis theorems. No free parameters are fitted to data to obtain the rates; step sizes are chosen to satisfy the stated inequalities.

assumptions (4)
  • domain assumption Assumption 2: The negative log-likelihood is δ-strongly convex in θ.
    Used to derive δ-strong convexity of L in (m,C) via Domke (2019, Thm 9), needed for the PL inequality and for the contraction factor in Theorem 2.
  • domain assumption Assumption 3: The negative log-likelihood is M-Lipschitz smooth in θ.
    Used to bound the Taylor expansion in Theorem 2 via Domke (2020, Thm 1).
  • domain assumption Assumption 4: Iterates stay bounded and covariance is uniformly positive definite.
    Needed for Lemma 1 and Lemma 2; not proven for the algorithm, only empirically asserted for logistic regression (Section 4).
  • standard math Domke's convexity/smoothness transfer theorems.
    The paper relies on Domke (2019, Thm 9) and Domke (2020, Thm 1) to lift θ-convexity/smoothness to (m,C)-parameterization; accepted background results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimization Guarantees for Square-Root Natural-Gradient Variational Inference." pith.science (2026). https://pith.science/paper/TKN2ZHXV

@misc{pith2026250707853,
  author       = {Pith},
  title        = {Pith review of: Optimization Guarantees for Square-Root Natural-Gradient Variational Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKN2ZHXV}},
  note         = {Machine review of arXiv:2507.07853}
}
read the original abstract

Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.

Figures

Figures reproduced from arXiv: 2507.07853 by the authors.

Figure 1
Figure 1. (a) Natural-gradient descent (circle) converges much faster (in just 1 step) than algorithms that use Euclidean (cross) or Wasserstein (star) geometries which take 10 and 470 iterations respectively. The illustration is on a 2-D Bayesian linear regression (quadratic loss in the background for all three figures). The 3 methods are taken from Khan & Rue (2023); Ranganath et al. (2014); Lambert et al. (2022b), (b) For … view at source ↗
Figure 2
Figure 2. For small-scale LIBSVM datasets, we show the training ELBO and test NLL w.r.t number of iterations (left and middle panel) and time (right panel). The min and max of the plotted values are displayed around their averages (taken over five random initializations). We also subtracted the min values achieved (for both training and testing) over all iterations from all three methods and only plotted the resulting values.… view at source ↗
Figure 3
Figure 3. Comparison on Large-Scale Datasets. The same trends hold at scale: both SR-VN and VN exhibit comparable performance, whereas BW-GD tends to be slower. dataset and the NLL on the test dataset. We report the average of the NLL over all test points, i.e. we compare the NLL on the test set computed as follows: − Pntest i=1 log(1 + exp{−yi(θˆ⊤xi)}/ntest where θˆ is parameter estimate and ntest is the number of examples i… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The provided comparison highlights the differences between the piecewise implementation and the (minibatch + MC sampling) stochastic implementation for SR-VN. The results clearly demonstrate that the stochastic approach is much slower compared to the piecewise method. …
Figure 5
Figure 5. Figure 5: We compared the run times of all three algorithms across increasing dimensions while keeping the number of data points (n) fixed. The run times were averaged over 1000 iterations per algorithm. The theoretical run times, derived in our rebuttal answer, are functions of…
Figure 6
Figure 6. Figure 6: Plots with respect to the number of iterations for small-scale datasets. [PITH_FULL_IMAGE:figures/full_fig_p031_6.png]
Figure 7
Figure 7. Figure 7: Plots with respect to the number of iterations for large-scale datasets. [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 8
Figure 8. Figure 8: Plots with respect to time for small-scale datasets. [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]
Figure 9
Figure 9. Figure 9: Plots with respect to time for large-scale datasets. [PITH_FULL_IMAGE:figures/full_fig_p034_9.png]
Figure 10
Figure 10. Figure 10: Comparison of SVGD-density: Bayesian linear regression (top) and Bayesian logistic regression (bottom). 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png]
Figure 11
Figure 11. Figure 11: To evaluate performance under non-convex conditions, we augment the Bayesian logistic regression problem in Equation (15) with a non-convex regularization term: Pd i=1 m2 i /(1 + m2 i ). 36 [PITH_FULL_IMAGE:figures/full_fig_p036_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 62 canonical work pages

  1. [1]

    Dual parameterization of sparse variational gaussian processes

    Vincent Adam, Paul Chang, Mohammad Emtiyaz E Khan, and Arno Solin. Dual parameterization of sparse variational gaussian processes. Advances in Neural Information Processing Systems, 34: 0 11474--11486, 2021

    work page 2021

  2. [2]

    On the properties of variational approximations of G ibbs posteriors

    Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of G ibbs posteriors. J. Mach. Learn. Res. (JMLR), 17: 0 239:1--239:41, 2016

    work page 2016

  3. [3]

    Natural gradient works efficiently in learning

    Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    1998

  4. [4]

    Information geometry and its applications: Convex function and dually flat manifold

    Shun-Ichi Amari. Information geometry and its applications: Convex function and dually flat manifold. In LIX Fall Colloquium on Emerging Trends in Visual Computing, pp.\ 75--102. Springer, 2008

    work page 2008

  5. [5]

    Information geometry and its applications, volume 194

    Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016

    work page 2016

  6. [6]

    Convergence of a random optimization method for constrained optimization problems

    N Baba. Convergence of a random optimization method for constrained optimization problems. Journal of Optimization Theory and Applications, 33: 0 451--461, 1981

    work page 1981

  7. [7]

    The theory of evolution strategies

    Hans-Georg Beyer. The theory of evolution strategies. Springer Science & Business Media, 2001

    work page 2001

  8. [8]

    Variational inference: A review for statisticians

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112 0 (518): 0 859--877, 2017

    2017

Show all 78 references
  1. [9]

    Weight uncertainty in neural networks

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning (ICML), 2015

    2015

  2. [10]

    Batch and match: black-box variational inference with a score-based divergence

    Diana Cai, Chirag Modi, Loucas Pillaud-Vivien, Charles C Margossian, Robert M Gower, David M Blei, and Lawrence K Saul. Batch and match: black-box variational inference with a score-based divergence. arXiv preprint arXiv:2402.14758, 2024

    2024 arXiv

  3. [11]

    Pac-bayesian supervised classification: The thermodynamics of statistical learning

    O Catoni. Pac-bayesian supervised classification: The thermodynamics of statistical learning. institute of mathematical statistics lecture notes—monograph series 56. IMS, Beachwood, OH. MR2483528, 5544465, 2007

    2007

  4. [12]

    Gaussian kullback-leibler approximate inference

    Edward Challis and David Barber. Gaussian kullback-leibler approximate inference. Journal of Machine Learning Research, 14 0 (8), 2013

    2013

  5. [13]

    LIBSVM : a library for support vector machines

    Chih-Chung Chang and Chih-Jen Lin. LIBSVM : a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2 0 (3): 0 1--27, 2011

    2011

  6. [14]

    Gradient flows for sampling: Mean-field models, gaussian approximations and affine invariance

    Yifan Chen, Daniel Zhengyu Huang, Jiaoyang Huang, Sebastian Reich, and Andrew M Stuart. Gradient flows for sampling: Mean-field models, gaussian approximations and affine invariance. arXiv preprint arXiv:2302.11024, 2023

    2023 arXiv

  7. [15]

    Convergence rates of variational inference in sparse deep learning

    Badr-Eddine Ch \'e rief-Abdellatif. Convergence rates of variational inference in sparse deep learning. In International Conference on Machine Learning, pp.\ 1831--1842. PMLR, 2020

    2020

  8. [16]

    A generalization bound for online variational inference

    Badr-Eddine Ch \'e rief-Abdellatif, Pierre Alquier, and Mohammad Emtiyaz Khan. A generalization bound for online variational inference. In Asian Conference on Machine Learning (ACML), 2019

    2019

  9. [17]

    Convex optimization and euclidean distance geometry

    Jon Dattorro. Convex optimization and euclidean distance geometry. Book, 2015. https://ccrma.stanford.edu/ dattorro/0976401304_v2015.04.11.pdf

    2015

  10. [18]

    Forward-backward gaussian variational inference via jko in the bures-wasserstein space

    Michael Ziyang Diao, Krishna Balasubramanian, Sinho Chewi, and Adil Salim. Forward-backward gaussian variational inference via jko in the bures-wasserstein space. In International Conference on Machine Learning, pp.\ 7960--7991. PMLR, 2023

    2023

  11. [19]

    Provable gradient variance guarantees for black-box variational inference

    Justin Domke. Provable gradient variance guarantees for black-box variational inference. Advances in Neural Information Processing Systems, 32, 2019

    2019

  12. [20]

    Provable smoothness guarantees for black-box variational inference

    Justin Domke. Provable smoothness guarantees for black-box variational inference. In International Conference on Machine Learning (ICML), 2020

    2020

  13. [21]

    Provable convergence guarantees for black-box variational inference

    Justin Domke, Robert Gower, and Guillaume Garrigos. Provable convergence guarantees for black-box variational inference. Advances in neural information processing systems, 36, 2024

    2024

  14. [22]

    Exponential natural evolution strategies

    Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and J \"u rgen Schmidhuber. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp.\ 393--400, 2010

    2010

  15. [23]

    Natural gradient variational bayes without fisher matrix analytic calculation and its inversion

    A Godichon-Baggioni, D Nguyen, and M-N Tran. Natural gradient variational bayes without fisher matrix analytic calculation and its inversion. Journal of the American Statistical Association, pp.\ 1--12, 2024

    2024

  16. [24]

    Practical variational inference for neural networks

    Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2011

    2011

  17. [25]

    On graduated optimization for stochastic non-convex problems

    Elad Hazan, Kfir Yehuda Levy, and Shai Shalev-Shwartz. On graduated optimization for stochastic non-convex problems. In International conference on machine learning, pp.\ 1833--1841. PMLR, 2016

    2016

  18. [26]

    Fast variational inference in the conjugate exponential family

    James Hensman, Magnus Rattray, and Neil Lawrence. Fast variational inference in the conjugate exponential family. Advances in neural information processing systems, 25, 2012

    2012

  19. [27]

    Gaussian processes for big data

    James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6835, 2013

    2013 arXiv

  20. [28]

    Scalable variational gaussian process classification

    James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian process classification. In Artificial Intelligence and Statistics, pp.\ 351--360. PMLR, 2015

    2015

  21. [29]

    Stochastic variational inference

    Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013

    2013

  22. [30]

    Natural conjugate gradient in variational inference

    Antti Honkela, Matti Tornio, Tapani Raiko, and Juha Karhunen. Natural conjugate gradient in variational inference. In Neural Information Processing: 14th International Conference, ICONIP 2007, Kitakyushu, Japan, November 13-16, 2007, Revised Selected Papers, Part II 14, pp.\ 3...

    2007

  23. [31]

    Approximate riemannian conjugate gradient learning for fixed-form variational bayes

    Antti Honkela, Tapani Raiko, Mikael Kuusela, Matti Tornio, and Juha Karhunen. Approximate riemannian conjugate gradient learning for fixed-form variational bayes. The Journal of Machine Learning Research, 11: 0 3235--3268, 2010

    2010

  24. [32]

    A variational approach to bayesian logistic regression models and their extensions

    Tommi S Jaakkola and Michael I Jordan. A variational approach to bayesian logistic regression models and their extensions. In International Conference on Artificial Intelligence and Statistics (AISTATS), 1997

    1997

  25. [33]

    Composing graphical models with neural networks for structured representations and fast inference

    Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29, 2016

    2016

  26. [34]

    Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models

    Mohammad Khan and Wu Lin. Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models. In Artificial Intelligence and Statistics, pp.\ 878--887. PMLR, 2017

    2017

  27. [35]

    Fast and scalable bayesian deep learning by weight-perturbation in adam

    Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International conference on machine learning, pp.\ 2611--2620. PMLR, 2018

    2018

  28. [36]

    Fast yet simple natural-gradient descent for variational inference in complex models

    Mohammad Emtiyaz Khan and Didrik Nielsen. Fast yet simple natural-gradient descent for variational inference in complex models. In 2018 International Symposium on Information Theory and Its Applications (ISITA), pp.\ 31--35. IEEE, 2018

    2018

  29. [37]

    The bayesian learning rule

    Mohammad Emtiyaz Khan and H vard Rue. The bayesian learning rule. Journal of Machine Learning Research, 24 0 (281): 0 1--46, 2023

    2023

  30. [38]

    Faster stochastic variational inference using proximal-gradient methods with general divergence functions

    Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin, Mark Schmidt, and Masashi Sugiyama. Faster stochastic variational inference using proximal-gradient methods with general divergence functions. In UAI'16, UAI'16, pp.\ 319–328, Arlington, Virginia, USA, 2016. AUAI Press

    2016

  31. [39]

    Variational adaptive-newton method for explorative learning

    Mohammad Emtiyaz Khan, Wu Lin, Voot Tangkaratt, Zuozhu Liu, and Didrik Nielsen. Variational adaptive-newton method for explorative learning. arXiv preprint arXiv:1711.05560, 2017

    2017 arXiv

  32. [40]

    On the convergence of black-box variational inference

    Kyurae Kim, Jisu Oh, Kaiwen Wu, Yian Ma, and Jacob Gardner. On the convergence of black-box variational inference. Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [41]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  34. [42]

    Non-conjugate variational message passing for multinomial and binary regression

    David Knowles and Tom Minka. Non-conjugate variational message passing for multinomial and binary regression. Advances in Neural Information Processing Systems, 24, 2011

    2011

  35. [43]

    The recursive variational gaussian approximation (r-vga)

    Marc Lambert, Silvere Bonnabel, and Francis Bach. The recursive variational gaussian approximation (r-vga). Statistics and Computing, 32 0 (1): 0 10, 2022 a

    2022

  36. [44]

    Variational inference via wasserstein gradient flows

    Marc Lambert, Sinho Chewi, Francis Bach, Silv \`e re Bonnabel, and Philippe Rigollet. Variational inference via wasserstein gradient flows. Advances in Neural Information Processing Systems, 35: 0 14434--14447, 2022 b

    2022

  37. [45]

    Smoothing-based optimization

    Marius Leordeanu and Martial Hebert. Smoothing-based optimization. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 1--8. IEEE, 2008

    2008

  38. [46]

    Fast and simple natural-gradient variational inference with mixture of exponential-family approximations

    Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Fast and simple natural-gradient variational inference with mixture of exponential-family approximations. In International Conference on Machine Learning, pp.\ 3992--4002. PMLR, 2019

    2019

  39. [47]

    Structured second-order methods via natural gradient descent

    Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Structured second-order methods via natural gradient descent. arXiv preprint arXiv:2107.10884, 2021 a

    2021 arXiv

  40. [48]

    Tractable structured natural-gradient descent using local parameterizations

    Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Tractable structured natural-gradient descent using local parameterizations. In International Conference on Machine Learning, pp.\ 6680--6691. PMLR, 2021 b

    2021

  41. [49]

    Simplifying momentum-based positive-definite submanifold optimization with applications to deep learning

    Wu Lin, Valentin Duruisseaux, Melvin Leok, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Simplifying momentum-based positive-definite submanifold optimization with applications to deep learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Si...

    2023

  42. [50]

    Turner, and Alireza Makhzani

    Wu Lin, Felix Dangel, Runa Eschenhagen, Kirill Neklyudov, Agustinus Kristiadi, Richard E. Turner, and Alireza Makhzani. Structured inverse-free natural gradient descent: Memory-efficient and numerically-stable KFAC . In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adri...

    2024

  43. [51]

    Towards understanding the dynamics of gaussian-stein variational gradient descent

    Tianle Liu, Promit Ghosal, Krishnakumar Balasubramanian, and Natesh Pillai. Towards understanding the dynamics of gaussian-stein variational gradient descent. Advances in Neural Information Processing Systems, 36, 2024

    2024

  44. [52]

    Information geometry of the G aussian distribution in view of stochastic optimization

    Luigi Malagò and Giovanni Pistone. Information geometry of the G aussian distribution in view of stochastic optimization. In Proceedings of the 2015 ACM Conference on Foundations of Genetic Algorithms , pp.\ 150--162, 2015

    2015

  45. [53]

    Piecewise bounds for estimating bernoulli-logistic latent gaussian models

    Benjamin M Marlin, Mohammad Emtiyaz Khan, and Kevin P Murphy. Piecewise bounds for estimating bernoulli-logistic latent gaussian models. In ICML, pp.\ 633--640, 2011

    2011

  46. [54]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PMLR, 2016

    1928

  47. [55]

    A theoretical analysis of optimization by gaussian continuation

    Hossein Mobahi and John Fisher III. A theoretical analysis of optimization by gaussian continuation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  48. [56]

    Differentiation of the cholesky decomposition

    Iain Murray. Differentiation of the cholesky decomposition. arXiv preprint arXiv:1602.07527, 2016

    2016 arXiv

  49. [57]

    Practical deep learning with bayesian principles

    Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. Advances in neural information processing systems, 32, 2019

    2019

  50. [58]

    K. B. Petersen and M. S. Pedersen. The matrix cookbook, nov 2012. URL http://www2.compute.dtu.dk/pubdb/pubs/3274-full.html. Version 20121115

    2012

  51. [59]

    Black box variational inference

    Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial intelligence and statistics, pp.\ 814--822. PMLR, 2014

    2014

  52. [60]

    Fast black-box variational inference through stochastic trust-region optimization

    Jeffrey Regier, Michael I Jordan, and Jon McAuliffe. Fast black-box variational inference through stochastic trust-region optimization. Advances in Neural Information Processing Systems, 30, 2017

    2017

  53. [61]

    Tim Salimans and David A. Knowles. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8 0 (4), December 2013. ISSN 1936-0975. doi:10.1214/13-ba858. URL http://dx.doi.org/10.1214/13-BA858

    2013 doi

  54. [62]

    Natural gradients in practice: Non-conjugate variational inference in gaussian process models

    Hugh Salimbeni, Stefanos Eleftheriadis, and James Hensman. Natural gradients in practice: Non-conjugate variational inference in gaussian process models. In International Conference on Artificial Intelligence and Statistics, pp.\ 689--697. PMLR, 2018

    2018

  55. [63]

    Variational learning is effective for large deep networks

    Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas M\" o llenhoff. Variational learning is effective for large deep networks. In Proceedings of the 41st Internationa...

    2024

  56. [64]

    Variational learning is effective for large deep networks

    Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, et al. Variational learning is effective for large deep networks. arXiv preprint arXiv:2402.17641, 2024 b

    2024 arXiv

  57. [65]

    Introduction to stochastic search and optimization: estimation, simulation, and control

    James C Spall. Introduction to stochastic search and optimization: estimation, simulation, and control. John Wiley & Sons, 2005

    2005

  58. [66]

    Variational optimization

    Joe Staines and David Barber. Variational optimization. arXiv preprint arXiv:1212.4507, 2012

    2012 arXiv

  59. [67]

    Efficient natural evolution strategies

    Yi Sun, Daan Wierstra, Tom Schaul, and J \"u rgen Schmidhuber. Efficient natural evolution strategies. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pp.\ 539--546, 2009

    2009

  60. [68]

    Introduction to reinforcement learning

    Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. vol. 135, 1998

    1998

  61. [69]

    Analytic natural gradient updates for cholesky factor in gaussian variational approximation

    Linda SL Tan. Analytic natural gradient updates for cholesky factor in gaussian variational approximation. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp.\ qkaf001, 2025

    2025

  62. [70]

    The variational predictive natural gradient

    Da Tang and Rajesh Ranganath. The variational predictive natural gradient. In International Conference on Machine Learning, pp.\ 6145--6154. PMLR, 2019

    2019

  63. [71]

    Entropic proximal mappings with applications to nonlinear programming

    Marc Teboulle. Entropic proximal mappings with applications to nonlinear programming. Mathematics of Operations Research, 17 0 (3): 0 670--690, 1992

    1992

  64. [72]

    A trust-region method for stochastic variational inference with applications to streaming data

    Lucas Theis and Matt Hoffman. A trust-region method for stochastic variational inference with applications to streaming data. In International conference on machine learning, pp.\ 2503--2511. PMLR, 2015

    2015

  65. [73]

    Doubly stochastic variational bayes for non-conjugate inference

    Michalis Titsias and Miguel L \'a zaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. In International conference on machine learning, pp.\ 1971--1979. PMLR, 2014

    1971

  66. [74]

    Graphical models, exponential families, and variational inference

    Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning , 1 0 (1--2): 0 1--305, 2008

    2008

  67. [75]

    Function optimization using connectionist reinforcement learning algorithms

    Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3 0 (3): 0 241--268, 1991

    1991

  68. [76]

    Understanding stochastic natural gradient variational inference

    Kaiwen Wu and Jacob R Gardner. Understanding stochastic natural gradient variational inference. arXiv preprint arXiv:2406.01870, 2024

    2024 arXiv

  69. [77]

    Theoretical analysis of a class of randomized regularization methods

    Tong Zhang. Theoretical analysis of a class of randomized regularization methods. In Proceedings of the twelfth annual conference on Computational learning theory, pp.\ 156--163, 1999

    1999

  70. [78]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.