{"id":"49b78fb7-9058-49de-bd21-7d44d39b73e6","arxiv_id":"1908.03097","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A manifold-based variational Bayes algorithm with natural-gradient updates is shown to converge at O(1/sqrt(T)) for non-convex ELBOs and faster under retraction-convexity, with Gaussian and Wishart implementations.","lead":"This paper develops a variational Bayes algorithm that runs directly on curved parameter spaces, such as covariance matrices, and combines the geometry of the space with the natural gradient for faster, more stable inference. It proves convergence rates and tests the method on logistic regression, a GARCH financial model, and a synthetic covariance estimation task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's strongly retraction-convex rate proof mishandles 1/(1−ρ): with ρ = 1−O(γ), the unrolled noise term is O(γ), not O(γ^2), so the advertised O(1/T^{2−2ε}) does not follow.","rationale":"The reader's weakest_assumption is that the practical algorithms are not covered by Theorem 1 because the natural-gradient updates in Eqs. (5.2), (5.7), (5.8) are approximations. That is a genuine concern. However, I found a more basic problem: the proof of the strongly retraction-convex rate in Theorem 1 itself contains an algebraic error that invalidates the advertised O(1/T^{2−2ε}) guarantee even when the exact Riemannian gradient is used. The proof defines ρ = 1 − O(γ) and then incorrectly treats γ^2/(1−ρ) as O(γ^2) instead of O(γ), so the unrolled recursion yields only O(γ) = O(T^{−ε}) with the stated step-size choice, not O(T^{2ε−2}). There is also a direct mismatch between γ^2 and the written T^{2ε−2} unless ε = 1/2. Because the central claim depends on this rate, the paper should be revised to either correct the proof, state the weaker rate, or restrict the claim accordingly. I nevertheless keep the reader's CONDITIONAL verdict: the non-convex O(1/√T) bound and the algorithmic framework remain valuable, and the error is local and likely fixable, so the paper is not beyond repair.","tokens_in":22242,"tokens_out":18604,"duration_ms":191199,"concrete_test":"Re-derive the unrolled bound from Eq. (6.27) exactly: substitute ρ = 1 − γ/(4μ~(1−ζ)) into EV_{t+1} ≤ ρ EV_t + C γ^2 and sum to get EV_T ≤ ρ^T EV_0 + C γ^2 (1−ρ^T)/(1−ρ). Compute the second term as 4μ~(1−ζ) C γ (1−ρ^T), which is O(γ). Then substitute γ = 1/T^ε: the noise term is O(T^{−ε}), and compare with O(T^{2ε−2}); the ratio is T^{2−3ε}, so the claimed rate cannot be recovered by adjusting constants. This one re-derivation settles whether the second rate in Theorem 1 is valid as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4, Theorem 1, second claim: the proof sets ρ := 1 − γ/(4μ~(1−ζ)) (just before Eq. 6.22), so 1−ρ = O(γ). The unrolled recursion after Eq. (6.27) gives EV_T ≤ ρ^T C0 + C γ^2/(1−ρ). The proof then replaces this by ρ^T C0 + C γ^2, which is algebraically false: γ^2/(1−ρ) = 4μ~(1−ζ) γ, i.e. O(γ), not O(γ^2). With γ = T^{−ε}, the correct noise floor is O(T^{−ε}), not O(T^{2ε−2}). A second inconsistency appears in the same paragraph: if γ = 1/T^ε, then γ^2 = T^{−2ε}, which equals the claimed T^{2ε−2} only for ε = 1/2. Thus the announced rate O(1/T^{2−2ε}) in the abstract and Theorem 1 is unsupported by the proof as written. This is load-bearing because the paper's central claim of a fast strongly-convex rate fails even in the exact-gradient setting, independent of the practical approximations in Section 5.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a manifold-based variational Bayes framework. It treats the variational parameter space as a Riemannian manifold, equips it with the Fisher-Rao metric, identifies the natural gradient with the Riemannian gradient (Lemma 3.1), handles submanifold and quotient cases, and runs Riemannian SGD with momentum. It states convergence rates of order O(1/sqrt(T)) for non-convex ELBOs and O(1/T^(2-2ε)) for strongly retraction-convex ELBOs. It then instantiates Manifold Gaussian VB and Manifold Wishart VB, with numerical experiments on logistic regression, GARCH volatility, and covariance estimation, plus comparisons to NAGVAC and SMC.","tokens_in":1491,"tokens_out":1702,"duration_ms":95219,"significance":"If the convergence theorem were correct, this would be a valuable generalization of variational Bayes to constrained parameter spaces, combining information-geometric preconditioning with Riemannian optimization. The geometric derivations in Lemmas 3.1 and 3.2 are clean and self-contained, the algorithmic descriptions are concrete, and the authors provide code and an SMC comparison. However, the advertised fast strongly-convex rate is not supported by the proof as written, and the implemented algorithms rely on unquantified approximations that the theorem does not cover. These issues affect the paper's central guarantees, so the significance is currently conditional on a repair of the proof and a clarification of the scope of the theoretical claims.","major_comments":[{"comment":"The proof sets rho = 1 - gamma/(4*mu_tilde*(1-zeta)) immediately before Eq. (6.22), so 1-rho = O(gamma). The unrolled recursion then gives EV_T <= rho^T C0 + C*gamma^2/(1-rho), but the next line replaces C*gamma^2/(1-rho) by C*gamma^2. This is algebraically false: C*gamma^2/(1-rho) = 4*C*mu_tilde*(1-zeta)*gamma = O(gamma). With gamma = T^{-epsilon}, the correct noise floor is O(T^{-epsilon}), not O(T^(2*epsilon-2)). A second inconsistency appears in the same paragraph: gamma^2 = T^(-2*epsilon) equals T^(2*epsilon-2) only when epsilon = 1/2. Thus the advertised rate O(1/T^(2-2*epsilon)) in the abstract and Theorem 1 is unsupported by the proof as written.","section":"Section 4, proof of Theorem 1(ii), around Eq. (6.27)"},{"comment":"The convergence theorem is proved for exact Riemannian gradients with unbiased estimators. The implemented Manifold Gaussian VB uses the approximation IF(Sigma) approx Sigma^{-1} tensor Sigma^{-1}, and the Manifold Wishart VB uses the natural-gradient-like replacements in (5.7) and (5.8). No error bound or perturbation analysis relates these approximate updates to the exact Riemannian gradient required by Theorem 1. Consequently, the stated convergence guarantees do not apply to the algorithms whose numerical behavior is reported. The paper should either prove that these approximations satisfy the theorem's conditions up to controlled errors, or explicitly state that the numerical results are heuristic demonstrations.","section":"Section 5.1, Eqs. (5.2)-(5.4), and Section 5.2, Eqs. (5.7)-(5.8)"},{"comment":"The quotient-manifold construction defines the horizontal space as the orthogonal complement under the Fisher-Rao metric, but the metric is not invariant along equivalence classes, as Remark 3.1 concedes. The claim that equation (3.10) is independent of the representative lambda in [lambda] is asserted rather than proved. Without metric invariance, the horizontal lift and the induced inner product on the quotient manifold are generally representative-dependent, and the usual Riemannian-submersion condition is exactly what is needed to rule this out. This is a gap in the claimed general framework for quotient manifolds, although the paper's numerical experiments do not depend on this part.","section":"Section 3.2, Remark 3.1"}],"minor_comments":[{"comment":"The abstract names Manifold Neural Net VB as one of the two developed algorithms, but the paper actually develops Manifold Gaussian VB and Manifold Wishart VB; the abstract should be corrected to match the body.","section":"Abstract"},{"comment":"The legend in Figure 3 reads NAGVAG instead of NAGVAC.","section":"Figure 3"},{"comment":"The theorem statement bounds E(lambda_T - lambda_*)^2, but the proof bounds E(L(lambda_T) - L(lambda_*)) and E|grad L|^2; the strong-convexity inequality linking squared retraction distance to the function suboptimality is not stated, so the final line of the proof does not directly deliver the theorem's claim.","section":"Section 4, end of proof of Theorem 1(ii)"},{"comment":"The sentence 'One can derive that IF(Sigma) approx Sigma^{-1} tensor Sigma^{-1}' gives no derivation or citation for the approximation; since this approximation is used in the implemented algorithm, a reference or explicit formula would clarify its accuracy.","section":"Section 5.1, Eq. (5.2)"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the algebra error in the proof of Theorem 1(ii): as written, the announced O(1/T^(2-2*epsilon)) rate does not follow, and the correct floor from the proof is O(T^{-epsilon}). If the authors can repair this step or restate the theorem with a correct rate, the paper could become suitable. The approximation gap in Section 5 and the quotient-metric issue in Section 3.2 also need substantive responses, not just presentational fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper gives a clean general template for VB on Riemannian manifolds with natural-gradient preconditioning and momentum, and the non-convex O(1/√T) convergence claim looks correct. Second, the faster O(1/T^{2−2ε}) rate for strongly retraction-convex ELBOs is not supported by the proof as written. I checked the appendix against the stress-test note; the note is right.\n\nThe genuinely new piece is combining the natural gradient with Riemannian SGD for a general VB problem, covering submanifolds (SPD, Stiefel) and quotient manifolds. The two instantiations, Manifold Gaussian VB and Manifold Wishart VB, are sensible. The experiments on logistic regression and the GARCH/SMC comparison show real practical benefit over no-natural-gradient and over NAGVAC. Lemmas 3.1 and 3.2 are standard geometry but correctly worked out.\n\nNow the soft spots, in order of severity. The fast-rate proof has a load-bearing algebraic error. In the appendix, after Eq. (6.27), the proof sets ρ := 1 − γ/(4μ~(1−ζ)), so 1−ρ = O(γ). The unrolled inequality gives EV_T ≤ ρ^T C0 + C γ²/(1−ρ), which is O(γ), not O(γ²). The proof silently replaces it with Cγ². With γ = T^{−ε}, the actual noise floor is O(T^{−ε}), not O(T^{2ε−2}). So the abstract and Theorem 1 overclaim the rate. The claimed exponent also gets worse as ε increases, which is the opposite of what you'd expect.\n\nSecond, Theorem 1 is proved for exact natural gradients, but the algorithms in Section 5 use approximations (Eqs. 5.2, 5.7, 5.8) and an adaptive update for the Wishart ν. There is no error analysis connecting those approximations to the theorem, so 'provably convergent' is technically scoped to the exact setting. That's a theory-practice gap, not necessarily fatal, but it needs to be stated.\n\nThird, Remark 3.1 asserts that the quotient-manifold gradient is well-defined without the usual metric invariance, but the argument does not actually show the metric on the quotient is independent of the representative. Under-specified.\n\nMinor: the experiments have no error bars and the code link is a general VBayesLab account, not a specific commit or data file.\n\nWho is this for? Anyone working on VB for constrained parameter spaces. It deserves a serious referee: the framework is useful and the flaws are fixable. If I were the editor, I'd send it out, but I'd require a corrected rate claim and a discussion of the approximate gradients before accepting.","headline":"Solid framework for natural-gradient VB on manifolds, but the advertised O(1/T^{2−2ε}) rate for strongly retraction-convex ELBOs doesn't follow from the proof.","tokens_in":23062,"tokens_out":6379,"would_cite":true,"duration_ms":57990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Variational Bayes can be run directly on curved parameter spaces, with proven convergence to a local mode of the evidence lower bound.","keywords":["variational Bayes","natural gradient","Riemannian manifold","stochastic gradient descent","Fisher information","evidence lower bound","Gaussian variational approximation","Wishart variational approximation"],"falsifier":"Run the Manifold Gaussian VB algorithm on a low-dimensional problem where the exact Fisher information $I_F(\\Sigma)$ is computable, compare the exact natural gradient with the approximation $(\\Sigma\\otimes\\Sigma)\\nabla_\\Sigma L$, and test whether the approximate update still drives $\\mathbb{E}|\\nabla L|^2$ to zero at the claimed rate; a measured bias that does not shrink with step size, or divergence for some initialization, would falsify the claim that the practical algorithm inherits Theorem 1's guarantees.","tokens_in":1925,"feed_emoji":"📐","tokens_out":7733,"duration_ms":158100,"temperature":0.7,"pith_summary":"Most variational Bayes methods optimize the evidence lower bound (ELBO) in Euclidean parameter spaces, which forces constraints like positive-definite covariance matrices into awkward reparameterizations. This paper tries to remove that restriction by treating the variational parameter space as a Riemannian manifold and using the natural gradient, preconditioned by the Fisher information matrix, as the search direction. The authors prove that the resulting manifold stochastic-gradient-with-momentum algorithm converges to a local mode of the ELBO, with rate $O(1/\\sqrt{T})$ in general and $O(1/T^{2-2\\epsilon})$ when the ELBO is strongly retraction-convex. They instantiate the method for Gaussian and inverse-Wishart variational families and show in experiments that it is stable and less sensitive to initialization than existing VB algorithms. If the proof holds, a broad class of constrained variational inference problems becomes unconstrained geometric optimization.","feed_headline":"Variational Bayes on curved spaces converges at O(1/√T)","feed_subtitle":"A manifold-based update handles constrained covariance matrices directly and is less sensitive to initialization.","key_machinery":"The central object is the variational family $\\mathcal{N}=\\{q_\\lambda\\}$ viewed as a Riemannian submanifold or quotient manifold of the parameter space, equipped with the Fisher-Rao metric $\\langle\\zeta,\\xi\\rangle=\\zeta^\\top I_F(\\lambda)\\xi$. Lemma 3.1 identifies the natural gradient $I_F(\\lambda)^{-1}\\nabla L(\\lambda)$ with the Riemannian gradient of the lower bound in the ambient manifold; for submanifolds it is projected onto the tangent space, and for quotient manifolds its horizontal lift is the Riemannian gradient. The update uses a retraction $R_\\lambda$, a first-order approximation of the exponential map that replaces the Euclidean move $\\lambda+\\xi$, and a vector transport that pushes momentum vectors between tangent spaces. Theorem 1 provides the convergence machinery: bounded gradients with martingale-difference noise, retraction-Lipschitz gradients, and, for the faster rate, strong retraction-convexity.","core_discovery":"The central claim is that constrained variational inference is not a special case to be patched with factorizations, but a genuine Riemannian optimization problem. Lemma 3.1 shows that the natural gradient $I_F(\\lambda)^{-1}\\nabla L(\\lambda)$ is the Riemannian gradient of the lower bound when the manifold is equipped with the Fisher-Rao metric, so the steepest-ascent direction automatically respects the information geometry of the variational family. Algorithm 1 iterates $\\lambda_{t+1}=R_{\\lambda_t}(\\epsilon Y_t)$, with $Y_t$ a momentum-averaged estimate of the projected natural gradient; for submanifolds the projection is orthogonal, and for quotient manifolds it is the horizontal lift. Theorem 1 states that with step size $1/\\sqrt{T}$ the squared Riemannian gradient norm is driven below $C/\\sqrt{T}$, and that strong retraction-convexity improves the expected squared distance to the optimum to $O(T^{2\\epsilon-2})$. The Gaussian instantiation uses the full covariance matrix on the manifold of symmetric positive-definite matrices with the approximation $I_F(\\Sigma)\\approx\\Sigma^{-1}\\otimes\\Sigma^{-1}$, and the Wishart instantiation uses analogous natural-gradient preconditioning; both are tested on data where they converge faster and more stably than previous VB methods.","pith_inferences":["Editorial inference: the convergence rates are proved for the exact Riemannian gradient, so an error analysis for the approximate natural gradients, such as $\\Sigma^{-1}\\otimes\\Sigma^{-1}$ in the Gaussian case, would turn the practical algorithms into theorems; until then, the rates are safest read as applying to the idealized update.","Editorial inference: the same template, projecting a natural-gradient direction onto the tangent space and retracting, should transfer to other constrained variational families, such as low-rank factor covariances or positive-definite matrices in Gaussian processes, wherever a retraction and vector transport exist.","Editorial inference: a direct comparison of updates using the exact Fisher information matrix against the $\\Sigma^{-1}\\otimes\\Sigma^{-1}$ approximation on low-dimensional Gaussian VB would give a testable measure of how much of the observed speedup comes from the approximation rather than from the manifold geometry."],"forward_implications":["For a non-convex evidence lower bound, running the manifold VB algorithm for $T$ steps with step size $1/\\sqrt{T}$ gives $\\min_{t\\le T}\\mathbb{E}|\\nabla L(\\lambda_t)|^2\\le C/\\sqrt{T}$.","For a strongly retraction-convex ELBO, choosing step size $1/T^\\epsilon$ yields $\\mathbb{E}(\\lambda_T-\\lambda^*)^2\\le C_\\epsilon T^{2\\epsilon-2}$, approaching $O(1/T^2)$ as $\\epsilon$ tends to zero.","Because the natural gradient is parameterization-invariant, the manifold update is the same geometric object regardless of how the variational family is parameterized.","Gaussian variational Bayes can use full covariance matrices on the positive-definite manifold rather than factorized covariance structures, with faster convergence and lower variance across random initializations than a factorized Gaussian VB.","The Wishart instantiation matches a gold-standard sequential Monte Carlo posterior estimate for a financial GARCH model while using a small fraction of the compute time."],"supporting_citations":[{"why":"Supplies the differential-geometry toolkit (retractions, vector transport, quotient-manifold calculus) that the manifold VB algorithm is built on.","marker":"Absil et al. (2009)"},{"why":"Introduces the natural gradient and establishes it as the steepest-ascent direction under the Fisher metric, the basis for Lemma 3.1.","marker":"Amari (1998)"},{"why":"Provides the totally retractive neighborhood assumptions and lemmas on parallel transport approximation used in the proof of Theorem 1.","marker":"Huang et al. (2015a)"},{"why":"Defines retraction-convexity and strong retraction-convexity, the curvature conditions behind the $O(1/T^{2-2\\epsilon})$ rate.","marker":"Huang et al. (2015b)"},{"why":"Gives the Fisher information matrix for the multivariate Gaussian, from which the $\\Sigma^{-1}\\otimes\\Sigma^{-1}$ approximation in Manifold Gaussian VB is derived.","marker":"Mardia and Marshall (1984)"},{"why":"Supplies the score-function estimator with control variates used to compute unbiased gradient estimates of the lower bound.","marker":"Tran et al. (2017)"},{"why":"The prior manifold VB method for factor Gaussian covariance that this paper generalizes to arbitrary Riemannian parameter spaces with natural gradient.","marker":"Zhou et al. (2020)"}],"fun_headline_variants":["VB on manifolds converges at root-T with natural gradients","Curved-space VB hits O(1/√T) without factorizing constraints","Manifold Gaussian VB speeds up inference and stabilizes fits","Natural gradient VB on Riemannian manifolds beats standard approaches","Full-covariance VB on manifolds converges provably fast"],"cache_read_input_tokens":25088,"weakest_assumption_plain":"The practical algorithms inherit the paper's convergence rates only if the approximate natural-gradient formulas are close enough to the true natural gradient; the paper gives no error bound for these approximations.","fun_headline_variants_meta":{"raw":{"variants":["VB on manifolds converges at root-T with natural gradients","Curved-space VB hits O(1/√T) without factorizing constraints","Manifold Gaussian VB speeds up inference and stabilizes fits","Natural gradient VB on Riemannian manifolds beats standard approaches","Full-covariance VB on manifolds converges provably fast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1572,"prompt_tokens":993,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":609,"tokens_out":579,"duration_ms":6637,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:25:23.483072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Manifold Gaussian VB algorithm on a low-dimensional problem where the exact Fisher information $I_F(\\Sigma)$ is computable, compare the exact natural gradient with the approximation $(\\Sigma\\otimes\\Sigma)\\nabla_\\Sigma L$, and test whether the approximate update still drives $\\mathbb{E}|\\nabla L|^2$ to zero at the claimed rate; a measured bias that does not shrink with step size, or divergence for some initialization, would falsify the claim that the practical algorithm inherits Theorem 1's guarantees.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the differential-geometry toolkit (retractions, vector transport, quotient-manifold calculus) that the manifold VB algorithm is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the natural gradient and establishes it as the steepest-ascent direction under the Fisher metric, the basis for Lemma 3.1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Fisher information matrix for the multivariate Gaussian, from which the $\\Sigma^{-1}\\otimes\\Sigma^{-1}$ approximation in Manifold Gaussian VB is derived."},{"cited_title":"J., and Kohn, R","cited_arxiv_id":null,"evidence_quote":"Supplies the score-function estimator with control variates used to compute unbiased gradient estimates of the lower bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior manifold VB method for factor Gaussian covariance that this paper generalizes to arbitrary Riemannian parameter spaces with natural gradient."}],"review_version":1}