REVIEW 3 major objections 5 minor 35 references
Joint State and Noise Covariance Estimation
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper establishes a closed-form update for the noise covariance in joint MAP/ML Gaussian state estimation, and shows that alternating this update with standard least squares recovers trajectories nearly as well as knowing the true…
desk verdict Correct conditional optimality, but the covariance estimate's bias is unquantified and the theory doesn't cover the tested algorithm; still worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the residual sample covariance $S(x)=\frac{1}{k}\sum_{i=1}^k r_i(x)r_i(x)^\top$. At any fixed state, the covariance subproblem minimizes $-\log\det P+\langle M(x),P\rangle$ over $P\succeq0$, a strictly convex program whose unique solution is $P^\star(x)=M(x)^{-1}$; substituting this solution back into the objective leaves a reduced problem in $x$, which in the unconstrained and diagonal cases is simply minimizing $\log\det M(x)$ (or its diagonal projection). The block-coordinate algorithm carries the argument by alternating the closed-form covariance update with a standard weighted-least-squares update for $x$, so the expensive part of the estimation remains the usual sparse nonlinear least-squares step and the covariance update reuses residuals that step already computes.
What would settle it
Construct residuals whose sample covariance $S(x_0)$ is singular, choose a nonzero nullspace vector $u$, and evaluate the unconstrained ML objective at $P_0+c u u^\top$ for growing $c$; the theory predicts the value decreases without bound. If instead it reaches a finite minimum, the ill-posedness claim fails.
Extended reading notes
Core claim
Under the joint MAP model with a Wishart prior on the information matrix $P=\Sigma^{-1}$, the conditional negative log-posterior at a fixed state $x$ is $-\log\det P+\langle M(x),P\rangle$, where $M(x)=(kS(x)+V^{-1})/(k+\nu-m-1)$ and $S(x)$ is the sample covariance of the residuals. This inner problem is strictly convex in $P$, and its unique solution is $P^\star(x)=M(x)^{-1}$; for the diagonal variant the solution is $\mathrm{Diag}(M(x))^{-1}$, for the eigenvalue-constrained variant the eigen-decomposition of $M(x)$ is used with the eigenvalues mapped through the clipping in (19), and for the combined variant the same clipping is applied to the diagonal entries. The maximum-likelihood version uses the same formulas with $S(x)$ in place of $M(x)$, except that a singular $S(x)$ makes the unconstrained ML objective unbounded below. Because the inner problem is solved exactly, the joint problem reduces to an outer nonconvex problem in $x$ alone, and the paper's block-coordinate algorithm alternates a weighted least-squares state step with this closed-form covariance step.
Load-bearing premise
The proof of convergence for the main alternating algorithm assumes the state space is bounded and the state step is a specific gradient step on the curved state space, while SLAM states include unbounded translations and the experiments use a different solver step, so the convergence guarantee does not yet cover the setting that was actually tested.
Editorial extensions
If this is right
- A robot or mapping system can refresh its measurement weights during operation: after each state step, replace the covariance with the closed-form conditional estimate and continue with the same sparse least-squares solver.
- No ground-truth calibration dataset is needed for the joint MAP variant; the noise model is learned from the measurements themselves, with the Wishart prior absorbing whatever calibration knowledge exists.
- When the residual sample covariance is singular or nearly singular, the theory says the plain ML problem has no solution, so the eigenvalue floor and the Wishart prior are not optional refinements but necessary safeguards.
- Structural side information—independent noise components, known sensor variance bounds—can be enforced exactly through the diagonal and eigenvalue constraints without changing the alternating structure.
- The conditional covariance step costs $O(k m^2)$ with small $m$ in typical SLAM, so the joint estimator's per-iteration overhead stays negligible relative to the nonlinear least-squares state step.
Reading between the lines
- A natural extension is to feed this closed-form update into a robust estimator: reweighted residuals would enter both the state least-squares step and the sample covariance, yielding an outlier-resilient joint estimator with the same alternation.
- The ill-posedness theorem marks an identifiability boundary for graph-structured SLAM: in poorly connected graphs the number of independent residuals per covariance block is small, so the prior and eigenvalue floor do real statistical work rather than just encoding prior knowledge.
- The same conditional MAP/ML update could serve as a fast warm-start or proposal for full-Bayesian treatments of unknown covariance, since it gives the conditional mode in closed form at negligible cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies joint estimation of states x and noise covariance (information) matrix P from Gaussian measurements. It formulates joint MAP and ML problems, shows that the inner covariance subproblem is convex for fixed x, and gives analytical solutions (Theorem 1) under diagonal and eigenvalue constraints. It also proves that the unconstrained joint ML problem is unbounded below when the sample covariance is singular (Theorem 2). On the algorithmic side, it proposes a variable-elimination scheme and block-coordinate-descent (BCD) schemes (Algorithms 2-4), with convergence theorems (Theorems 3-4), and validates the approach on linear measurement models, pose-graph optimization benchmarks, and the RIM 3D PGO dataset.
Significance. The conditional optimality result is clean and useful: Theorem 1 correctly characterizes the exact minimizer of the stated objective for each variant, and Theorem 2 correctly identifies the ML ill-posedness due to singular sample covariance. The proposed BCD update has negligible per-iteration cost and can be integrated into existing least-squares solvers, which is a practical strength. If the convergence and bias concerns below are addressed, the framework would be a valuable reference for online covariance estimation in SLAM and related problems. The paper is also commendably honest about several limitations, including the compactness restriction in Section VII.
major comments (3)
- [§V-B, Theorem 3, §VI-B] Theorem 3, as stated, does not apply to the algorithm actually tested in §VI-B. The theorem assumes that M and P are compact submanifolds and that Step 1 of Algorithm 3 is a Riemannian gradient step with η=1/eL, leading to Eq. (29). The PGO experiments instead use g2o's Powell's Dog-Leg method on SE(2), whose translational component is noncompact, and the constraint sets for P in Problems 3 and 4 are compact convex sets with nonempty boundary, not smooth submanifolds. The acknowledgment in Section VII that "Theorem 3 requires M to be compact" is accurate, but the gap remains load-bearing because the paper's stated contribution includes convergence analysis for the BCD algorithm and the experimental validation is precisely in the regime not covered by Theorem 3. I request either a convergence result for the actual descent update used in the experiments or a clearly scoped statement that Theorem 3 applies only to an idealized setting, with the PGO experiments presented as heuristic validation.
- [§VI-B, Eq. (3), Fig. 5] The evaluation of covariance estimates does not separate bias from variance. Since S(x) in Eq. (3) is the sample covariance of residuals, evaluating it at the jointly estimated x̂ instead of at x_true systematically reduces the residuals; in a linear model with n estimated parameters and K=km scalar residuals, the expected residual covariance is approximately ((K−n)/K)Σ_true under a balanced projection. For the homoscedastic PGO setting in §VI-B, K=16,794 and n≈10,500, giving a naive factor of about 0.375, so the small 2-Wasserstein errors reported in Fig. 5 could be consistent with a substantial downward bias rather than an accurate covariance estimate. Because the motivation in Section I is uncertainty-critical decision making, the paper should quantify this bias, for example by reporting the expected ratio of estimated to true eigenvalues or the calibration of confidence ellipsoids, and should discuss how the eigenvalue constraints and Wishart prior mitigate it.
- [Appendix A, Eq. (41), Remark 7] The mode-matching procedure in Algorithm 1 sets ν = w_prior k + m + 1, which makes the prior strength grow with the number of measurements. Consequently, Eq. (41) shows that the conditional MAP estimator converges to (w_prior Σ0 + Σ_true)/(1+w_prior) as k→∞, not to Σ_true, whenever Σ0 ≠ Σ_true. Remark 7 discloses this non-vanishing influence of the prior, but the abstract and Section I present the result as "the optimal noise covariance estimate" without this qualification, and the experiments in §VI-B use exactly this prior construction with w_prior=0.1 and a deliberately inaccurate Σ0. I recommend either using a fixed-ν prior so that the data dominate as k grows, or explicitly rephrasing the optimality and covariance-accuracy claims as being relative to the chosen objective and prior, and studying the resulting bias in the reported error metrics.
minor comments (5)
- [Abstract and Section I] The phrase "maximuma posteriori" appears with a missing space in the abstract and in Section I; it should be "maximum a posteriori".
- [§V-B] In the sentence beginning "In this section, present two types of algorithms", the word "we" is missing before "present".
- [§VI-C] The RIM experiment is only qualitative; the claim that BCD trajectories are "significantly more accurate" would be stronger with a quantitative comparison, such as RMSE or ATE relative to the fixed-covariance baseline.
- [Fig. 5] Figure 5 does not show error bars or confidence intervals even though 50 Monte Carlo runs are used; adding them would make the comparisons consistent with Figure 3 and would help assess statistical significance.
- [Theorem 3] The phrase "compact submanifold" is imprecise for the constraint sets P in Problems 3 and 4, which are closed convex sets with boundary rather than smooth manifolds; the statement should say "compact constraint sets" or justify the manifold-with-boundary treatment.
Circularity Check
No circularity: the covariance update is the exact minimizer of the stated objective, the Wishart prior is a disclosed modeling input, and the convergence theory is imported from an external theorem.
full rationale
The derivation chain is self-contained. The joint MAP/ML objective (Problem 1 and Remark 1) is the negative log posterior/likelihood; Theorem 1 obtains P*(x) by solving the convex inner subproblem (14) via KKT conditions (Appendix C), so the covariance update is the exact minimizer of the stated objective, not an assumed conclusion. The Wishart prior enters as a modeling input through Algorithm 1 with explicit mode-matching, and its asymptotic behavior is disclosed in Remark 7. M(x) is defined from residuals and prior, and P*(x)=M(x)^-1 follows algebraically from first-order optimality; no equation in the derivation is equivalent to the target result. The convergence guarantees in Theorems 3-4 are imported from an external reference [25] (Peng and Vidal), not from the authors' own prior work; the only self-citation, [19], concerns reliable graphs for SLAM and is not load-bearing for the covariance-estimation claims. The use of Powell's Dog-Leg in experiments while Theorem 3 analyzes Riemannian gradient descent, and the compactness assumption in Theorem 3, are validity/scope limitations, not circularity. No fitted parameter is renamed as a prediction; the experimental Wasserstein/RMSE comparisons are external evaluations against simulated ground truth.
Assumptions & free parameters
free parameters (5)
- lambda_min =
1e-4
- lambda_max =
1e4
- w_prior =
0.1
- Sigma_0 =
0.002I (PGO), 0.01I (RIM)
- Number of outer BCD iterations =
13 (PGO), 10 (RIM)
assumptions (4)
- domain assumption Measurement noise is zero-mean Gaussian with covariance Sigma_true shared across all measurements of a given type.
- domain assumption A Wishart prior W(P; V, nu) is placed on the noise information matrix P.
- domain assumption The state manifold M and measurement functions h_i are smooth; M is a product manifold containing R^d and SO(d) components.
- ad hoc to paper Compactness of M and P, and Lipschitz smoothness of R, as required by Assumptions 1 and 2 for Theorem 3.
Cite this review
Pith. "Pith review of Joint State and Noise Covariance Estimation." pith.science (2026). https://pith.science/paper/DHDV7VW6
@misc{pith2026250204584,
author = {Pith},
title = {Pith review of: Joint State and Noise Covariance Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/DHDV7VW6}},
note = {Machine review of arXiv:2502.04584}
}
read the original abstract
This paper tackles the problem of jointly estimating the noise covariance matrix alongside states (parameters such as poses and points) from measurements corrupted by Gaussian noise and, if available, prior information. In such settings, the noise covariance matrix determines the weights assigned to individual measurements in the least squares problem. We show that the joint problem exhibits a convex structure and provide a full characterization of the optimal noise covariance estimate (with analytical solutions) within joint maximum a posteriori and likelihood frameworks and several variants. Leveraging this theoretical result, we propose two novel algorithms that jointly estimate the primary parameters and the noise covariance matrix. Our BCD algorithm can be easily integrated into existing nonlinear least squares solvers, with negligible per-iteration computational overhead. To validate our approach, we conduct extensive experiments across diverse scenarios and offer practical insights into their application in robotics and computer vision estimation problems with a particular focus on SLAM.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Paul D Abramson.Simultaneous estimation of the state and noise statistics in linear dynamical systems, volume
-
[2]
Optimization algorithms on matrix manifolds
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009
work page 2009
-
[3]
Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. Ceres Solver, 10 2023. URL https://github.com/ ceres-solver/ceres-solver
work page 2023
-
[4]
Cambridge University Press, 2024
Timothy D Barfoot.State estimation for robotics. Cambridge University Press, 2024
work page 2024
-
[5]
Timothy D Barfoot, James R Forbes, and David J Yoon. Exactly sparse Gaussian variational inference with appli- cation to derivative-free batch nonlinear state estimation. The International Journal of Robotics Research, 39(13): 1473–1502, 2020
work page 2020
-
[6]
Springer, 2006
Christopher M Bishop and Nasser M Nasrabadi.Pattern recognition and machine learning, volume 4. Springer, 2006
2006
-
[7]
An introduction to optimization on smooth manifolds
Nicolas Boumal. An introduction to optimization on smooth manifolds. Available online, Aug 2020. URL http://www.nicolasboumal.net/book
2020
-
[8]
Boyd and L
S. Boyd and L. Vandenberghe.Convex optimization. Cambridge university press, 2004
2004
Show all 35 references
-
[9]
Simultaneous localization and mapping: Present, future, and the robust-perception age.arXiv preprint arXiv:1606.05830, 2016
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, Jose Neira, Ian D Reid, and John J Leonard. Simultaneous localization and mapping: Present, future, and the robust-perception age.arXiv preprint arXiv:1606.05830, 2016
2016 arXiv
-
[10]
G ´omez, Jos ´e M
Carlos Campos, Richard Elvira, Juan J. G ´omez, Jos ´e M. M. Montiel, and Juan D. Tard ´os. ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM.IEEE Transactions on Robotics, 37 (6):1874–1890, 2021
2021
-
[11]
Initialization techniques for 3d slam: A survey on rotation estimation and its use in pose graph optimization
Luca Carlone, Roberto Tron, Kostas Daniilidis, and Frank Dellaert. Initialization techniques for 3d slam: A survey on rotation estimation and its use in pose graph optimization. In2015 IEEE international conference on robotics and automation (ICRA), pages 4597–4604. IEEE, 2015
2015
-
[12]
Kalman filter auto- tuning through enforcing chi-squared normalized error distributions with bayesian optimization.arXiv preprint arXiv:2306.07225, 2023
Zhaozhong Chen, Harel Biggie, Nisar Ahmed, Simon Julier, and Christoffer Heckman. Kalman filter auto- tuning through enforcing chi-squared normalized error distributions with bayesian optimization.arXiv preprint arXiv:2306.07225, 2023
2023 arXiv
-
[13]
borglab/gtsam, May 2022
Frank Dellaert and GTSAM Contributors. borglab/gtsam, May 2022. URL https://github.com/borglab/gtsam
2022
-
[14]
Factor graphs for robot perception.Foundations and Trends® in Robotics, 6(1-2):1–139, 2017
Frank Dellaert, Michael Kaess, et al. Factor graphs for robot perception.Foundations and Trends® in Robotics, 6(1-2):1–139, 2017
2017
-
[15]
Present and future of SLAM in extreme environ- ments: The DARPA SubT challenge.IEEE Transactions on Robotics, 2023
Kamak Ebadi, Lukas Bernreiter, Harel Biggie, Gavin Catt, Yun Chang, Arghya Chatterjee, Christopher E Denniston, Simon-Pierre Desch ˆenes, Kyle Harlow, Shehryar Khattak, et al. Present and future of SLAM in extreme environ- ments: The DARPA SubT challenge.IEEE Transactions on R...
2023
-
[16]
Look ma, no ground truth! ground- truth-free tuning of structure from motion and visual SLAM.arXiv preprint arXiv:2412.01116, 2024
Alejandro Fontan, Javier Civera, Tobias Fischer, and Michael Milford. Look ma, no ground truth! ground- truth-free tuning of structure from motion and visual SLAM.arXiv preprint arXiv:2412.01116, 2024
2024 arXiv
-
[17]
Anyfeature-vslam: Automating the usage of any chosen feature into visual SLAM
Alejandro Fontan, Javier Civera, and Michael Milford. Anyfeature-vslam: Automating the usage of any chosen feature into visual SLAM. InRobotics: Science and Systems, volume 2, 2024
2024
-
[18]
Matrix-valued measures and wishart statistics for target tracking applications.arXiv preprint arXiv:2406.00861, 2024
Robin Forsling, Simon J Julier, and Gustaf Hendeby. Matrix-valued measures and wishart statistics for target tracking applications.arXiv preprint arXiv:2406.00861, 2024
2024 arXiv
-
[19]
Reliable graphs for SLAM.The International Journal of Robotics Research, 38(2-3):260–298, 2019
Kasra Khosoussi, Matthew Giamou, Gaurav S Sukhatme, Shoudong Huang, Gamini Dissanayake, and Jonathan P How. Reliable graphs for SLAM.The International Journal of Robotics Research, 38(2-3):260–298, 2019
2019
-
[20]
g2o: A general framework for graph optimization
Rainer Kuemmerle, Giorgio Grisetti, Hauke Strasdat, Kurt Konolige, and Wolfram Burgard. g2o: A general framework for graph optimization. InProc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), 2011
2011
-
[21]
SLAM-supported self- training for 6d object pose estimation
Ziqi Lu, Yihao Zhang, Kevin Doherty, Odin Severinsen, Ethan Yang, and John Leonard. SLAM-supported self- training for 6d object pose estimation. In2022 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pages 2833–2840. IEEE, 2022
2022
-
[22]
Edmond Malinvaud.Statistical methods of econometrics. 1980
1980
-
[23]
Approaches to adaptive filtering.IEEE Transactions on automatic control, 17(5):693–698, 1972
Raman Mehra. Approaches to adaptive filtering.IEEE Transactions on automatic control, 17(5):693–698, 1972
1972
-
[24]
Olson, J
E. Olson, J. Leonard, and S. Teller. Fast iterative alignment of pose graphs with poor initial estimates. In Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, pages 2262–2269. Ieee, 2006
2006
-
[25]
Block coordinate descent on smooth manifolds: Convergence theory and twenty-one examples.arXiv preprint arXiv:2305.14744, 2023
Liangzu Peng and Ren ´e Vidal. Block coordinate descent on smooth manifolds: Convergence theory and twenty-one examples.arXiv preprint arXiv:2305.14744, 2023
2023 arXiv
-
[26]
UKAEA, 1970
MJD Powell.A new algorithm for unconstrained opti- mization. UKAEA, 1970
1970
-
[27]
Learning covariances for estimation with constrained bilevel optimization
Mohamad Qadri, Zachary Manchester, and Michael Kaess. Learning covariances for estimation with constrained bilevel optimization. In2024 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 15951– 15957. IEEE, 2024
2024
-
[28]
George A. F. Seber and C. J. Wild.Nonlinear Regression. Wiley-Interscience, 1989
1989
-
[29]
Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping
Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Rus Daniela. Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5135–5142. IEEE, 2020
2020
-
[30]
Bundle adjustment—a modern synthesis
Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment—a modern synthesis. InInternational workshop on vision algorithms, pages 298–372. Springer, 1999
1999
-
[31]
Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St ´efan J
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St ´efan J. van der Walt, and et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.Nature...
2020 doi
-
[32]
Variational inference with parameter learning applied to vehicle trajectory estimation.IEEE Robotics and Automation Letters, 5 (4):5291–5298, 2020
Jeremy Nathan Wong, David Juny Yoon, Angela P Schoellig, and Timothy D Barfoot. Variational inference with parameter learning applied to vehicle trajectory estimation.IEEE Robotics and Automation Letters, 5 (4):5291–5298, 2020
2020
-
[33]
Gen- eralized maximum likelihood estimation for perspective- n-point problem.IEEE Robotics and Automation Letters, 2025
Tian Zhan, Chunfeng Xu, Cheng Zhang, and Ke Zhu. Gen- eralized maximum likelihood estimation for perspective- n-point problem.IEEE Robotics and Automation Letters, 2025
2025
-
[34]
On the identification of noise covariances and adaptive kalman filtering: A new look at a 50 year-old problem
Lingyi Zhang, David Sidoti, Adam Bienkowski, Krishna R Pattipati, Yaakov Bar-Shalom, and David L Kleinman. On the identification of noise covariances and adaptive kalman filtering: A new look at a 50 year-old problem. IEEE Access, 8:59362–59388, 2020. APPENDIXA WISHARTPRIOR Co...
2020
-
[332]
National Aeronautics and Space Administration, 1970
1970
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.