REVIEW 3 major objections 4 minor 28 references
Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that a landmark-based SLAM estimator's posterior covariance can be predicted in near-constant time per timestep, independent of the number of landmarks, using a structureless interval filter with Lie-Taylor updates.
desk verdict A genuine approximation stack for observability-aware planning, with the central proxy-to-real-estimator validation missing; deserves refereeing and an external validation experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Structureless Interval Information Filter (SIIF), an approximation that bundles all measurements of an interval into a joint update over the initial ego-error $e_0$ and a nuisance noise vector $W$, then marginalizes out the landmark errors, avoiding any explicit landmark covariance. The Lie-Taylor update is the key computational simplification: instead of integrating the linearized dynamics step by step, it approximates the measurement history by the first $r$ Lie derivatives of the observation model and a precomputable timestamp matrix $W(T)$, which keeps the information update fixed-dimensional. The affine-landmark identity (27)--(29) does the decisive work for scaling: whenever the observation model is affine in the landmark parameter, the marginalization operator is landmark-independent and the total information is a sum over products $\eta_{ij}\,{}^iH^\top {}^{j}H$ of a few coefficient matrices, so a whole landmark cloud or a landmark density contributes as easily as a single landmark.
What would settle it
Run the SIIF/Lie-Taylor covariance predictor alongside an actual landmark-based estimator (for example an EKF or MSCKF visual-inertial filter) over the same trajectory and landmark distribution, and compare the predicted posterior covariance with the realized estimator error covariance over the planning horizon; if they diverge sharply, or if trajectories chosen by the objective fail to reduce realized error, the proxy is not faithful.
Extended reading notes
Core claim
The central claim is that recursive covariance prediction for SLAM does not require tracking landmark uncertainty or enumerating each landmark. The SIIF treats each time interval as a single bundled measurement over the initial ego-error $e_0$ and an artificial process-noise variable $W$, marginalizes each landmark's linearized contribution into a left-nullspace update, and propagates the accumulated information matrix via $\Lambda_K = M^{-\top}\Lambda_0 M^{-1}$ before a Schur complement extracts the ego-covariance. The Lie-Taylor approximation (Eqs. 14--25) replaces the interval's batch of Jacobians with the first $r$ Lie derivatives of the measurement model, with a timestamp-dependent coupling matrix $W(T)$, so the update cost no longer grows with the interval length $K$. For the affine class (26), the sum over $N$ landmarks in Eq. (15) becomes Eq. (29), weighted by visibility-aware mass coefficients $\eta_{ij}$, which makes evaluation near-constant in $N$ and permits landmark distributions in place of finite linearization points. The paper validates that this predicted covariance tracks the explicit SIIF update over moderate intervals and that trajectories planned with it reduce estimation error nearly as well as the explicit form while being about an order of magnitude cheaper.
Load-bearing premise
The SIIF approximation, which discards loop closures and validates only against itself, faithfully tracks the covariance evolution of a real landmark-based SLAM estimator.
Editorial extensions
If this is right
- Gradient-based trajectory optimization can directly minimize a posterior-covariance penalty for landmark-based estimators, because the SIIF objective is deterministic and differentiable.
- With an affine observation model, the cost of evaluating the objective is nearly independent of the number of landmarks: reported timings stay around 17--18 ms for $N=10$ through $N=100$, versus 151 ms for the non-affine Lie-Taylor form.
- Because landmark information can be expressed against a distribution rather than a list, planners can anticipate new landmarks beyond currently known space instead of only exploiting the current map.
- In the quadrotor evaluation, posterior-covariance refinement keeps position uncertainty below a safety threshold for much larger fractions of the trajectory than visibility-maximizing or Gramian-maximizing heuristics, at comparable cost increases.
Reading between the lines
- The paper validates the Lie-Taylor approximation only against the explicit SIIF, not against an actual EKF or SLAM filter; a natural next check is to compare both predicted and realized covariances on a real visual-inertial estimator.
- The interval filter deliberately discards inter-interval loop closures, so its covariance predictions are likely to under-estimate information on long horizons; the method's online promise therefore depends on receding-horizon replanning.
- The affine decomposition suggests a concrete extension the paper leaves open: replacing orthographic projection with a locally affine surrogate of perspective projection, then checking whether the planning improvements survive with actual perspective cameras.
- Plugging a learned or occupancy-based landmark density into the mass coefficients $\eta_{ij}$ would turn the predictor into a fully anticipatory active-mapping objective; the paper identifies this as future work rather than demonstrating it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a deterministic, differentiable approximation of posterior covariance evolution for landmark-based SLAM estimators, to be used as an objective in trajectory optimization. The main ingredients are: (i) an interval-based 'Structureless Interval Information Filter' (SIIF) that bundles measurements over time intervals and marginalizes landmark uncertainty, avoiding explicit landmark covariance; (ii) a Lie-Taylor approximation of the interval update that fixes the dimension of the update independent of interval length; and (iii) an affine-observation-model decomposition that aggregates information from many landmarks through 'mass coefficients,' with the potential to replace finite landmark sets by distributions. The method is evaluated on a simulated VIO-equipped quadrotor with orthographic camera, comparing SIIF-based posterior-covariance planning against visibility and Gramian heuristics, and reporting computational timings for various landmark counts.
Significance. If the claims were fully supported, the paper would make a useful contribution to belief-space planning: a compact, differentiable covariance predictor for landmark-based estimators would enable observability-aware trajectory optimization at a cost that scales weakly with map size. The Lie-Taylor bundling and the affine landmark decomposition are conceptually attractive and are derived with care; the Pareto-style comparison against heuristic objectives is a sensible evaluation design. However, the central accuracy claim is not yet established. All numerical validation compares one variant of the SIIF proxy against another, never against a real estimator, and the claimed near-constant scaling rests on an unspecified computation of the mass coefficients. The paper is a well-written technical report with promising ideas, but the evidence currently supports only internal consistency of the approximations, not the headline claim of accurately predicting SLAM performance.
major comments (3)
- [§6.1, §3.2, Figs. 2–4] The accuracy claim for the SIIF proxy is not validated against an actual landmark-based estimator. Figure 2 compares the Lie-Taylor update (Eq. 25) with the explicit SIIF update (Eq. 14), which are two instances of the same interval-LTV model family. Section 3.2 explicitly concedes that the interval filter is 'clearly sub-optimal' and discards inter-interval loop closures, and Footnote 3 concedes that a real EKF's covariance is observation-dependent and random, whereas the SIIF covariance is a deterministic function of the nominal trajectory. Consequently, the planning results in Figs. 3 and 4 show improvement in a proxy criterion, not demonstrated improvement in actual estimator performance. A concrete test is needed: evaluate the optimized trajectories with a standard filter (e.g., an EKF or MSCKF on the same IMU and camera model) and report the resulting estimation RMSE or covariance consistency, comparing pc-lie and pc-exact against that baseline.
- [Table 1, Eqs. (29)–(30), Algorithm 2] The claimed near-constant scaling with the number of landmarks is not supported by the equations as written. Eq. (30) defines the mass coefficients ηij(x) as a sum over N landmarks, which is O(N) for a finite landmark set. Algorithm 2 calls ComputeMassCoefficients(x0, M) but the manuscript does not specify how M is represented or how η is obtained in constant time. Section 5.2 mentions pre-computation and lookup, but Table 1 does not state whether a lookup table was used, nor the discretization error, memory cost, or interpolation scheme. Unless the mass coefficients are evaluated from a precomputed or learned distribution with O(1) per-query cost, the affine update remains O(N) to form η, and only the inner accumulation over affine components is O(n_l^2). Please state exactly what operation is timed and what assumptions on M make Table 1 valid.
- [§6, §7, Eq. (9)] The numerical evidence is confined to an orthographic camera model, which the authors themselves state in Section 7 is only a proxy for perspective cameras. Since the observation model is the defining component of a landmark-based estimator, this limits support for the claim that the framework predicts 'SLAM performance' in realistic settings. Additionally, the constant-visibility assumption in Eq. (9) is untested against situations where landmarks enter or leave the field of view within an interval, which is common in the intended visual-SLAM use case. I recommend adding a sensitivity analysis or an evaluation with a perspective projection model and with visibility transitions occurring inside intervals, to show that the approximation remains accurate where its axioms are violated.
minor comments (4)
- [Fig. 2 caption] The caption says 'empirical and Lie-Taylor implementations,' but the comparison is between the explicit SIIF update (Eq. 14) and the Lie-Taylor update (Eq. 25), not against an empirical rollout. Please reword to avoid ambiguity.
- [Eqs. (22)–(25)] The stacked notation reuses the symbol jH(n) for both the Jacobian of the j-th Lie derivative and as a row block; consider writing [0H(n)⊤, 1H(n)⊤, ..., (r−1)H(n)⊤]⊤ explicitly, since the current notation makes the Kronecker structure hard to follow.
- [Section 6.2 and Figs. 3–4] The plots aggregate 50 random trials, but no error bars or variability bands are shown. Reporting medians with interquartile ranges, or at least stating the spread, would make the Pareto-claim and the safety-time histograms more convincing.
- [Throughout] There are occasional typographical inconsistencies, such as 'foregoing' in the Introduction where 'aforegoing' or 'preceding' is intended, and 'i-th' vs. 'j-th' in Section 4.2. A careful proofread would improve readability.
Circularity Check
No significant circularity; the main limitation is unvalidated external accuracy, not circular derivation.
full rationale
The paper's derivation chain is self-contained and no step reduces to its own inputs. The SIIF covariance is computed from the stated LTV dynamics (Eqs. 7-8), nullspace marginalization (Eq. 13), and information propagation (Eqs. 15-17), none of which presuppose the target posterior covariance. The Lie-Taylor update (Eq. 25) is explicitly an approximation of the explicit SIIF update (Eq. 14), and its validation in Section 6.1 against Eq. 14 checks internal consistency of that approximation; this is not a circular reduction because Eq. 14 is not fitted from Eq. 25. The affine mass-coefficient decomposition (Eq. 29) is a mathematical identity under the affine observation assumption (Eq. 26). The paper contains no fitted parameter relabeled as a prediction and no load-bearing self-citation chain: its cited precedents (MSCKF [17], Preiss et al. [20], Zhang & Scaramuzza [28]) are external. The genuine weakness is external validity: the paper admits in Section 3.2 that the interval filter is "clearly sub-optimal" and discards loop closures, and Footnote 3 concedes that a real EKF covariance is observation-dependent, so the SIIF covariance is only a proxy. Section 7 also concedes orthographic projection is only a proxy for perspective cameras. These are limitations on whether the predicted covariance matches a real SLAM estimator, and they should be weighed as correctness risk, not as circularity. The manuscript never claims to derive real-SLAM covariance from SIIF by definition; it claims an approximation, which can be under-validated without being circular.
Assumptions & free parameters
free parameters (2)
- Interval length K =
K=7 in experiments
- Number of Lie derivatives r =
not stated in paper
assumptions (4)
- domain assumption Linear time-varying approximation of the nonlinear dynamics and measurement model is valid over the planning horizon.
- ad hoc to paper Visibility is constant over each interval.
- ad hoc to paper Process noise can be modeled as an instantaneous jump at interval start.
- domain assumption Orthographic projection is an adequate proxy for perspective cameras for the purpose of planning.
Cite this review
Pith. "Pith review of Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators." pith.science (2026). https://pith.science/paper/DIR6UI6U
@misc{pith2026190803790,
author = {Pith},
title = {Pith review of: Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators},
year = {2026},
howpublished = {\url{https://pith.science/paper/DIR6UI6U}},
note = {Machine review of arXiv:1908.03790}
}
abstract
As autonomous systems increasingly rely on onboard sensing for localization and perception, the parallel tasks of motion planning and state estimation become more strongly coupled. This coupling is well-captured by augmenting the planning objective with a posterior-covariance penalty -- however, prediction of the estimator covariance is challenging when the observation model depends on unknown landmarks, as is the case in Simultaneous Localization and Mapping (SLAM). This paper addresses these challenges in the case of landmark- and SLAM-based estimators, enabling efficient prediction (and ultimately minimization) of this performance metric. First, we provide an interval-based filtering approximation of the SLAM inference process which allows for recursive propagation of the ego-covariance while avoiding the quadratic complexity of explicitly tracking landmark uncertainty. Secondly, we introduce a Lie-derivative measurement bundling scheme that simplifies the recursive "bundled" update, representing significant computational savings for high-rate sensors such as cameras. Finally, we identify a large class of measurement models (which includes orthographic camera projection) for which the contributions from each landmark can be directly combined, making evaluation of the information gained at each timestep (nearly) independent of the number of landmarks. This also enables the generalization from finite sets of landmarks $\{\ell^{(n)} \}$ to distributions, foregoing the need for fully-specified linearization points at planning time and allowing for new landmarks to be anticipated. Taken together, these contributions allow SLAM performance to be accurately and efficiently predicted, paving the way for online, observability-aware trajectory optimization in unknown space.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
The International Journal of Robotics Research 36(10), 1053–1072 (2017) Observability-Aware Traj
Bloesch, M., Burri, M., Omari, S., Hutter, M., Siegwart, R.: Iterated extended kalman filter based visual-inertial odometry using direct photometric feedback. The International Journal of Robotics Research 36(10), 1053–1072 (2017) Observability-Aware Traj. Optimization for Landmark-based Estimators 23
work page 2017
- [3]
-
[4]
Burri, M., Nikolic, J., Gohl, P., Schneider, T., Rehder, J., Omari, S., Achtelik, M.W., Siegwart, R.: The euroc micro aerial vehicle datasets. Int. J. of Robotics Research (2016). DOI 10.1177/0278364915620033. URL http://ijr.sagepub.com/ content/early/2016/01/21/0278364915620033.abstract
-
[5]
IFAC Proceedings Volumes 33(27), 663–668 (2000)
Conticelli, F., Bicchi, A., Balestrino, A.: Observability and nonlinear observers for mobile robot localization. IFAC Proceedings Volumes 33(27), 663–668 (2000)
work page 2000
-
[6]
IEEE transactions on pattern analysis and machine intelligence 29(6), 1052–1067 (2007)
Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelligence 29(6), 1052–1067 (2007)
work page 2007
-
[7]
Dellaert, F., Kaess, M.: Square root sam: Simultaneous localization and mapping via square root information smoothing. Int. J. of Robotics Research 25(12), 1181– 1203 (2006)
work page 2006
- [8]
Show all 28 references
-
[9]
In: Proc
Elisha, Y.B., Indelman, V.: Active online visual-inertial navigation and sensor cal- ibration via belief space planning and factor graph based incremental smoothing. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 2616–2622. IEEE (2017)
2017
-
[10]
In: Proc
Falanga, D., Foehn, P., Lu, P., Scaramuzza, D.: PAMPC: Perception-aware MPC for quadrotors. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 1–8. IEEE (2018)
2018
-
[11]
Indelman, V., Carlone, L., Dellaert, F.: Planning in the continuous domain: A generalized belief space approach for autonomous navigation in unknown environ- ments. Int. J. of Robotics Research 34(7), 849–882 (2015)
2015
-
[12]
Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J.J., Dellaert, F.: isam2: Incremental smoothing and mapping using the bayes tree. Int. J. of Robotics Research p. 0278364911430419 (2011)
2011
-
[13]
Philo- sophical Transactions of the Royal Society of London356(1740), 1153–1173 (1998)
Kanade, T., Morris, D.D.: Factorization methods for structure from motion. Philo- sophical Transactions of the Royal Society of London356(1740), 1153–1173 (1998)
1998
-
[14]
Kopitkov, D., Indelman, V.: No belief propagation required: Belief space planning in high-dimensional state spaces via factor graphs, the matrix determinant lemma, and re-use of calculation. Int. J. of Robotics Research 36(10), 1088–1130 (2017)
2017
-
[15]
In: Conf
Krener, A.J., Ide, K.: Measures of unobservability. In: Conf. on Dec. and Control (CDC), pp. 6401–6406. IEEE (2009)
2009
-
[16]
IEEE Trans
Mariottini, G.L., Morbidi, F., Prattichizzo, D., Vander Valk, N., Michael, N., Pap- pas, G., Daniilidis, K.: Vision-based localization for leader–follower formation con- trol. IEEE Trans. Robotcs 25(6), 1431–1438 (2009)
2009
-
[17]
In: Proc
Mourikis, A.I., Roumeliotis, S.I.: A multi-state constraint kalman filter for vision- aided inertial navigation. In: Proc. IEEE Conf. Robot. Autom. (ICRA), pp. 3565–
-
[18]
In: American Control Conference (ACC) (2019)
Murali, V., Spasojevic, I., Guerra, W., Karaman, S.: Perception-aware trajectory generation for aggressive quadrotor flight using differential flatness. In: American Control Conference (ACC) (2019)
2019
-
[19]
Nobre, F., Heckman, C.: Learning to calibrate: Reinforcement learning for guided calibration of visualinertial rigs. Int. J. of Robotics Research (2019)
2019
-
[20]
Preiss, J.A., Hausman, K., Sukhatme, G.S., Weiss, S.: Simultaneous self-calibration and navigation using trajectory optimization. Int. J. of Robotics Research (2018)
2018
-
[21]
Prentice, S., Roy, N.: The belief roadmap: Efficient planning in belief space by factoring the covariance. Int. J. of Robotics Research 28(11-12), 1448–1465 (2009) 24 Kristoffer M. Frey et al
2009
-
[22]
In: Conf
Rafieisakhaei, M., Chakravorty, S., Kumar, P.: The use of the observability gramian for partially observed path planning problems. In: Conf. on Dec. and Control (CDC), pp. 1523–1528. IEEE (2017)
2017
-
[23]
https://j-towns
Townsend, J.: Differentiating the singular value decomposition. https://j-towns. github.io/papers/svd-derivative.pdf (2016). Accessed: 2019-05-14
2016
-
[24]
Van Den Berg, J., Abbeel, P., Goldberg, K.: LQG-MP: Optimized path planning for robots with motion uncertainty and imperfect state information. Int. J. of Robotics Research 30(7), 895–913 (2011)
2011
-
[25]
Van Den Berg, J., Patil, S., Alterovitz, R.: Motion planning under uncertainty using iterative local optimization in belief space. Int. J. of Robotics Research 31(11), 1263–1278 (2012)
2012
-
[26]
Mathematical program- ming 106(1), 25–57 (2006)
W¨ achter, A., Biegler, L.T.: On the implementation of an interior-point filter line- search algorithm for large-scale nonlinear programming. Mathematical program- ming 106(1), 25–57 (2006)
2006
-
[27]
In: Proc
Zhang, Z., Scaramuzza, D.: Perception-aware receding horizon navigation for MAVs. In: Proc. IEEE Conf. Robot. Autom. (ICRA), pp. 2534–2541. IEEE (2018)
2018
-
[28]
In: Proc
Zhang, Z., Scaramuzza, D.: Beyond point clouds: Fisher information field for active visual localization. In: Proc. IEEE Conf. Robot. Autom. (ICRA). IEEE (2019) A Modeling Field-of-View Because landmarks are often distributed non-uniformly, it is important that generated traject...
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.