Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that a landmark-based SLAM estimator's posterior covariance can be predicted in near-constant time per timestep, independent of the number of landmarks, using a structureless interval filter with Lie-Taylor updates.

desk verdict A genuine approximation stack for observability-aware planning, with the central proxy-to-real-estimator validation missing; deserves refereeing and an external validation experiment. read the letter →

arxiv 1908.03790 v6 pith:DIR6UI6U submitted 2019-08-10 cs.RO

classification cs.RO
keywords observability-awaretrajectoryoptimizationbelief-spaceplanningposteriorcovariancepredictionStructurelessIntervalInformationFilterLie-TaylorapproximationaffineobservationmodelsorthographiccameraprojectionvisualSLAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the uncertainty of a landmark-based estimator (visual SLAM) can be predicted cheaply and deterministically along a candidate trajectory, making observability-aware trajectory optimization feasible online. It replaces the full SLAM estimation process with the Structureless Interval Information Filter (SIIF), which bundles high-rate measurements over short intervals and marginalizes out landmarks, so the posterior covariance becomes a smooth function of the nominal trajectory alone. A Lie-Taylor approximation of the bundled measurement update removes dependence on interval length, and for observation models affine in the landmark parameters (including orthographic camera projection) the landmark contributions collapse into a few distribution coefficients, making the objective nearly cost-free in the number of landmarks. If true, planners can minimize an explicit posterior-covariance penalty with gradient-based optimization even in unknown space, instead of relying on visibility or observability-gramian heuristics.

What carries the argument

The load-bearing object is the Structureless Interval Information Filter (SIIF), an approximation that bundles all measurements of an interval into a joint update over the initial ego-error $e_0$ and a nuisance noise vector $W$, then marginalizes out the landmark errors, avoiding any explicit landmark covariance. The Lie-Taylor update is the key computational simplification: instead of integrating the linearized dynamics step by step, it approximates the measurement history by the first $r$ Lie derivatives of the observation model and a precomputable timestamp matrix $W(T)$, which keeps the information update fixed-dimensional. The affine-landmark identity (27)--(29) does the decisive work for scaling: whenever the observation model is affine in the landmark parameter, the marginalization operator is landmark-independent and the total information is a sum over products $\eta_{ij}\,{}^iH^\top {}^{j}H$ of a few coefficient matrices, so a whole landmark cloud or a landmark density contributes as easily as a single landmark.

What would settle it

Run the SIIF/Lie-Taylor covariance predictor alongside an actual landmark-based estimator (for example an EKF or MSCKF visual-inertial filter) over the same trajectory and landmark distribution, and compare the predicted posterior covariance with the realized estimator error covariance over the planning horizon; if they diverge sharply, or if trajectories chosen by the objective fail to reduce realized error, the proxy is not faithful.

Watch

Extended reading notes

Core claim

The central claim is that recursive covariance prediction for SLAM does not require tracking landmark uncertainty or enumerating each landmark. The SIIF treats each time interval as a single bundled measurement over the initial ego-error $e_0$ and an artificial process-noise variable $W$, marginalizes each landmark's linearized contribution into a left-nullspace update, and propagates the accumulated information matrix via $\Lambda_K = M^{-\top}\Lambda_0 M^{-1}$ before a Schur complement extracts the ego-covariance. The Lie-Taylor approximation (Eqs. 14--25) replaces the interval's batch of Jacobians with the first $r$ Lie derivatives of the measurement model, with a timestamp-dependent coupling matrix $W(T)$, so the update cost no longer grows with the interval length $K$. For the affine class (26), the sum over $N$ landmarks in Eq. (15) becomes Eq. (29), weighted by visibility-aware mass coefficients $\eta_{ij}$, which makes evaluation near-constant in $N$ and permits landmark distributions in place of finite linearization points. The paper validates that this predicted covariance tracks the explicit SIIF update over moderate intervals and that trajectories planned with it reduce estimation error nearly as well as the explicit form while being about an order of magnitude cheaper.

Load-bearing premise

The SIIF approximation, which discards loop closures and validates only against itself, faithfully tracks the covariance evolution of a real landmark-based SLAM estimator.

Editorial extensions

If this is right

  • Gradient-based trajectory optimization can directly minimize a posterior-covariance penalty for landmark-based estimators, because the SIIF objective is deterministic and differentiable.
  • With an affine observation model, the cost of evaluating the objective is nearly independent of the number of landmarks: reported timings stay around 17--18 ms for $N=10$ through $N=100$, versus 151 ms for the non-affine Lie-Taylor form.
  • Because landmark information can be expressed against a distribution rather than a list, planners can anticipate new landmarks beyond currently known space instead of only exploiting the current map.
  • In the quadrotor evaluation, posterior-covariance refinement keeps position uncertainty below a safety threshold for much larger fractions of the trajectory than visibility-maximizing or Gramian-maximizing heuristics, at comparable cost increases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper validates the Lie-Taylor approximation only against the explicit SIIF, not against an actual EKF or SLAM filter; a natural next check is to compare both predicted and realized covariances on a real visual-inertial estimator.
  • The interval filter deliberately discards inter-interval loop closures, so its covariance predictions are likely to under-estimate information on long horizons; the method's online promise therefore depends on receding-horizon replanning.
  • The affine decomposition suggests a concrete extension the paper leaves open: replacing orthographic projection with a locally affine surrogate of perspective projection, then checking whether the planning improvements survive with actual perspective cameras.
  • Plugging a learned or occupancy-based landmark density into the mass coefficients $\eta_{ij}$ would turn the predictor into a fully anticipatory active-mapping objective; the paper identifies this as future work rather than demonstrating it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a deterministic, differentiable approximation of posterior covariance evolution for landmark-based SLAM estimators, to be used as an objective in trajectory optimization. The main ingredients are: (i) an interval-based 'Structureless Interval Information Filter' (SIIF) that bundles measurements over time intervals and marginalizes landmark uncertainty, avoiding explicit landmark covariance; (ii) a Lie-Taylor approximation of the interval update that fixes the dimension of the update independent of interval length; and (iii) an affine-observation-model decomposition that aggregates information from many landmarks through 'mass coefficients,' with the potential to replace finite landmark sets by distributions. The method is evaluated on a simulated VIO-equipped quadrotor with orthographic camera, comparing SIIF-based posterior-covariance planning against visibility and Gramian heuristics, and reporting computational timings for various landmark counts.

Significance. If the claims were fully supported, the paper would make a useful contribution to belief-space planning: a compact, differentiable covariance predictor for landmark-based estimators would enable observability-aware trajectory optimization at a cost that scales weakly with map size. The Lie-Taylor bundling and the affine landmark decomposition are conceptually attractive and are derived with care; the Pareto-style comparison against heuristic objectives is a sensible evaluation design. However, the central accuracy claim is not yet established. All numerical validation compares one variant of the SIIF proxy against another, never against a real estimator, and the claimed near-constant scaling rests on an unspecified computation of the mass coefficients. The paper is a well-written technical report with promising ideas, but the evidence currently supports only internal consistency of the approximations, not the headline claim of accurately predicting SLAM performance.

major comments (3)
  1. [§6.1, §3.2, Figs. 2–4] The accuracy claim for the SIIF proxy is not validated against an actual landmark-based estimator. Figure 2 compares the Lie-Taylor update (Eq. 25) with the explicit SIIF update (Eq. 14), which are two instances of the same interval-LTV model family. Section 3.2 explicitly concedes that the interval filter is 'clearly sub-optimal' and discards inter-interval loop closures, and Footnote 3 concedes that a real EKF's covariance is observation-dependent and random, whereas the SIIF covariance is a deterministic function of the nominal trajectory. Consequently, the planning results in Figs. 3 and 4 show improvement in a proxy criterion, not demonstrated improvement in actual estimator performance. A concrete test is needed: evaluate the optimized trajectories with a standard filter (e.g., an EKF or MSCKF on the same IMU and camera model) and report the resulting estimation RMSE or covariance consistency, comparing pc-lie and pc-exact against that baseline.
  2. [Table 1, Eqs. (29)–(30), Algorithm 2] The claimed near-constant scaling with the number of landmarks is not supported by the equations as written. Eq. (30) defines the mass coefficients ηij(x) as a sum over N landmarks, which is O(N) for a finite landmark set. Algorithm 2 calls ComputeMassCoefficients(x0, M) but the manuscript does not specify how M is represented or how η is obtained in constant time. Section 5.2 mentions pre-computation and lookup, but Table 1 does not state whether a lookup table was used, nor the discretization error, memory cost, or interpolation scheme. Unless the mass coefficients are evaluated from a precomputed or learned distribution with O(1) per-query cost, the affine update remains O(N) to form η, and only the inner accumulation over affine components is O(n_l^2). Please state exactly what operation is timed and what assumptions on M make Table 1 valid.
  3. [§6, §7, Eq. (9)] The numerical evidence is confined to an orthographic camera model, which the authors themselves state in Section 7 is only a proxy for perspective cameras. Since the observation model is the defining component of a landmark-based estimator, this limits support for the claim that the framework predicts 'SLAM performance' in realistic settings. Additionally, the constant-visibility assumption in Eq. (9) is untested against situations where landmarks enter or leave the field of view within an interval, which is common in the intended visual-SLAM use case. I recommend adding a sensitivity analysis or an evaluation with a perspective projection model and with visibility transitions occurring inside intervals, to show that the approximation remains accurate where its axioms are violated.
minor comments (4)
  1. [Fig. 2 caption] The caption says 'empirical and Lie-Taylor implementations,' but the comparison is between the explicit SIIF update (Eq. 14) and the Lie-Taylor update (Eq. 25), not against an empirical rollout. Please reword to avoid ambiguity.
  2. [Eqs. (22)–(25)] The stacked notation reuses the symbol jH(n) for both the Jacobian of the j-th Lie derivative and as a row block; consider writing [0H(n)⊤, 1H(n)⊤, ..., (r−1)H(n)⊤]⊤ explicitly, since the current notation makes the Kronecker structure hard to follow.
  3. [Section 6.2 and Figs. 3–4] The plots aggregate 50 random trials, but no error bars or variability bands are shown. Reporting medians with interquartile ranges, or at least stating the spread, would make the Pareto-claim and the safety-time histograms more convincing.
  4. [Throughout] There are occasional typographical inconsistencies, such as 'foregoing' in the Introduction where 'aforegoing' or 'preceding' is intended, and 'i-th' vs. 'j-th' in Section 4.2. A careful proofread would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the main limitation is unvalidated external accuracy, not circular derivation.

full rationale

The paper's derivation chain is self-contained and no step reduces to its own inputs. The SIIF covariance is computed from the stated LTV dynamics (Eqs. 7-8), nullspace marginalization (Eq. 13), and information propagation (Eqs. 15-17), none of which presuppose the target posterior covariance. The Lie-Taylor update (Eq. 25) is explicitly an approximation of the explicit SIIF update (Eq. 14), and its validation in Section 6.1 against Eq. 14 checks internal consistency of that approximation; this is not a circular reduction because Eq. 14 is not fitted from Eq. 25. The affine mass-coefficient decomposition (Eq. 29) is a mathematical identity under the affine observation assumption (Eq. 26). The paper contains no fitted parameter relabeled as a prediction and no load-bearing self-citation chain: its cited precedents (MSCKF [17], Preiss et al. [20], Zhang & Scaramuzza [28]) are external. The genuine weakness is external validity: the paper admits in Section 3.2 that the interval filter is "clearly sub-optimal" and discards loop closures, and Footnote 3 concedes that a real EKF covariance is observation-dependent, so the SIIF covariance is only a proxy. Section 7 also concedes orthographic projection is only a proxy for perspective cameras. These are limitations on whether the predicted covariance matches a real SLAM estimator, and they should be weighed as correctness risk, not as circularity. The manuscript never claims to derive real-SLAM covariance from SIIF by definition; it claims an approximation, which can be under-validated without being circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The derivation rests on user-chosen interval length and Lie-derivative order, plus several approximations such as constant visibility, jump-process noise, and the orthographic camera model. No parameters are fitted to data; the main unvalidated premise is that the SIIF proxy matches a true SLAM filter.

free parameters (2)
  • Interval length K = K=7 in experiments
    User-specified bundling horizon; approximation error grows with K (Fig. 2), and K is chosen by hand.
  • Number of Lie derivatives r = not stated in paper
    Truncation order of the Lie-Taylor series in Eq. 18; accuracy and cost depend on r, but its value is not reported.
assumptions (4)
  • domain assumption Linear time-varying approximation of the nonlinear dynamics and measurement model is valid over the planning horizon.
    Eq. 3 linearizes about a nominal trajectory; assumes small perturbations, standard in belief-space planning.
  • ad hoc to paper Visibility is constant over each interval.
    Eq. 9 replaces sigma(x_k; l) with sigma(x_0; l); an approximation that is reasonable only for short intervals and is not validated against a full filter.
  • ad hoc to paper Process noise can be modeled as an instantaneous jump at interval start.
    Eqs. 19 and 20 replace the full path of Brownian noise with a single jump; a heuristic stochastic extension with no formal error bound.
  • domain assumption Orthographic projection is an adequate proxy for perspective cameras for the purpose of planning.
    The affine decomposition in Eq. 31 applies to orthographic projection; the authors state that perspective projection is not included and leave a better approximation to future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators." pith.science (2026). https://pith.science/paper/DIR6UI6U

@misc{pith2026190803790,
  author       = {Pith},
  title        = {Pith review of: Towards Online Observability-Aware Trajectory Optimization for Landmark-based Estimators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIR6UI6U}},
  note         = {Machine review of arXiv:1908.03790}
}
abstract

As autonomous systems increasingly rely on onboard sensing for localization and perception, the parallel tasks of motion planning and state estimation become more strongly coupled. This coupling is well-captured by augmenting the planning objective with a posterior-covariance penalty -- however, prediction of the estimator covariance is challenging when the observation model depends on unknown landmarks, as is the case in Simultaneous Localization and Mapping (SLAM). This paper addresses these challenges in the case of landmark- and SLAM-based estimators, enabling efficient prediction (and ultimately minimization) of this performance metric. First, we provide an interval-based filtering approximation of the SLAM inference process which allows for recursive propagation of the ego-covariance while avoiding the quadratic complexity of explicitly tracking landmark uncertainty. Secondly, we introduce a Lie-derivative measurement bundling scheme that simplifies the recursive "bundled" update, representing significant computational savings for high-rate sensors such as cameras. Finally, we identify a large class of measurement models (which includes orthographic camera projection) for which the contributions from each landmark can be directly combined, making evaluation of the information gained at each timestep (nearly) independent of the number of landmarks. This also enables the generalization from finite sets of landmarks $\{\ell^{(n)} \}$ to distributions, foregoing the need for fully-specified linearization points at planning time and allowing for new landmarks to be anticipated. Taken together, these contributions allow SLAM performance to be accurately and efficiently predicted, paving the way for online, observability-aware trajectory optimization in unknown space.

Figures

Figures reproduced from arXiv: 1908.03790 by the authors.

Figure 1
Figure 1. The original SLAM factor graph includes both pose and landmark vari [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Numerical comparison between empirical and Lie-Taylor implementations [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. Each choice of Jobs defines an effective trade-off curve between trajec￾tory cost Jc and estimation improvement (larger is better). We plot this curve by sweeping through the weighting parameter r0 and aggregating results for a batch of random trials. Refinement based on heuristic objectives (see max-viz and max-gramian) or approximate sensor models (see pc-lie-cond) does not always yield significant estimation impr… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Evolution of the covariance traces shown for each refined trajectory, un [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Our visibility weighting function (34) plotted against [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages

  1. [1]

    In: Proc

    Arneberg, J., Tal, E., Karaman, S.: Guidance laws for partially-observable inter- ception based on linear covariance analysis. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 4185–4191. IEEE (2018)

  2. [2]

    The International Journal of Robotics Research 36(10), 1053–1072 (2017) Observability-Aware Traj

    Bloesch, M., Burri, M., Omari, S., Hutter, M., Siegwart, R.: Iterated extended kalman filter based visual-inertial odometry using direct photometric feedback. The International Journal of Robotics Research 36(10), 1053–1072 (2017) Observability-Aware Traj. Optimization for Landmark-based Estimators 23

  3. [3]

    In: Proc

    Bry, A., Roy, N.: Rapidly-exploring random belief trees for motion planning under uncertainty. In: Proc. IEEE Conf. Robot. Autom. (ICRA), pp. 723–730. IEEE (2011)

  4. [4]

    Burri, M., Nikolic, J., Gohl, P., Schneider, T., Rehder, J., Omari, S., Achtelik, M.W., Siegwart, R.: The euroc micro aerial vehicle datasets. Int. J. of Robotics Research (2016). DOI 10.1177/0278364915620033. URL http://ijr.sagepub.com/ content/early/2016/01/21/0278364915620033.abstract

  5. [5]

    IFAC Proceedings Volumes 33(27), 663–668 (2000)

    Conticelli, F., Bicchi, A., Balestrino, A.: Observability and nonlinear observers for mobile robot localization. IFAC Proceedings Volumes 33(27), 663–668 (2000)

  6. [6]

    IEEE transactions on pattern analysis and machine intelligence 29(6), 1052–1067 (2007)

    Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelligence 29(6), 1052–1067 (2007)

  7. [7]

    Dellaert, F., Kaess, M.: Square root sam: Simultaneous localization and mapping via square root information smoothing. Int. J. of Robotics Research 25(12), 1181– 1203 (2006)

  8. [8]

    In: Proc

    Elimelech, K., Indelman, V.: Scalable sparsification for efficient decision making under uncertainty in high dimensional state spaces. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 5668–5673. IEEE (2017)

Show all 28 references
  1. [9]

    In: Proc

    Elisha, Y.B., Indelman, V.: Active online visual-inertial navigation and sensor cal- ibration via belief space planning and factor graph based incremental smoothing. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 2616–2622. IEEE (2017)

  2. [10]

    In: Proc

    Falanga, D., Foehn, P., Lu, P., Scaramuzza, D.: PAMPC: Perception-aware MPC for quadrotors. In: Proc. IEEE Conf. Int. Rob. Sys. (IROS), pp. 1–8. IEEE (2018)

  3. [11]

    Indelman, V., Carlone, L., Dellaert, F.: Planning in the continuous domain: A generalized belief space approach for autonomous navigation in unknown environ- ments. Int. J. of Robotics Research 34(7), 849–882 (2015)

  4. [12]

    Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J.J., Dellaert, F.: isam2: Incremental smoothing and mapping using the bayes tree. Int. J. of Robotics Research p. 0278364911430419 (2011)

  5. [13]

    Philo- sophical Transactions of the Royal Society of London356(1740), 1153–1173 (1998)

    Kanade, T., Morris, D.D.: Factorization methods for structure from motion. Philo- sophical Transactions of the Royal Society of London356(1740), 1153–1173 (1998)

  6. [14]

    Kopitkov, D., Indelman, V.: No belief propagation required: Belief space planning in high-dimensional state spaces via factor graphs, the matrix determinant lemma, and re-use of calculation. Int. J. of Robotics Research 36(10), 1088–1130 (2017)

  7. [15]

    In: Conf

    Krener, A.J., Ide, K.: Measures of unobservability. In: Conf. on Dec. and Control (CDC), pp. 6401–6406. IEEE (2009)

  8. [16]

    IEEE Trans

    Mariottini, G.L., Morbidi, F., Prattichizzo, D., Vander Valk, N., Michael, N., Pap- pas, G., Daniilidis, K.: Vision-based localization for leader–follower formation con- trol. IEEE Trans. Robotcs 25(6), 1431–1438 (2009)

  9. [17]

    In: Proc

    Mourikis, A.I., Roumeliotis, S.I.: A multi-state constraint kalman filter for vision- aided inertial navigation. In: Proc. IEEE Conf. Robot. Autom. (ICRA), pp. 3565–

  10. [18]

    In: American Control Conference (ACC) (2019)

    Murali, V., Spasojevic, I., Guerra, W., Karaman, S.: Perception-aware trajectory generation for aggressive quadrotor flight using differential flatness. In: American Control Conference (ACC) (2019)

  11. [19]

    Nobre, F., Heckman, C.: Learning to calibrate: Reinforcement learning for guided calibration of visualinertial rigs. Int. J. of Robotics Research (2019)

  12. [20]

    Preiss, J.A., Hausman, K., Sukhatme, G.S., Weiss, S.: Simultaneous self-calibration and navigation using trajectory optimization. Int. J. of Robotics Research (2018)

  13. [21]

    Prentice, S., Roy, N.: The belief roadmap: Efficient planning in belief space by factoring the covariance. Int. J. of Robotics Research 28(11-12), 1448–1465 (2009) 24 Kristoffer M. Frey et al

  14. [22]

    In: Conf

    Rafieisakhaei, M., Chakravorty, S., Kumar, P.: The use of the observability gramian for partially observed path planning problems. In: Conf. on Dec. and Control (CDC), pp. 1523–1528. IEEE (2017)

  15. [23]

    https://j-towns

    Townsend, J.: Differentiating the singular value decomposition. https://j-towns. github.io/papers/svd-derivative.pdf (2016). Accessed: 2019-05-14

  16. [24]

    Van Den Berg, J., Abbeel, P., Goldberg, K.: LQG-MP: Optimized path planning for robots with motion uncertainty and imperfect state information. Int. J. of Robotics Research 30(7), 895–913 (2011)

  17. [25]

    Van Den Berg, J., Patil, S., Alterovitz, R.: Motion planning under uncertainty using iterative local optimization in belief space. Int. J. of Robotics Research 31(11), 1263–1278 (2012)

  18. [26]

    Mathematical program- ming 106(1), 25–57 (2006)

    W¨ achter, A., Biegler, L.T.: On the implementation of an interior-point filter line- search algorithm for large-scale nonlinear programming. Mathematical program- ming 106(1), 25–57 (2006)

  19. [27]

    In: Proc

    Zhang, Z., Scaramuzza, D.: Perception-aware receding horizon navigation for MAVs. In: Proc. IEEE Conf. Robot. Autom. (ICRA), pp. 2534–2541. IEEE (2018)

  20. [28]

    In: Proc

    Zhang, Z., Scaramuzza, D.: Beyond point clouds: Fisher information field for active visual localization. In: Proc. IEEE Conf. Robot. Autom. (ICRA). IEEE (2019) A Modeling Field-of-View Because landmarks are often distributed non-uniformly, it is important that generated traject...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.